Audience member Alex Chalekian, the founder of Lake Avenue Financial, was so "disgusted" by Mr Fisher's "absolutely horrifying" comments he posted a video about his reaction to Twitter, news.com.au reports.
In it, he claimed the money-management guru also "made a reference to Jeffrey Epstein", spoke about "picking up a girl", "what's in your pants" and "tripping on acid" and "other inappropriate comments".
He said after hearing Mr Fisher's comments his "jaw dropped" and he was like "a deer in the headlights" — and it was time to change that sort of behaviour.
As the initial backlash started to grow, Mr Fisher told Bloomberg he didn't see what all the fuss was about and claimed his words were taken out of context.
"I have given a lot of talks, a lot of times, in a lot of places and said stuff like this and never gotten that type of response," he told the publication.
"Mostly, the audience understands what I am saying."
However, the married father of three eventually apologised in a statement sent to CNBC.
"Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn't have made them," he said.
"I realise this kind of language has no place in our company or industry. I sincerely apologise."
But it was too little, too late, with the state of Michigan pulling $US600 million ($955,278,000) of its pension fund from Mr Fisher's company as a direct result of the outcry.
In a letter from Michigan chief investment officer Jon Braeutigam sent to board members, which was published in full by CNBC, Mr Fisher's comments were slammed as "completely unacceptable".
"There is no excuse to not treat everyone with dignity and respect," part of the letter reads.
"We have high expectations of our managers (and staff), not just with regards to returns but also in how they exhibit integrity and respect to all individuals."
Fisher Investments and the Michigan Department of Treasury's Bureau of Investments had previously enjoyed a 15-year relationship, with the investment fund responsible for managing more than $US600 million of State of Michigan Retirement Systems funds.
Bloomberg also reported similarly vulgar comments made by Mr Fisher at a 2018 conference in the US when he joked about the worst way to sell a mutual fund.
"The most stupid thing you can do is what every mutual fund firm in the world always did … and brag about performance in a direct mail piece," Mr Fisher allegedly said last year.
"(It's) a little bit like walking into a bar and you are a single guy and want to get laid and walking up to some girl and saying: 'Hey you want to have sex?' You just turn yourself into a jerk."
He also reportedly said he regretted not having more sex in his younger years and made a crass comparison between his virility and a Christmas tree.
Mr Fisher — executive chairman and co-chief investment officer of Fisher Investments — has an estimated net worth of $US3.7 billion according to Forbes.
He famously founded his company back in 1979 with just $US250, dropped out of high school "because it was a waste of time" and has published 11 books.
