"I felt ill. I felt like a little kid. I felt like there was a ball in my throat. I thought I was going to cry," he told the magazine.
Berk has vehemently denied the accusation.
Fraser claimed he left the event immediately and didn't tell anyone other than his wife about the incident.
"I didn't want to contend with how that made me feel, or it becoming part of my narrative. I became depressed. I was blaming myself and I was miserable — because I was saying, 'This is nothing; this guy reached around and he copped a feel.' That summer wore on — and I can't remember what I went on to work on next."
Fraser explained that the encounter "made me retreat. It made me feel reclusive".
Afterwards, the actor felt shunned by the Association.
"I don't know if this curried disfavour with the group, with the HFPA. But the silence was deafening."
He told GQ he was rarely invited back to the Golden Globes after 2003, but Berk dismissed the claim to the magazine, telling them the HFPA never acted against him, and that Fraser's career "declined through no fault of ours".
Berk also insisted that Fraser's version of events was a "total fabrication". In his memoir, he described placing his hand on the star's backside during a hug — but as a joke.