Speaker Trevor Mallard is refusing to front over his use of the term "rape" last week, which is being described as slanderous, reckless and a serious lapse in judgment.
Mallard's comments led to a series of events that saw a parliamentary staffer stood down for a historical allegation of assault, which had already been looked at but was reopened.
That staffer, who the Herald has chosen not to identify, has since lashed out at Mallard, saying he feels bullied out of the workplace.
National deputy leader Paula Bennett also slammed Mallard today, accusing him of a "serious lapse in judgment" for his comments.
But Prime Minister Jacinda Ardern is standing by Mallard, saying she continues to have confidence in him.
Ardern would not comment on an individual's employment matter, but noted that the victim's own voice had not been heard and there were always many sides to a story.
"We haven't heard all of those voices necessarily in the media, and we need to keep that in mind," Ardern said.
"When it comes to an employment investigation, I need to leave that to the Speaker. It's not for me, and it's not for trial by media either."
The parliamentary worker said that he is the victim of Mallard's "slanderous" comments.
"I never thought I would ever find myself in this situation, it's not who I am, I'm thoroughly devastated," the employee told Newstalk ZB political editor Barry Soper.
"I would like to be able to return to work to clear my name and I expect, at the very least, an apology from the Speaker for labelling me as a rapist which I most certainly am not."
Last week saw the release of the Francis review, which showed a toxic culture of bullying and harassment at Parliament.
The following day, Mallard said in a Radio NZ interview that his reading of the report was that a man was responsible for three serious sexual assaults mentioned in the review, and that he believed the man was still working at Parliament.
He added that the incidents were tantamount to rape and, when the man was stood down, that a threat to safety had been removed from the premises.
Mallard has refused multiple interview requests.
The Francis review spoke to people about their experiences in Parliament, but did not speak to anyone accused of bullying, harassment or sexual assault.
New Zealand First leader Winston Peters has said that the review should have talked to people accused of wrongdoing.
"If there are going to be those sort of allegations, then they need to be checked out from the point of view of the person [who] was the subject of the allegation. Otherwise we are all caught in this allegation, and that's not fair, and it's not our system of justice," Peters told Magic Radio.
"That is why New Zealand First put our own report in on her report to make it very clear that we were not going to wear any of those allegations because we have the lowest attrition rate of any party in Parliament by miles."
Bennett said she did not know any details of the case and could not say whether the man should have been stood down.
But she slammed Mallard for his "reckless" comments.
"There has been poor judgment and I think it's at the serious end ... It was extraordinary.
"He has caused a whole lot of problems, and quite frankly, I question his judgment on this."
Bennett took part in meetings with Mallard following widespread concern about Mallard's rape comments, and she said Mallard appeared to know nothing more than what was in the Francis report.
"What concerned us a lot was that he was then, in our mind, adding two and two together and coming up with 10.
"I certainly challenged him on the use of the language of rape. He certainly stood by that. I just didn't know anything further and he couldn't verify whether it had been or not.
"To then recognise that that might not actually have been what had happened, for me, was really alarming."
She said the man's claims that he was investigated, and exonerated, raised further questions that Mallard had the responsibility to answer.
"I don't think you get to hide behind an employment matter at this point. He has made comments in the past, very recently, and the public and certainly this Parliament has a right to have answers."
The parliamentary worker said that his family was dumbfounded and the harassment claims were unsubstantiated.
He claimed the three allegations were related to hugging a colleague, complimenting another colleague on her hair, and kissing another on her cheek as he said goodbye to her after she visited him and his wife for tea.
However, the first complainant alleged he hugged her from behind, pushing his groin up against her, and that he was staring at the breasts of the woman whose hair he complimented. He believes the third complainant was put up to lodging the complaint by someone else.