"And it's not simply the fact that it's cost a tremendous amount of money but actually, it's cost a former staff member of Parliamentary Service their job and their reputation."
She said Mallard's actions along with not apologising earlier were "deplorable" for a Speaker - "who sets the standard of behaviour for MPs in Parliament".
He has now apologised and the Prime Minister Jacinda Ardern has acknowledged his "mistake".
In a statement released on Tuesday last week, Mallard said it was "incorrect" of him to suggest the man had been accused of rape "as that term is defined in the Crimes Act 1961".
Mallard had provided a personal apology to the man for the "distress and humiliation" caused to the worker and his family, the statement said.
But Collins said that was not enough.
"There is no accountability at the moment except for the victim of the situation."
She said National's view was: "Trevor Mallard as the Speaker of Parliament sets the behaviour and tone of Parliament, he has taken up the role of saying he was going to stop bullying and bad behaviour and here we have an example where his behaviour and his failure to apologise far earlier has destroyed a staff member's career and their reputation and their livelihood.
"For the Prime Minister to back him is unconscionable."
However, she said National would consider treating the Speaker like other ministers who get the privilege of their legal fees being covered by taxes.
It was a "huge step for the National Party to come out and to say we have no confidence in Trevor Mallard as the Speaker of Parliament".
"We are not going to let this drop."
Collins said Ardern talked about kindness, "but what about kindness and accountability to the victim here?"
