Mallard also threatened to name National MP David Bennett when he initially hesitated to apologise for a remark he made about one of the Speaker's rulings.
That is a more serious sanction than being removed from the House, as it can lead to an MP being suspended and losing their voting rights.
Mallard also made National MP Matt King apologise to him for making a "barnyard noise".
After Brownlee objected, Mallard said King should apologise for his "guffaw".
National yesterday accused Mallard of feeding information to the media about one of its MPs calling Prime Minister Jacinda Ardern a "silly little girl" - an allegation which later got international coverage.
Brownlee gave Mallard until 2pm today to respond to National's complaint.
Brownlee raised a point of order about the issue at the beginning of Question Time today, but the Speaker did not respond to it.
Outside the House today, Brownlee said it was clear that the only person who could have told the media what the comment was Mallard.
"He has not responded to that and we are disappointed that he hasn't done that."
Brownlee said he hoped Mallard would use the next two weeks while Parliament is not sitting to consider whether he would address National's complaint.
"If that is not the case then quite clearly we would pursue what we can by way of standing order expressing no confidence in him, but recognising that the Government of course will support him and so that motion would be lost. It's in the Government's interests that he remains in the chair as Speaker and I think they'd therefore be quite tolerant of some of his inappropriate behaviour," Brownlee said.
Mallard yesterday confirmed to the Herald he did tell a journalist he had heard that comment. "I was asked by a journalist what the comment was. I answered."
Brownlee said Mallard's practice of giving and taking away supplementary questions as a means of discipline, a practice which has caused some frustration among the Opposition, was absent from today's Question Time.
"We'll take that as meaning that he's accepted that that's an unacceptable way for him to interfere with the process of Question Time, and we'll take that," he said.
Bennett tweeted that her party had discussed a walkout from Parliament as a protest but had decided against it.
"We had discussed it. I knew he would be gunning for me and thought he might throw me out given a chance. We decided against a mass walkout today. Try to build a bridge first."
Speaking alongside Brownlee outside the House, Bennett said National had gone into the debating chamber today in good faith.
"We went in there in good faith today. And then we just had ridiculous rulings that led to some disorder in the House."
Bennett said she was disappointed to have been kicked out of Parliament today and questioned whether Mallard was getting personal.
"He does seem to treat me differently than he does other people in the House."
Mallard declined to comment to the Herald this afternoon.
Ardern, who was in Auckland today, told reporters all sides would find the Speaker either in their favour or not at some point.
On Mallard's practice of giving and taking away supplementary questions, Ardern said all MPs should be prepared to ask additional questions.
"The idea of Question Time often is that you're asking spontaneous questions. I think, actually, politicians are quite used to that," she said.
