Newshub reported on 15 May that a source had told it the remark was "stupid little girl" but it could not be heard on Parliament TV.
Brownlee said if Mallard had passed on such information it was unacceptable.
No National MPs had heard the comment in question and all denied saying it. "If the source is the Speaker that is unacceptable. The Speaker cannot go briefing the media against the Opposition."
Brownlee has asked for an explanation by 2pm Thursday and said National would decide what other steps it would take after considering the response.
"We are not rushing to any position like [a motion of no confidence.]
"All Speakers at times get Parliament a little bit aggravated. It's a very difficult job. But when a Speaker gets to a position where it's abundantly clear that perhaps their neutrality is not as strong as it should be then I think that's a difficult situation."
It came on the same day National's deputy leader Paula Bennett walked out of Parliament after the Speaker docked five questions from National because he took offence at a comment made by Brownlee while Bennett was asking questions of the PM.
Bennett later said she had done so out of frustration at Mallard's system of docking questions when he was displeased with an MP because it was arbitrary and unpredictable and made it hard for the Opposition to do its job.
In his letter, Brownlee said National was concerned about Mallard's running of Parliament.
"Your arbitrary taking of questions from the Opposition and the disruption to proceedings are seriously limiting our ability to do our job of holding the Government to account."
Brownlee told media that system effectively meant the Speaker was determining the level of accountability the Government faced. "That's not acceptable."
National are not the only ones who have been upset by the Speaker. Police Minister Stuart Nash objected when he was likened to a "naughty child" by Mallard, another episode Brownlee referred to in his letter. And NZ First MP Shane Jones objected when the Speaker docked three questions for laughing.
Read Gerry Brownlee's letter below