Mr Mallard had been riled by Mr Key twice referring to someone being "brighter" than Mr Robertson without reprimand. But when Mr Mallard told Mr Carter to sit down while he was speaking he was thrown out and when Mr Hipkins tried several times to question Mr Carter, he was asked to leave too.
"I have tried to be extremely patient and not asked people to leave the chamber," said Mr Carter, calling Mallard's remarks "indefensible".
In relation to Wednesday's incident he said: "I think in hindsight it was out of order (Mr Key's comment) but a significant reason as to why I didn't ask for that reply to be withdrawn is that it was directed at Grant Robertson and Grant Robertson sat and appeared to take no offence.
"If he had raised it and taken offence, I'm sure the outcome would have been the Prime Minister would have been asked to withdraw that part and maybe even withdraw and apologise."
Asked about whether the Prime Minister had stirred things up, Mr Carter said: "I think the Prime Minister was incredibly unhelpful to the order of the House."
Greens co-leader Russel Norman often criticises Mr Carter in the House for not being like former Speaker Lockwood Smith. Last week he went to see Mr Carter about Question Time.
And this week Labour leader David Shearer went to Mr Carter about the same thing.
Opposition parties are frustrated that Mr Carter is not applying the same method as Dr Smith.
Generally Dr Smith would decide whether a question was "straight" or "political" and if he deemed it a straight question he would not accept a political answer - one that contained a political attack on a party.
That was hugely different from the days when ministers could simply use a word from the question and be deemed to have acceptably "addressed the question", which is the requirement.
Mr Carter has opted for a halfway house. If he believes a minister has not addressed a question adequately, he will allow an MP to repeat it, sometimes several times, and Mr Hipkins has used it to the greatest effect with his questioning over the resignation of Education Secretary Lesley Longstone.
"The reason is he is asking straight questions," said Mr Carter.
Mr Carter said he thought Dr Smith was the best Speaker he had ever seen in action "but I never thought for one minute I would do things exactly as Lockwood did".
"He tended to paraphrase the question as he saw it and paraphrase the answer as he saw it and then draw a conclusion as to whether the answer was adequate enough."
Mr Carter said he attempted to do that for the first couple of days but the result was that some MPs sought to bring the Speaker's comments into a question in the House.
Ultimately if the minister hadn't given a satisfactory answer, it was not the Speaker's responsibility, it was the minister's responsibility.
"At some stage in proceedings you have got to move on and then the Members of Parliament and anybody listening to Parliament will judge the accuracy and ability of that minister."
He accepted that the result had been a lot more political hurly burly had been injected back into Question Time. "It's a political debating chamber. I don't want Question Time to be totally sterile."
Mr Carter has also been criticised for not explaining his rulings well enough - in contrast to Dr Smith who would discuss his rulings at length.
"It's just my nature. I tend to be a person who speaks relatively crisply, sharply, to the point and doesn't elaborate. That's my nature.
"The House will be more difficult between now and the next election that, according to the polls, will be close and you've got an Opposition that has been in opposition now for two terms, and I know from my experience in Opposition it's soul-destroying. It's a dreadful time in anyone's political career so I have no doubt that the tensions and the challenges of being the Speaker will only increase as we approach the next election."
