Green co-leader James Shaw had demanded Mr Key apologise for his claim that Greenpeace and Amnesty were implicated in the Panama Papers, when the charities had been used without their knowledge in sham trusts. Mr Little said Mr Key had lost control.
"It's pretty shameful when the Prime Minister gets kicked out. The one person who we do need to know keeps his cool in Parliament is the Prime Minister. He lost it. He's on the back foot on this issue, so his point of attack now is to distract, throw various dead cats on the table so you start talking about that and when that doesn't work, start demeaning people."
Mr Shaw said it was important the public saw Mr Key was held to account for his actions. "He got himself chucked out because he could not handle it. He got angry and he got himself thrown out of the Chamber."
Deputy Prime Minister Bill English, who was left to field a question to the PM from NZ First's Ron Mark, defended Mr Key, saying his lapse was "accidental".
"He was simply addressing the Greens vigorously and he just didn't see the Speaker stand up. If he had seen it, he would have sat down because he was warned about it yesterday."
He did not believe it was a bad look for the public. "I think they know Parliament is a pretty robust place for debate. I don't think they'll be overly concerned about it."
He said he had seen far more disorderly Question Time sessions than today's.
Mr Shaw and Mr Little fronted media together after Question Time, saying it was important for the Opposition to take a joint stand holding Mr Key to account for his behaviour.
The last time a Prime Minister was told to leave was Helen Clark in June 2005. Then National Party leader Don Brash was ejected at the same time. Neither Mr Little or Mr Shaw have ever been ejected.
Minister Nick Smith said he had been ejected several times himself "so I would be the last to judge".
Mr Key has refused to apologise for using those examples, despite discovering the charities' names were used as part of a scam by people trying to hide the real identities behind trusts and benefit from more favourable tax treatment of charities.
The Prime Minister had raised the charities' names yesterday saying it showed that not everybody named in the papers was necessarily guilty by insinuation.
Greenpeace and Amnesty had both asked for apologies, but Mr Key said he was factually correct to say that they were named in the documents so he did not need to apologise.
He said PR maven Deborah Pead was another who was inadvertently and innocently caught up in the release of the papers.
Although Mr Carter had turned off Mr Key's microphone and repeatedly told Mr Key to sit down, Mr Key said later he had not realised Mr Carter was on his feet. "I was in the middle of an answer. I was talking to James Shaw." He said it was the first time he had been kicked out since becoming Prime Minister. "The Speaker makes the call."
