"I do have the cheek because I am in the Speaker's chair, I am being very tolerant and I'm asking the member to come and debate here … the matter of public urgent debate."
At the end of his speech, Peters called Tolley's comments a "darn disgrace".
She said: "On that note, I'll ask the member to leave the House."
Speaking to the Herald on his way out of the House, Peters called the Deputy Speaker's decision "disgraceful".
"The Acting Speaker decided that the parameters of the debate did not include the key elements of the defence of the minister's [Iain Lees-Galloway] actions, and tried to rule me out.
"I told her I was not going to be brow-beaten by that sort of behaviour."
It was the first time Peters had been kicked out of the House "for a long time", he said.
Peters said he was going to "utterly ignore that behaviour" from Tolley and "get the facts out there".
Although this was the first time since becoming a minister in this Government he had been kicked out of the House, Peters had a reputation for testing the Speaker.
He has been thrown out numerous times over his lengthy political career.