Mr Peters also told the Speaker during Question Time yesterday that he was sick of Dr Smith's interpreting what a minister has said.
It happened regularly if an Opposition MP believes a minister has not answered the question and Dr Smith explains why he thinks the minister has or has not answered the question.
After one such incident yesterday, after Dr Smith said what he thought Education Minister Hekia Parata had said, Mr Peters said most people in the House were sick and tired of having ministers' statements interpreted and becoming the record of the House.
"We cannot quote them because the words are from you, not from the person who gave them, namely the minister. I am sick and tired of it and I think the House and the public are."
Moving motions of no confidence in a Speaker are rare.
It is unlikely that Mr Peters motion will even make it to the floor of the House, let alone be passed, because it needs the Government agreement to get onto the agenda, or to have no objection from any party.
But Labour has said it will it discuss it at next week's caucus meeting.
Shadow leader of the House Trevor Mallard said that many people believed the Speaker had made an error in his call on Tuesday by calling Mr Key first.
Mr Peters first serious clash with Dr Smith this term occurred soon after the House had paid tribute to three soldiers killed in Afghanistan.
Question time was underway and Mr Peters was asking questions about the sale of the Crafar farms when Dr Smith objected to the banter taking place saying that after the tributes, they were"carrying on like spoilt brats."
Dr Smith is expected to named soon as the next High Commissioner to London.