The Serjeant-at-Arms is mostly a ceremonial position, but the role is to keep order in the House. The role is essentially that of the Speaker's enforcer.
Late last night, while Parliament was sitting under urgency, Smith was back in the House.
Labour MP Michael Wood raised a point of order, saying Smith was not allowed in the House.
Deputy Speaker of the House, Labour MP Adrian Rurawhe, asked Smith to leave but the National MP refused.
"When you send us to prison, we're told how long [we're there] for," Smith complained.
Earlier in the day, Speaker Trevor Mallard kicked Smith out of the House but did not say when he could return.
Smith said this was not fair and he attempted to clarify with the Speaker's office a number of times as to when he was allowed back in the House.
He said he was only arguing that the rules of the House should be applied fairly and that he was only trying to defend his colleague Nikki Kaye from attacks by Winston Peters.
But Rurawhe was not moved by Smith's defence and called for the Serjeant-at-Arms to remove the unruly MP.
Smith persisted: "I'm asking to be told when I'm allowed to return – it's not an unreasonable thing."
The Serjeant-at-Arms then escorted Smith out of the House.
Wood was not impressed by Smith's behaviour.
"We know that he is feeling under pressure in his seat of Nelson this year, but staggering about in the dead of night, ranting about Nazis and having to be forcibly removed from the chamber is not the way to behave as an MP."
Wood said Smith's behaviour was unlikely to endear him to local voters, who want an MP focused on representing them.