Brazilian journalist Alexandre Tortoriello saw Smith being taken through Rio's airport. "The operation was planned in secret," he told Radio New Zealand. "The federal police in Brazil, it's like the American FBI but it's in charge of our borders, so basically they can deport someone who is illegally in the country."
Smith is understood to have boarded a plane to Santiago, Chile, where he will join a connecting flight to Auckland. He is expected to land in New Zealand at 4am tomorrow.
Smith's lawyer Tony Ellis said his client's deportation had taken him by surprise as his client had filed a writ in Brazil earlier this week to delay proceedings.
"I'm surprised he's on his way back today if he's filed a writ of habeas corpus. I'm not surprised that he's been deported, I was somewhat surprised he hadn't been deported," he told Radio New Zealand.
"I do have some concerns that, if he says he's filed a writ of habeas corpus and he's on a plane, that doesn't sound right."
The Brazilian Government had yet to confirm Smith's deportation, Tortoriello said.
Smith's biological father John Traynor said he had been given "prior information" that his son was to be deported. "A judge had about a week ago issued an order that he be deported from Brazil before December 1, so I had prior knowledge that he was going to be back in New Zealand before next Monday," he told Radio New Zealand.
He was pleased Smith was being returned to the country, he said. "It gives closure to other people, to his victims' family. To be perfectly honest they are very much on edge, and my feelings go out to them of course. He will presumably be going back to prison."
Smith was caught in Rio de Janeiro on November 13 a week after fleeing New Zealand while on temporary release from Spring Hill Corrections Facility in South Auckland. He was serving a life sentence for murder and other violent crimes.
READ MORE: Smith fooled judge over tax scam
Smith had arrived in Rio de Janeiro with a large suitcase and a backpack, checked into a youth hostel as James Paul Andrews from Brisbane and joked about finding himself a Brazilian girlfriend.
He had paid $40 for two nights in a bottom bunk in a six-person dorm called the Ipanema room at the Cidade Maravilhosa, and for all intents and purposes was just another tourist in Rio de Janeiro.
But the next morning, a staff member recognised him in a television news report about his escape. When he left to buy food, the employee decided to call the police.
Officers swooped and the international manhunt for Smith was over.
Smith was remanded in custody to the notorious Ary Franco Prison, he was jailed under the name Phillip John Traynor, the name on the passport he travelled to Brazil on.
He was transferred from Ary Franco to the Bandeira Stampa public jail, part of the Gericinó Penitentiary Complex in Bangu in Rio's Western Zone.
No reason was given for the transfer, other than it is a "unit in accordance with his profile".
When he was sentenced to 60 days preventative detention on 12 November, the judge Flavio Roberto de Souza said: "It must be born in mind that, given the deportation request, that he could escape to an unknown location, as he did during his prison sentence in his country of origin."
Smith is facing one charge of escaping from custody, and further charges are likely when he returns.
It was not clear whether anyone will be charged with assisting in his escape. His sister Joanne Smith, one of his release sponsors, picked him up from the jail. Police had raided her Palmerston North home seizing a laptop and other items.
The man who was molested by Smith as a child and who saw his father being murdered by Smith said he could now carry on with his life "without having that fear [of Smith finding him] over my shoulders".
Earlier this week the Government confirmed the terms of reference for an independent inquiry into Smith's escape.
State Services Minister Paula Bennett said the inquiry would look into three aspects of Smith's escape while on a three-day release from Spring Hill Prison.
The draft terms of reference focused on Smith's temporary release from prison and departure to South America, his use of multiple identities to evade authorities, and the adequacy of information sharing between Government agencies.
Mrs Bennett said: "It is imperative the inquiry provide answers to the many questions New Zealanders have around how Mr Smith/Traynor was able to abscond overseas while on temporary release from prison."
Smith escaped New Zealand by renewing a passport in his birth name Traynor and using it flee to Chile and Brazil while on temporary release.
The inquiry would be chaired by QC John Priestley, who Mrs Bennett said had "significant knowledge of the justice and public service systems".
Mr Priestley would be assisted by high-ranking public servant Simon Murdoch, who has previously served as head of the Department of Prime Minister and Cabinet.
Mrs Bennett said the inquiry would report back in the second half of next year.
The inquiry was separate from an internal Department of Corrections inquiry and a multi-agency inquiry led by the Ministry of Justice.
