The 64-year-old was arrested following a lengthy investigation by Immigration NZ and NZ Police.
Allegations made by the victims include not being paid for work completed, having their passports taken, and being subjected to physical assaults and threats.
It is claimed his alleged offending had been ongoing since the 1990s, across the Hawke's Bay region.
The victims also allege that their movements were closely monitored and controlled by the man, and there were restrictions on both where they went and who they had contact with.
Immigration NZ assistant general manager Peter Devoy said the arrest was the result of around two years of detailed investigative work by INZ and NZ Police.
"We are absolutely committed to eliminating people trafficking in New Zealand," Devoy said.
"Today's arrest follows an extensive investigation by INZ and NZ Police, which has lasted more than two years."
Lees-Galloway said this action demonstrates why this Government has made cracking down on migrant exploitation one of our top priorities in Immigration.
"This reflects how seriously both of our agencies take these types of allegations, and our commitment to combating Transnational Organised Crimes, including people trafficking.
"There is no place for exploitative practices in New Zealand. Migrant workers have the same employment rights as all other workers."
Lees-Galloway said through the in-depth research the Government will engage with representatives of migrants, businesses, workers, academics and international students to provide independent perspectives and to test ideas.
Devoy said the situation was "significant" but because the matter was before the court, he could not go into any detail of the case, including any instances of threats.
Devoy said some victims have been "under his control" for some years.
He said INZ became aware of this through talking to one of the victims about two years ago.
"This is a new low for New Zealand," Devoy said.
"This is about a labour contractor."
He added that the people running the orchards probably didn't know about the alleged offending.
He said this is the fourth time INZ has charged someone with people trafficking, but it is the first time the police have been involved.
Devoy said she believes "the combination of slavery and the treatment of these people sets [this case] apart."
Foster said the alleged victims of the scheme are spread across New Zealand and Samoa.
INZ has been working with the Samoan police and there would be more working between the two.
"It will be incumbent on us as New Zealand, for us to be more aware what's going on in the labour force," Devoy said.
INZ is working to limit this happening again, he said.
Lees-Galloway was briefed on the case before INZ made it public today.
Detective Inspector Mike Foster, Eastern District Police said there were 10 alleged victims.
He said he didn't know if there are more, and is appealing to the public for more information.
Foster said between 15 and 20 people have been working on this case over the last two years.
The man will be the only one to face charges, Foster said.
Foster said it was an "easy choice" for the police to get involved in the case – the decision for police to get on board was after INZ talked to the first victim.
Foster said the man had allegedly recruited people in Samoa, promising them "well-paid" jobs.
"Information collected during the joint investigation suggests that the man, who was seen as a respected member of his community in Samoa, targeted vulnerable people, who had limited education and literacy," Foster said.
"We have received valuable assistance from the Samoan authorities during our investigation, and I would like to extend our sincere thanks to them on behalf of both NZ Police and INZ."
* Anyone being forced to work here illegally for less than the minimum wage and/or excessive hours can contact INZ or the Labour Inspectorate, where their concerns will be handled in a safe environment.
People can also contact CrimeStoppers anonymously on 0800 555 111.
