Tuilaepa said officials had blacklisted some villages, but he refused to identify them.
Issues they had dealt with included workers turning up drunk, damaging company vehicles and criminal offending that included sexual assault.
The Immigration NZ manager of the scheme, Matt Hoskin, said many RSE employers were happy with workers. It was only a fraction of those who had behaved badly over the years.
"The vast majority of RSE workers are of good character and fulfil their contracts without issues," he said.
"Only a very small minority are deported each year."
Mr Hoskin said the agency worked closely with employers, police and members of the Pasifika community when investigating reported criminal incidents.
Over the years they had introduced workshops ensuring workers were properly informed of the rules before leaving the islands and then again when they arrived here.
Samoan community leader and Mangere MP Su'a William Sio said he understood Tuilaepa's stance, as honour, and dishonour, was a huge part of Samoan culture.
Mr Sio said: "From a Samoan perspective, they value this scheme and what it can do, and has done, for families.
"When you get workers who get drunk or do something stupid, it reflects badly on their family back home, their village and their country."
The chairman of the Marlborough Viticultural Contracting Association, Ross Young, said adjusting to Kiwi culture was difficult for many Pacific workers.
"There's a lot of pressure on them. There are a lot of opportunities or temptations."