"The Kairakau, Mangakuri and Pourere beach residents are right behind us. We don't want them out of there - we just want them to move out a little bit."
He said he witnessed them trawling off the Hinemahanga rocks, about 1km off Kairakau Beach, on Friday afternoon and Saturday morning last weekend.
"I only went to mow the bloody lawns and there he was - it was the closest I've ever seen them in there.
"We went out last year and approached one of them. We weren't rude or anything and politely said, 'Would you mind giving us a bit more space?'
"His exact words were, 'If I could trawl any closer without damaging my gear I would, so **** off and leave me alone'."
Mr Grey approached the Waipukurau police over the matter in case the situation boiled over.
"I asked them if they could go and approach them because of the tempers out there - the residents of the three beaches - are a bit frayed and you don't want anything stupid to happen like somebody going off their rocker.
"It is something that should be resolved amicably - we feel we are not getting a fair go."
Local Department of Conservation ranger Rod Hansen said the fishing boats were entitled to come in close, so long as they stayed outside the Te Angiangi Marine Reserve between Aramoana and Blackhead Beach.
Hawke's Bay Seafoods owner/operator fisherman Nino D'Esposito said his company fished the coast but there was "no way" they would come in close.
"We don't do that," he said.
"The only boat that would fish in close to the rocks would be a crayfish boat.
"Our boats are too big to go in close."
He said even operators with smaller boats wouldn't risk it.
"You would lose your gear on the rocks or end up beaching it."