The couple had been fishing slightly closer to the man when they heard him calling for help from the water.
"As far as I know, they just saw him out in the water calling for help."
But by the time the friends climbed up the rocks to see the man, he had disappeared.
One of the people on the rocks dialled 111.
"It's terrible, absolutely terrible," the fisherman said.
"To be honest I have no idea how he got in the water.
"None actually saw it happened," he said.
"I have no idea where he fell from. No one will ever know - the only person who will ever know is him unfortunately."
Waikato police Search and Rescue Sergeant Vince Ranger confirmed the search would continue early Wednesday morning and would target the low-tide area around Papanui Pt and both north and south of the Ruapuke coastline.
Ranger said the rough weather conditions would continue to be a factor in the search, but it would not stop them from looking for the man.
A team of four carried out a land search today.
Ranger said police had been working with the family. "They haven't been out there physically searching because they have other things to deal with."
Police are not releasing the man's name at this stage.