"They were jumping in front of, behind and alongside the boat. We yelled at them to get out of the road, and they didn't - they thought it was a great joke," Graeme said.
"This is not an isolated incident. We go out regularly, and it's an ongoing problem."
Graeme said they have been sworn at and one of the boat's windows was kicked in. They have laid a complaint of intimidation and wilful damage.
"It's a boat ramp for boaties, it's not a swimming pool," Ray said. "If they want to swim there when there's no boats there, there's no problem."
The two men said the area had been very busy this year, thanks to a very warm summer. There was a sign up warning people not to swim when boats are present, but it had made no difference.
"When you're trying to manoeuvre a boat, you have to concentrate - you can't be worrying about people in the water," Graeme said.
"Something has to be done before somebody gets hurt."
Whanganui police did not answer a request for comment.