"So I thought I'd just check the hole and I walked up, took the flag out and there it was sitting in the hole," he says with a grin, regarding it as a miracle.
He should know because in his profession he often witnessed cancer patients make complete recovery without any surgery.
"It's a miracle because God moves in."
The Grayson saga has another twist. His first hole-in-one, an albatross, came 70 years ago, when, as a 22-year-old Otago University medical student, he aced the No14 par-4 hole, then called Roy's, at the Otago Golf Club (Balmacewen) course with a driver.
"It was a straight drive down and had a creek on the left-hand side," says Grayson of the hole that has now metamorphosed to a white-tee 220m dog-leg hole with a sloping fairway leading to an elevated green amid water hazards and trees.
The contemporary Roy's hole is now a No17 par-4 that the club website describes as "a distant birdie opportunity".
"Again, I didn't see it going into the hole," he says of that day when he was playing in the qualifying rounds of the club championship.
"There were players waiting to hit off the No15 hole [mound] and they started jumping up and down so at least I had some witnesses that day."
He recalls it wasn't such a big deal those days for a varsity student making ends meet.
"In those days you had to shout everyone at the bar and golf clubs weren't insured," he says, reflecting on how he furiously cycled home to ask his father, the late Harry Grayson, a good golfer, if he could spare some cash.
"He gave me a five-pound note and, in those days, six pence gave you a glass of beer and you could get a whiskey for a shilling."
For the benefit of the post-imperial measurement generation, the then three-handicapper was able to buy everyone drinks and take some change home because a pound had 12 pence in a shilling and 20 shillings, or 240 pence.
His father was a former Standard Insurance Company employee and, ironically, these days, clubs take out insurance policies to cover such shouts.
Grayson says golf was a great form of relaxation "to get away from the old medical sessions" for the steady-hands brigade.
Nowadays, he likes to get out of the house three to four times a week to the Bridge Pa golf course, especially after "losing my wife, Mardi, in February".
"We used to play golf all around the world," says Grayson, who was born in New Plymouth but moved to Dunedin as a youngster when his father got a job transfer in 1928.
Peter Grayson and Mardi settled in Hawke's Bay in 2006 after she was diagnosed with Alzheimer's.
Grayson relishes his time with "a nice group of elderly members who play on Wednesday afternoons and always make me feel welcome but their numbers are diminishing".
He rates the home of the late Stuart "Emperor" Jones a "lovely course".
Grayson has driven the No13 par-3 hole for two and believes the par-4 No14, where he strives to get on the green, "a great hole that would grace any golf course".
"You take two great shots to a sleeping green so if you don't get the approach right the ball will roll to the back."
An avid collector of golfing memorabilia, including hickory shaft clubs from second-hand shops, he puts his aces down to chance.
"I just got lucky because how else do you explain very good golfers who never get a hole in one."
However, further investigation suggests there is an innate ability for the sport among the Graysons.
Son Anthony, a financial consultant in Auckland, has aced two holes in one while another son, David Grayson, a surgeon in Auckland, was a scratchie.
The senior Grayson, who twice a week tames the Bridge Pa fairways with fellow superannuitant son, Rick, a retired lawyer, didn't tell anyone about last Sunday's feat except close family members.
Rick sunk a hole-in-one at Royal Wellington Golf Club when he used to work there.
Someone posted it on Facebook and now the story is doing its rounds around the world, including son Timothy, a vicar in Baltimore, and daughter Virginia, a 2008 Dobell Prize winner for art, in Melbourne.
"I can now die content," says Grayson who will recount the latest hole-in-one over a whiskey or two with his children when they visit him over the holidays.
