"It's every person's dream," he said. "I'd always dreamed of going.
"I was just a young boy from Waipuk, who spent a couple of years in Wellington and all of a sudden I'm going to Munich."
Although the Hastings resident had travelled to Germany to represent New Zealand as a track rider, he ended up doing so as a road rider.
During his time there, Mr Oliver stayed in the Munich Olympic Village with his fellow athletes.
It was at the village when, in the second week of the Games, Palestinian terrorist group Black September took 11 Israeli athletes hostage - an event which would become known as the Munich Massacre.
Early that morning - as far as he could recall - Mr Oliver and his teammates had set off on a four-hour ride on a network of roads near the village.
"When we came back there was tanks, and machine guns, and police armed to the teeth," he said.
"We didn't understand what had changed from the time we went training to the time we went back, so we missed all of what was going on.
"It was quite a shock to come back to that," he said.
The initial stand-off between the terrorists and police was reported to have lasted almost 18 hours.
By the end of the following day, after a failed rescue attempt at a nearby airbase, 17 people including the athletes, a German police officer, and five of the terrorists had been killed.
Following the attack, the Games were suspended for 24 hours.
The attack "totally changed the atmosphere" of the Olympic village, Mr Oliver said.
"There's quite a carnival atmosphere in the village and there had been, right until the moment that happened."
Traditionally, when an athlete finished their race they would stay in the village, however soon athletes were being "shipped home" as soon as they finished competing.
Mr Oliver's race was one of the last, but when asked if he felt the attack affected his performance, Mr Oliver said he didn't think so.
"A good athlete is narrow-minded," he said. "You're there for one purpose."
While the village atmosphere had changed, Mr Oliver said the "competition atmosphere was still as hard as ever".
He recalled watching the last day of competition for athletics, when he and 80,000 other spectators turned out.
"It had to go on," he said. "You can't bow to their pressure, you've got to carry on."
However, Mr Oliver said "a lot of our guys didn't go into the closing ceremony because we thought something else might happen, you just don't know".
In 1974, Mr Oliver retired from cycling after missing selection for the Commonwealth Games, and sustaining a knee injury.
However, his involvement with cycling was far from over - he and his wife Brenda returned to Hawke's Bay, and bought the Hub cycling shop in Hastings in 1982.
A life member of the Ramblers cycling group, Mr Oliver still cycles, still runs the Heretaunga St West shop, and when the 2016 Olympics are broadcast, will be watching as much cycling as he can.
His advice to the Hawke's Bay representatives on the Olympic team was: "enjoy it".
"I was training with a friend and he said 'it's a funny thing being an Olympian, you can be anything else, Commonwealth Games, national champion but once you're an Olympian you're an Olympian for life', and it's quite true."
