"But that Bonecrusher/Waverley Star race changed everything - I was sold on the gallops after that."
Perhaps there is a partially hidden storyline there that Rutherford can use in his new position.
For a period there during the height of his headlining cricketing career Rutherford was well immersed in horse racing as a punter.
Someone penned the line that he was matched in that area only by legendary Australian cricketer Doug Walters.
In his new position it is understandable Rutherford is reluctant to give that much credence, but more likely it's a result of his having spent the last 12 years in the racing/sporting betting industry.
He joined the New Zealand TAB in 2001, quickly working his way to heading the horse racing bookmakers' bench before five years later accepting an offer to work with a huge betting office in Singapore.
"That was a huge step forward in my career and I was very grateful for it."
While there the International Cricket Council sought Rutherford's advice during the time match fixing reared its ugly head.
Someone with Rutherford's experience - he captained New Zealand for three years - who worked in the betting industry at the highest level, was going to have some idea of what was going on.
Some of what Rutherford had to say at the time:
"I would be shocked if a World Cup game was interfered with. The spotlight is at its most extreme and the fixers could find better options for their skulduggery in matches where the focus is less strong."
However, he said the ICC had a lot of work to do going forward. Betting markets, he said, should be closely monitored by experts during games for any suspicious movement of money. "[This could be done] by employing traders from the sports betting industry whose job it would be to watch the markets on a daily basis. They could set up some kind of software that would be applied to the markets that would act as an alert should money be moving oddly - that is, against the natural progression of a match."
Citing the example of football, Rutherford said that the ICC should form information-sharing partnerships with the world's biggest bookies. "When odd betting behaviour is suspected, the network is informed. Often markets will be closed as a result of this information. The network has a direct line through to UEFA (European football's governing body), who are informed prior to a match beginning if any match fixing is suspected.
"At my old job at Singapore Pools, we often heard things in advance from our own network of 'spies'. If something was suspected in football, the match officials would warn the players prior to the game that the match was suspected of being fixed, and that they better not try anything."
He said yesterday: "The underground bookmaking in India is huge."
Rutherford has moved to New Zealand and is living in Cambridge with his South African wife and family.
He says the danger of living in South Africa, where he has spent his last two and a half years in an executive position in horse racing with Tellytrack, is sometimes overplayed, but he's not saying it's safe.
"We love Cambridge and it's lovely to know your daughter can get off the school bus 800m from home and walk home safely.
"And to watch the kids playing their sport is magnificent.
"There is a high degree of security you can employ, but it doesn't always create a guarantee - our next door neighbours were held up at gunpoint last year."
With such a high percentage of racing's revenue emanating from the betting dollar, one of Rutherford's jobs is to attract more patrons to Te Rapa and to somehow guarantee the younger set not only get there, but do not avoid walking to the on-course tote windows and bet instead with the TAB, or worse, with overseas agencies.
He has plenty of experience with the South African system, which bars bets being placed on high-profile sites such as Betfair and the major European betting agencies.
But Rutherford prefers a different approach, going head-to-head with such bookies and offering better odds, better value.
Meantime he has plenty to drag his attention - his 24-year-old test cricketing son Hamish, who made such an impact against England earlier this year.
"To think he was pouring lattes in a Dunedin cafe this time last year," he says.
"It's remarkable what he's achieved."
