In truth, Williams should not have to justify his worth at the majors. No caddie has won more than the 43-year-old - 14 - and no caddie has looked more assured in doing so. The problem is that in 13 of these, Tiger Woods happened to be swinging the clubs, so the "Williams has won" bit has been ridiculed by punters who believe they, too, could have lugged around the Nikes with similar effect. And this perception has been only lent further hilarity by some of Williams's self-congratulatory acts and announcements.
Williams has even written a book, for goodness sake. Out of the Rough is published in the United Kingdom this week, having already caused a reaction and yes, controversy, on its prior releases around the world.
Ostensibly it charts Williams's days caddying for five-time Open champion Peter Thomson as a 13-year-old, all the way through his trysts with, among many others, Iain Baker-Finch, Greg Norman, Ray Floyd and latterly Scott. Yet, of course, it is his decade and a half with Woods that provides the juicy parts and is why he received the advance in the first place.
Woods fired Williams in 2011 and the adjective "vindictive" appeared in many of the reviews, most of which concentrated on Williams's claims that Woods treated him "like a slave".
Apart from the racial overtones, the fact he was at one time reported to be New Zealand's highest-paid sportsman because of his employment with Woods inevitably led to yet more mockery on social media. "Not quite Roots, is it?" cracked one Twitter user. Williams, however, is unrepentant.
"The book has done great and been received well, with a few exceptions in the US," he said. "But then, you expect them to pick up on one word and distort the rest of it. The 'slave' word should not have appeared, should have been picked up by myself or whoever in the proof-reading stage, but the thing is, where I'm from, 'slave' does not have the same connotations as it does in the States, because we never had slavery."
And the feeling of being put-upon, despite collecting millions in the process? "Look I was just trying to show what it was like all these years working for Tiger," he said. "I was called many things, 'a bully', 'rude', 'arrogant'. But I would have challenged anyone to have spent a tournament week with me and not to have understood what it was like working in that circus. I was just doing my job, trying to give Tiger the best chance of winning with all that craziness going around him. If they saw that, then people would not be so quick to judge purely on what they witnessed at any one moment."
Williams has always maintained he knew nothing about the sex scandal going on like a ticking time-bomb in the background and his autobiography's insight into their largely emotion-free relationship allows the reader to believe it is eminently possible.
Williams clearly feels aggrieved that Team Tiger failed to do more to emphasise his disassociation and it is easy to see why the parting was far from amicable. In one stunning segment, Williams recounts how, when shaking Woods by the hand on the 18th at Muirfield at the 2013 Open, he held him close and told him exactly what he thought. We in the media had all believed they were words of reconciliation.
But by then, Williams had "won" the Masters - his fourth - with Scott and he probably felt in a strong enough position to deliver his reprimand. However, it was not as if the spotlight had left Williams when Woods had left him - anything but. Just a month after receiving his P45, he was with Scott when he lifted the WGC Bridgestone Invitational.
Williams stood on the 18th green at Akron and delivered an infamous victory speech. "I've caddied for 33 years - 145 wins now - and that's the best win I've ever had," he said. Cue more merciless ribbing.
"I say what I think. Too many say what others want them to hear and I've never been able to do that. But that's the only thing in my career I really regret. It was inappropriate, I shouldn't have taken the attention away from Adam. If I had the chance, that's the one instant I'd go back and change."
Not calling Phil Mickelson "a p---" on stage at an awards bash, or, most notoriously, saying on stage into a microphone, that he had been so happy at the Bridgestone because "my aim was to shove it right up that black a----."? Certainly not the latter.
"It was a caddies' awards do, a bun fight and so many things were said that night and what I said was definitely not the worst," he said. "The tournament director had earlier got up and swore his mouth off. That was ridiculous the furore created by that and was again the result of taking it all out of its context."
Scott stuck with Williams through that and, two years later at Augusta, was rewarded with his loyalty.
It is absurd and frankly wrong to suggest Williams did not have a sizeable influence on his major winners because there have been many instances where Woods and Scott have hailed his contributions. And in the twilight of a caddying career unlike any other before, he senses more glory. "I've told Adam over and over that you need to win multiple majors to be considered one of the top performers," Williams said. And, with 14, he should know.
The Daily Telegraph