The book, which shares the extraordinary tales of the radio and TV personality's 50-year career in media, shows Packer was essentially the captain of the Australian cricket team whenever he wanted to be.
He just had to pick up the phone.
This was the Australian test captaincy — the highest office in Australia, above even the Prime Minister.
Not above Packer, apparently.
In this extract from his new book, Carlton details the bizarre way he discovered Packer's iron hold on the Aussie cricket team during the mid-1980s.
"Like most rich and successful business figures, he (Packer) could also radiate charm when he felt like it," Carlton writes.
"Many years later, when I was working in radio, he heard that I was with my young family and some friends and their kids at Palm Beach (on Sydney's Northern Beaches), on the sand just below his big pink beach house on Ocean Road.
"It was summer, Saturday afternoon. He sent down an invitation — well a command really — to come up and watch the cricket on television, and he welcomed us up on the front deck with his enormous frame swathed in a voluminous Indonesian sarong and naked from the waist up. It was an imposing sight."
Carlton described the sprawling pink Packer property — which is still owned by the Packer family — as "one of the most expensive pieces of real estate north of the Harbour Bridge".
"The butler swept up the kids and took them out the back to the jacuzzi, where they were plied with ice cream," Carlton continues.
"We sprawled on lounges while James Packer, still a teenager, bustled back and forth with beers and Coke for his father (who famously didn't drink alcohol).
"Kerry was genial and Ros, his wife, chatted with the women. Most startling of all, Kerry seemed to have a direct line to the Australia dressing room at the Sydney Cricket Ground. Australia was in the field.
"Every so often he would pick up the phone and snap an order: 'Put (fast-bowler Craig) McDermott on... Give Waugh a few overs'.
"A few minutes later the bowlers would obediently change. When the time came to leave, we whispered to the kids to thank their host. As one they turned to the butler, 'Thankyou Mr Packer,' they chorused. Kerry snorted with a laugh."
Carlton said Packer was so nonchalant with the power he wielded over the Australian cricket team, it took a while for guests to realise what they were witness to.
"Every now and then he'd pick up the phone and re-set the field," Carlton said.
"He never explained it. He never said, 'I'll just give the dressing room a ring'."
They just happened to do exactly what he was saying.
Allan Border and Kim Hughes were Australia's Test captains during the mid-1980s and it seems likely both of them were at some point taking orders from Packer's hotline through to the Australian dressing room.
Being overruled by Packer is nothing new in Australian cricket.
In his first meeting with former captain Ian Chappell as they plotted their World Series Cricket revolution, one of the first things Packer did was overrule Chappell on the selection of spinner Ashley Mallett.
"I'm not paying that bloody straight breaker," Packer told Chappell, as revealed by The Daily Telegraph in 2012.
When Chappell tried to remind the former Channel 9 supremo that he was no longer captain of the Australian team and shouldn't be involved in selection discussions, Packer forever changed the landscape of the Australian cricket captaincy.
"What do you think this is, a bloody democracy?" he exploded.
"I pay the bills, I pick the captain. You're the adjectival captain."
His influence over the captaincy, it seems, never wavered.
