Ben Rhodes, who spent eight years in the White House serving as Obama's deputy national security adviser and foreign policy speechwriter, knew more about his boss than almost anyone.
In an interview with The Australian, promoting his new book The World as It Is, Rhodes admitted Obama had a "very difficult" relationship with Abbott due to their opposing world views.
He said he found his successor Turnbull, who he described as part of "the mainstream of Centre-Right" leaders, more palatable.
Of the four prime ministers, Obama arguably had the best relationship with Ms Gillard.
They immediately struck up a rapport when they first met in the White House back in 2011. He described her as a "quick study", praising her for getting her carbon price through parliament.
Political commentator Michelle Grattan noted in the Sydney Morning Herald that, while John Howard and George W. Bush had a more solemn, "emotionally based" relationship, Gillard's and Obama's was one of "professional easiness".
In photos they were genuinely physically affectionate, and they're said to still be in touch.
Obama also enjoyed an amicable relationship with former Labor leader Kevin Rudd, with Grattan noting the two "connected at an intellectual level".
She also said Obama had "considerable interest in (Rudd's) China expertise".
President Barack Obama signs the visitors book inside the Parliament House as Australian Prime Minister Julia Gillard looks on in 2011. Photo / Getty Images
Former Liberal prime minister Turnbull also enjoyed a spot near the top of Obama's besties list.
Just six months into Turnbull's leadership, The Atlantic magazine published an article ranking world leaders "on a continuum reflecting the state of their relations with Obama".
Turnbull came in second place for world leaders, behind only German Chancellor Angela Merkel.
"Obama has apparently formed a good impression of the new Australian prime minister," the magazine said, quoting a White House official who said: "Our allies all give us headaches, except for Australia.
We don't know all the details about why Obama didn't get along with Abbott.
news.com.au scoured Rhodes' book for clues, but other than a fleeting reference to a US military base in Darwin, Australia barely received a mention.
But there's plenty of grounds for speculation.
President Barack Obama with Prime Minister Turnbull in 2016. Photo / Getty Images
It may have been their opposing views on climate change. Grattan noted that Obama admired — "even envied" — Gillard for getting the carbon tax through Parliament during her time as leader, saying he would have liked to legislate a cap-and-trade scheme, but was restricted by America's political system.
Abbott, on the other hand, railed against the carbon tax as opposition leader, more recently gave a speech urging Australia to abandon the Paris climate agreement, and once described climate science as "absolute crap".
Or perhaps it was the issue of marriage equality. Obama was dubbed America's "First Gay President" by Newsweek for his decision to openly support same-sex marriage, which was then passed under his leadership.
Abbott has long been a staunch opponent of it, having led the charge for the "No" vote in Australia last year, and — during his time as Opposition Leader — saying he would "probably feel a bit threatened … as most people do" when asked how he felt about homosexuality during a 60 Minutes interview.
Look, it could have been anything. Alas, we may never know.
So instead, we'll just leave this photo here and slink away.
