Gillard's Transport Minister Anthony Albanese was sworn in as Rudd's deputy while former Immigration Minister Chris Bowen was appointed treasurer, with other senior appointments yet to be made.
Labor's popularity has tanked under Gillard but Rudd, who ended a decade of conservative rule with a landslide 2007 election win, remains popular with voters and his elevation is expected to give the party a significant boost.
Analysts are tipping he will bring the election forward to August 24 in order to capitalise on an expected surge in Labor's popularity, although Albanese said a "proper discussion" was needed first.
"The party will unite behind Kevin Rudd's leadership," the new deputy said.
Rudd could face his first test on Thursday if the opposition move a parliamentary motion of no confidence in the minority Labor government, which is dependent of the support of independents.
There are seven crossbenchers in the House of Representatives and Rudd needs to secure the support of the majority, which he should achieve with at least five expected to get behind him.
The embattled Gillard called the leadership ballot after a day of internal party manoeuvring in favour of her arch-rival.
Rudd won the vote of Labor lawmakers 57 to 45 _ the third time since the 2010 election that Gillard's hold on power was tested.
The charismatic Rudd himself launched an unsuccessful challenge in early 2012 while foreign minister, but was routed 71 votes to 31 and was forced onto the backbenches.
Then in March this year, Labor elder statesman Simon Crean made an abortive attempt to reinstall Rudd, who refused and vowed never to challenge Gillard again.
In his pre-caucus pitch Wednesday Rudd said that he could no longer stand by and watch Abbott elected to office.
"I recognise that Mr Abbott is a man steeped in the power of negative politics and he is formidable at negative politics," he said.
"But I see no evidence of a real positive plan for our country's future."
Rudd promised a better relationship with the business community and appealed to younger voters, with his return expected to see Abbott reset his campaign approach for a more popular opponent.
Abbott has slammed three years of "back room deals and backstabbing" within Labor ranks and suggested Rudd was more interested in ego than the Australian people.
"Yes, he's had his revenge, but is this all about Kevin Rudd's ego or is it about the Australian people?" he said on Fairfax radio.
"That's what we'll find out in the next day or so."
The conservative leader had few compliments about Gillard, who famously accused him of a being a misogynist and will sit on the backbenches until retiring at the election.
"I have a feeling that history will judge her pretty harshly because I think it was a poor prime ministership," he said.
- AFP
