But he didn't invest any of it.
Instead, he shuffled the money around various accounts to repay earlier investors and used some of it to pay off his Visa, along with business and personal bills.
From 1991 to 2012, he operated what had all the hallmarks of a Ponzi scheme, the court heard after he pleaded guilty to 34 fraud and theft charges laid by the Serious Fraud Office (SFO).
Of the $2.8m he stole, around $2m has never been recovered, with little chance of it ever returning to his victims, the court heard.
Rachael Reed, counsel for the SFO, described Milne as a "consummate thief and liar" during sentencing today.
She described the crimes as "predatory offending" where Milne enticed victims to give money to him through the promise of high interest rates and security of funds.
It amounted to a "staggering" breach of trust, Ms Reed said.
Judge Jane Farish said the 21 victim impact statements made for "very sad and distressing reading".
One victim impact statement read to the court outlined the sad case of one 69-year old victim.
The man, known only as 'Mr P', had known Milne - a practising lawyer since 1960 - since around 1996 and considered him as a friend.
During one meeting, Milne told if if he ever had "any spare money" he could get him a good interest rate on it.
Milne took the money and they never saw it again.
His wife lost her life savings of $13,000 which she had set aside for her funeral.
The man said he lost $1.14m.
The couple are now struggling to enjoy retirement, living off their pensions, and battling ill health.
Many of the victims were elderly, vulnerable and "an easy target", the court heard.
One victim told Milne that she had lost a lost of money in finance companies and that she couldn't afford to lose any more.
However, he still took her money, spent it, and "constantly assured her it was safe, when it was gone", the court heard.
Another victim told the court that it "breaks her heart" to have seen her father live so carefully in order to save money for his family only to be ripped off by Milne.
Defence counsel Karen Feltham said "since he was a boy, he's borrowed and repaid".
"He thought, to start off with, that he was able to do so," she said.
Ms Feltham said the scam was not designed to fund an extravagant lifestyle.
"He's gained nothing with respect financially from these transactions, as opposed to (former Bridgecorp boss Rod) Petricevic and others who lived a very expensive lifestyle, this man hasn't," she said.
She asked Judge Farish take into account his age, saying that he was "not likely to survive a huge amount" of time behind bars, adding that it would be a "significant hardship to him".
But Judge Farish said she was not prepared to give him a discount due to his age, given that he was being given credit for his previous good character, and his albeit late guilty plea on the morning his trial was due to begin.
She jailed him for a total of eight years and one month.
Judge Farish concluded that a minimum period of imprisonment, as called for by the SFO, was appropriate in this case.
"Thank you, your honour," Milne said as he was being led into custody.
A complaint to the SFO was received from the New Zealand Law Society Otago Standards Committee regarding Milne's activities around client funds in June 2012.
In April last year, a Lawyers and Conveyancers Disciplinary Tribunal decided to strike him off its roll of barristers and solicitors.
New Zealand Law Society President Chris Moore said the gross abuse of trust was "reprehensible".
In 2012, Dunedin solicitor Alistair Paterson launched an "affected persons' register" to collate the names and extent of loans made to Milne.
Milne was adjudged bankrupt in the High Court at Christchurch in November 2012.
He had his own Dunedin legal practice from 1960 until it was bought out by another local law firm, Craig Paddon Law, in 2008.
He worked in their Christchurch office from 2008, but left in June last year.
SFO Director Julie Read today emphasised that despite the successful prosecution, the agency remains concerned for Milne's victims "for whom it is too late for many to recover from the impact of his dishonesty".
