"We are very upset with Sophie Mirabella," Howard's son Mervyn told the Age.
"We believe she adopted a deliberate tactic to split our family. And it happened immediately after my father told me that he had made a financial contribution to her by cashing in a super entitlement to help them buy their house in Wangaratta."
Mirabella replied in a brief statement: "It is less than 12 days since Dr Howard was buried and no application has been made to obtain grant of probate of his will. There is no public interest in denigrating the memory of a private man."
English-born Howard arrived in Australia in 1958, teaching at Queensland, Adelaide and Melbourne universities and for three years was general counsel to the commonwealth Attorney-General.
Former barrister Mirabella, nee Panopoulos, was born in Melbourne in 1968, a fervent Liberal hardliner who has attacked even moderate colleagues as "political terrorists" for doubting former Prime Minister John Howard's policies on asylum seekers.
In 1994 Mirabella, then still Panopoulos, met Howard, and they became live-in lovers the following year.
Although accepted by Howard's children, British-based Mervyn and his sister Lesley, the relationship was kept secret from the Panopoulos family even to the extent of the young solicitor renting a flat as a cover.
In 2000 Sophie Panopoulos moved to Wangaratta to run for the seat of Indi. The Age said Howard lent her his car and paid A$100,000 for campaign expenses. The newspaper said that later, although by then they were no longer lovers, Howard was believed to have given as much as A$100,000 to help newlyweds Sophie and Greg Mirabella buy a home.
Howard and Mirabella's relationship had withered as Howard, by then 74, found the drive between Melbourne and Wangaratta increasingly difficult.
Although the pair kept in touch, the Age said the socially awkward Howard retreated into a tiny circle and by 2004 was beginning to show the early signs of Alzheimers: his normally neat house was filthy, his memory was slipping, he could no longer use his TV or stereo, and he feared at times that Mervyn and Lesley were part of a conspiracy against him. Mervyn and Lesley paid for the house to be cleaned, and for a regular cleaner.
The Age said that during Mervyn's family visit at Christmas 2006, he and Lesley became even more worried about their father and whether Mirabella was influencing him, especially when he said he cashed in one of his pensions to lend her money for a house.
When Mervyn questioned Mirabella he was attacked in return as "incorrect and insulting". Howard severed contacts with his children.
The Age says that on March 13, 2007, Howard sent a typed letter looking to stop all contact with Mervyn. "Any further such correspondence to Sophie or me will be ignored. I wish never to hear from you again," it said.
Mervyn Howard told the Age the letter left him devastated.
Howard later vanished from his home, and was only found to be living in a Wangaratta cottage rented by Mirabella when he suffered serious injuries in a fall. Despite the earlier breach, Howard was delighted to see his children, telling them: "I know I didn't want to see you again but I don't remember why."
Howard recovered but was diagnosed with Alzheimers and placed in a nursing home. He died on September 2, aged 83.
It later emerged that Mirabella was both executor and chief beneficiary of his will, but the Age said she had refused to discuss it with Mervyn and Lesley Howard.
Mervyn and Lesley Howard told the Age they were not interested in their father's money.
But they said they did want to be assured their father was treated with honour and dignity after they were locked out of any decision about his care, and if necessary would fight in the courts to be sure that Mirabella "did the right thing".