At the same time, the commissioner notified two grandsons of the Vogels that they would not be offered the property. The grandsons, Tim and Geoff Vogel, had previously asked for the entire property to be given to them.
The Vogels applied for a rehearing of the issue, and an independent lawyer was tasked with looking into the case and filing a report on it.
It was then decided neither the grandsons nor the charities should receive the house, but that it should be sold on the market, with the proceeds going to the Crown.
The grandsons have now taken the matter to court for a judicial review, which was heard in court today.
Fowler argued there had not been consideration of the emotional hardship the Vogels would suffer if the house was not offered to them.
Reading from Tim Vogel's affidavit, Fowler spoke of the family connection to the house, the family associations in the area stretching back 150 years, and the fact that no other private family had ever lived in the house.
"[I] can't think of a single property in New Zealand which can claim this," Fowler said.
In his affidavit, Fowler also said his grandmother had "often remarked she wished the Crown would return the property to the family because it was no longer being used as the prime minister's residence".
The Land Act states the property can be allotted to someone without competition from the public if there were special circumstances and the applicant would suffer hardship by the call for public applications, if it would be equitable to do so.
Fowler said this did not just include financial hardship, but could also mean emotional hardship. He said the independent review of the case did not give the Vogels' emotional hardship correct consideration.
"The Crown has got itself into a right royal tangle on this one," he said.
Co-counsel Richard Moon said there was a "truly unique combination" of family events, history, and associations attached to the property.
"Of course it will cause emotional suffering to see that land in the hands of another party."
He said the decision not to give the property to the Vogels was "not made lawfully".
Lawyer for the commissioner, Rachel Roff, said there was no error of law - and no grounds for judicial review.
"Everything has been considered and carefully addressed," she said.
Roff pointed to the report by the independent lawyer, saying it made mention and gave consideration to emotional hardship but did not find it to be at a high enough threshold for the house to be given to the Vogels.
The judge has reserved his decision.
