Ross Reid built the property single-handedly in the 1950s and '60s and invited his son and daughter-in-law to live there with him when Esme died.
He says that in March 2009 he proposed to buy $1.7m of shares and put them in Castleton-Reid's name so his son could sign share transfers while he was away on long trips to Australia.
Castleton-Reid was to hold the shares and any profits on his father's behalf, Reid argued in a hearing before Justice Christine Gordon last year.
In return, Reid said he promised that his son would get the funds on his death because there "were no pockets in a shroud".
Castleton-Reid described this turn of events very differently.
He told the court that in March 2009 his father called him into his bedroom and told him he had an "inheritance".
Reid, according to his son, indicated it was a large amount of money and that he would like to put it into shares and manage it on his son's behalf.
Castleton-Reid said that he spoke to his wife immediately afterwards who asked why the two of them would put all their money into shares.
"Mr Castleton-Reid told her that he thought his father was bored and it would give him something to get involved in. He said that neither he nor his wife had any need for the money at the time, and they both just accepted the situation," Justice Gordon said in her decision.
The share trading account was opened, $1.7m transferred into it, and Reid began trading.
Numerous deposits and withdrawals were made before Castleton-Reid sold the shares in 2010 and stopped his father from accessing the account.
"Mr Castleton-Reid said that he and his wife decided that they needed to take charge of his financial situation and revoke his father's authority to manage his shareholding account, as he said his father's recent activities were becoming increasingly suspect," the judge said.
"Mr Castleton-Reid said his father had been dealing with Mr Castleton-Reid 's inherited property as if it was his own, paying himself more than Mr Castleton-Reid had agreed to out of dividends earned on the shareholding account and using the Castle as security for his debts when he was only registered in his capacity as an executor [of Esme's estate]. He informed his father that he was taking control of his inheritance as he was alarmed at the potential implications of his father's management," Justice Gordon said.
When the pair could not resolve the dispute, Reid issued High Court action against his son.
But in her decision, the judge was not satisfied that Reid had proven his case and ruled that his claim had failed.
"Having determined that there was no arrangement as asserted by Mr Reid, the court is then left with the evidence of Mr Castleton-Reid. I find no reason to reject his evidence that Mr Reid represented to him that he was giving him the money…I am further satisfied that Mr Reid intended to gift the money to Mr Castleton-Reid," Justice Gordon said.
