"The house was up a very long driveway of more than 100 metres, which made it difficult for crews to get access."
Mr Pilgrim said the house, on a steep cliff overlooking the Hauraki Gulf, was partly propped up on poles that the flames were beginning to damage, putting the structure at risk of collapsing.
Firefighters advanced on the property with great caution.
Fire crews managed to save much of the property but there was extensive damage.
No one was inured in the fire.
"The owners were home, but they got out and they stayed out, which helped reduce the risk of injury," Mr Pilgrim said.
The blaze did not appear to be suspicious.
A fire crew remained on standby this morning, before fire investigator arrived later today to establish a cause.
Last year, the house had a capital valuation of $3,625,000.
Mr Ebert, now in real estate, won his gold at the Games in Christchurch in 1974 in the men's middleweight division. Four years earlier, at Edinburgh, he won silver. He also competed at the 1972 Summer Olympics in Munich.