Two-and-a-half years back, leaks were becoming commonplace, triggering emergency plant shutdowns and evacuations, and the roof was in need of attention.
One winter, Charlton arrived at work to discover a gap had opened up in the building, and rain was falling on the finance team.
About that time, Fonterra called "time-out".
Fonterra has ploughed about $40 million into the company over the last two years to get it back up to scratch, culminating in the upgraded frontage of the building unveiled this week.
First came a revamp for the ammonia system, then the roof. The often intricate machinery used in the icecream making process has also been upgraded to improve safety. While the more visible problems were being attended to, the company's internal systems were also reorganised.
The company had nine legacy information systems to deal with, which made data gathering difficult.Tip Top's internal systems have been reorganised with the help of German systems specialist SAP. At the same time Charlton said Tip Top became more focused - dumping unprofitable lines and concentrating on strong and stable returns.
In an interview at his office, which offers a commanding but somewhat unsettling vista of south-bound traffic hurtling down the motorway towards the viewer, Charlton said the company was focused on achieving steady, incremental, growth.
Tip Top's gross sales are worth more than $200 million a year - a drop in the bucket compared to Fonterra's revenue of about $19.9 billion a year.
From an earnings before interest and tax perspective, the company rates as "mid-range" in the so called fast moving consumer goods (FMCG) market.
In New Zealand, it has 55 per cent market share, competing head-to-head with Streets, which is part of Unilever.
"We are not a high margin company but we are not a low margin company either," Charlton said.
"We pride ourselves for the fact that we charge a fair price for our products but we don't gouge the market," he said. "We are very mainstream."
In terms of profile, Tip Top is the envy of the marketing world, with 95 per cent brand awareness among New Zealanders.
But for one of the nation's most well known brands surprisingly little is known about the company behind icecream company Tip Top.
Perhaps that's because Tip Top has had a bewildering number of masters over the decades.
It started off as an icecream parlour in Wellington in 1936. By 1962, the company's Mt Wellington plant had become the Southern Hemisphere's biggest icecream factory. Two years later, Tip Top had expanded and a parent company - General Foods - was formed.
The company went to Goodman Fielder in 1987, Heinz in 1992, West Australia's Peters and Brownes in 1997, Kiwi Dairy Co in 2000 and finally to the current owner, Fonterra, in 2001.
He agreed "icon" is over-used , but be said Tip Top, which spans three generations, probably makes the grade. "We have been a constant and regular part of every summer for almost every Kiwi," Charlton said.
Charlton, an Australian with a marketing background at beverage giants Diageo and Pepsico, is surprisingly devoid of any sales hype, with a "steady as she goes" message.
"Our future is very much around low [to] middle digit growth, and making sure that that growth is strong and constant."
Achieving growth in a country that already has the highest per capita consumption of icecream in the world would appear to be tall order.
Not so, according to Charlton. Opportunities open up in certain age groups, he says, and the 15 to 30 age group is where the action is, he says.
"Our role has been very clearly defined by the Fonterra strategy and that is that we are a cash business and the cash that we provide is too stable, that it needs to grow - not exponentially - but it needs to grow."
"For us, it is very much about continuing to be just us."
