The base of a large brick oven filled with round stones and cobbles from the small store in front were thought to have been built in the 1870s, Heritage NZ archaeologist Rachel Darmody said.
"It's unusual to have the stones we've got in it. [The site archaeologist] thinks it's all to do with heating and retaining heat. It really is significant. It's very rare to excavate a site like this," Dr Darmody said.
"It is rare to find such a well-preserved bakery from this time period anywhere in New Zealand. The only other examples we have are from the goldfields of Central Otago and one site in Dunedin. It is exciting to find that part of the early township still survives to tell us more about our past."
The oven may have been about 2.5m high.
"They sold all sorts of things, baking, confectionary and general supplies," Dr Darmody said.
In front of the original shop lay the remains of a slightly newer oven and shop front made from brick and concrete with steel reinforcing, which had not yet been accurately dated.
Amongst the newer parts also lay the base of a chimney thought to be part of the older oven, assistant archaeologist Cathleen Hauman said.
"In the chimney there were lots of clay pipes and sewing needles," she said.
Council staff were working with Heritage NZ to work out how the find could be preserved and left on display as part of the park upgrade.