The site was also popular with tourists and photographers due to its picturesque nature, and steps needed to be taken to prevent people entering the building, while still allowing them to enjoy it.
"Perhaps the community needs to take up its paintbrushes and hammers. We need to hear from locals to make something happen here, because if nothing happens, inertia will make the decision for us."
Mr Witt said although it was a council-owned building, due to its historic nature it was also protected under the council's district plan, complicating what remedial steps could be taken.
The council's immediate proposal was to "isolate" the site with fencing, thereby preventing further unmonitored entry by the public.
This would be discussed at the next council committee meeting, he said.
West Otago ward councillor John Herbert, whose great-great-uncle James Herbert built the factory, said he believed most local residents would favour preserving the building, given it could be done "with minimal cost".
"It's a local landmark and testament to the rich history of the area. But we need to get the message out there that its future isn't guaranteed."
richard.davison@odt.co.nz