"If it has been contained, as we hope, I think it will be a bit of a blip."
If contaminated products had made it on to supermarket shelves - the problem Mr Scrimgeour feared - then it could make the market nervous.
That could lead to the value of milk products dropping, and come back on farmers in the form of lower milk prices.
"It's nervous times," said Mr Scrimgeour.
"It has the potential to [impact the wider economy]. It's just a matter of how nervous the customers are really."
After the melamine scare in 2008 the Chinese market turned sharply in favour of New Zealand milk products for their quality and safety aspects that were now in question, he said.
"A situation like this does have the potential to dent the reputation quite severely."
Infant formula manufacturer Nutricia has recalled Karicare Infant Formula Stage 1 (0-6 months) with batch numbers 3169 and 3170 (use by 17.06.2016 and 18.06.2016) and Karicare Gold+ Follow On Formula Stage 2 (6-12 months) with batch number D3183 (use by 31.12.2014).
The company said none of its products tested and sold in New Zealand indicated any contamination.
But, given new information by Fonterra, it made the decision to undertake a precautionary recall on specific products.
Tauranga father Josh Norton said: "Even if it's not on the shelves in New Zealand it's still pretty shocking that it was produced at all and that the quality control systems are not good enough to catch it."
Authorities recalled up to 1000 tonnes of dairy products across New Zealand and seven other countries after Fonterra announced tests had found the bacterium.
A dirty pipe at Fonterra's Hautapu plant has been blamed for the contamination.