But they had failed to do, and even covered up their continued offending.
"It's particularly important where babies and children are concerned the product safety standards are complied with," Judge Ronayne said.
"Choking hazards in this case exposed young children to the risk of injury or even death."
No children are thought to have been harmed by 123 Mart's products.
With around 60 retail stores throughout New Zealand, 123 Mart operate under the names The 123 Mart, Dollar Store 123, King Dollar Store and Max!Out.
The Commerce Commission said it investigated the retailer three times and laid charges after 123 Mart was warned in 2013.
The company went into liquidation in September, making the payment of their fine unlikely, but Judge Ronayne felt that was immaterial.
"It's appropriate to send the right message in this case," Judge Ronayne said.