"They didn't handle it very well," he says. "They missed the point of why I was ringing them and all they were concerned about was not giving me a credit."
Armstrong, 39, didn't want a credit, just a solution to the problem. "If you don't want to give me a credit, which is not what I want anyway, give me a list of your top 10 customers and I'll start a bakery and I'll take them off you," he declared.
True to his word, he set up a baking operation out of a garage in Otahuhu, all the while continuing to chef at Prime, which did lunches and functions from its downtown Auckland base, while also squeezing in training for the Coast to Coast multisport race.
Armstrong admits his wife, Kate, didn't see a lot of him during those first few months.
"It was just this challenge to get there and get the next hurdle and the next hurdle, whether it be a group of customers or whether it be a dollar thing or whether it be just a new product, getting things looking better or better systems.
"Everything was a challenge or a step or a milestone, another cobblestone in the past." That was 10 years ago.
Since then he has sold out of Prime, shifted bakery sites four times and his product range has grown to encompass more than a dozen styles of handcrafted bread, plus Danish pastries, slices, sweet loafs, muffins, doughnuts and the doughnut-croissant hybrid, the cronut.
Add to that a cookbook, appearances on TV show MasterChef and a finals spot in last year's EY Entrepreneur of the Year award.
What hasn't changed is his attitude to customer service, which means that someone is available to listen, and will front up in person to sort out problems.
"The only thing we didn't listen to was when people said 'can you make this because this is what [the competition] makes?'" Armstrong says a couple of times he did give in to pressure to create a "me too" product, but always regretted it and went back to doing things his way.
The Loaf way is handmade by 62 staff, meaning products aren't always symmetrical or perfectly formed, with automation only introduced to complete low-level tasks such as melting chocolate for the rocky road slice or creating 9000-odd, perfectly round, 30g slider buns every day.
Business has not been without its hiccups, says Armstrong.
Products that to his mind were amazing have not been what his customers wanted.
"How many things worked, how many things tanked?
"More tanked than worked but you just make the decision 'hey, thought it was a great idea, clearly it's not'. Shelve it, move away, don't forget about it because it might be a good idea one day and just move on to the next thing."
One of his biggest challenges came in 2007, when, in a bid to break into the retail market, he created a separate company, Sugarloaf.
"It's probably the best thing I ever did in business because Sugarloaf actually went incredibly pear-shaped from a cost perspective." Assumptions he'd made based on his experience with food service clients didn't necessarily translate to Sugarloaf's retail customers.
"The best thing about having two separate businesses is it was completely transparent so we could see what was failing and what wasn't." Armstrong says he "killed it", pulled out the successful elements and folded them into Loaf.
"If we'd never kept them separate something tells me we could have been in a slightly different position than what we are today." He says a "helluva lot of hard work", trial and error has gone into creating Loaf's line-up, where the key products are his ciabatta, lemon loaf, chocolate brownie and newly released mini bites range of Loaf's most popular slices.
Volumes are talked about in tonnes - 14 to 15 tonnes of flour every week; 3 tonnes of butter and two tonnes of sugar every month.
A new measurement has been added too: shipping containers.
A 20ft container is sitting outside his Penrose office ready to go to Singapore, a market that has munched its way through five 40ft containers of Loaf products since early this year.
Mini bites are the flagship product for Singapore, as well as Australia, Hong Kong, Malaysia and Indonesia, and have been formulated for a longer shelf life with reformatted packaging to suit local tastes.
Armstrong says he is excited about the future and a next phase of growth that could potentially be huge.
"Whatever we do and whatever we make and whatever we sell I won't sell anything I don't believe in and I don't enjoy and won't eat myself."
