He told RNZ it was a phrase he used often and he meant no disrespect to the Prime Minister by it.
Asked if he was apologising, he said: "I can do. She's pretty tough. I'm sure she's not in the least bit worried about it ... we're in a robust political environment."
He said he would just as readily have directed it to Finance Minister Grant Robertson.
In fact, Goldsmith has used the same phrase before toward NZ First MP Shane Jones in Parliament.
"I use the phrase all the time. The point of it is to say: 'Stick to your task rather than meddling in somebody else's.'"
He maintained his core argument was valid.
"It is not for politicians to determine what business decisions should be made in terms of a private company trying to stay afloat at a difficult time.
The Prime Minister's focus should be on clearly articulating a credible growth plan to get us back on track as a country.
What we're seeing at the moment is a plan that consists of debt-fuelled government spending on a colossal scale and waiting for a vaccine. We need to do better than that."
Goldsmith told RNZ the wage subsidy and its extension made sense, but the country needed to focus on private investment, business confidence, and bringing back international students, rather than putting more restrictions on overseas investment.
Asked if National would have international students returning now, he said: "We are not privy to the official advice from health authorities."
National Party leader Todd Muller was in hospital having minor surgery on pre-cancerous moles before returning home to recover, leaving Kaye to deal with Goldsmith.
Goldsmith was defended by Act leader David Seymour, who said men also knitted and that Goldsmith's criticism of Ardern's comments was well-founded.