"Some of them on the other side are pretty big," he says. "Willie Jackson's pretty strong. I'm not sure I'd want to take on Willie."
"The thing about Taekwondo is that at the end of a hard day, just going and kicking some pads and having a sparring session is one of the best ways to release and have some fun."
In seriousness, if his black belt reveals anything about Goldsmith's approach to his new job (he's been finance spokesman for three months) it's as evidence of a highly focused and disciplined approach.
He is also an accomplished classical pianist, although when I ask him what else the public needs to know about him, he's keener to emphasise his family life.
He's a busy father of four - so martial arts are on the back burner for a while.
Right now the goal is take on the Government's narrative about the state of the economy.
It's not terrible he concedes. But it could be a lot better.
On stage debating Finance Minister Grant Robertson for the first time this week at the Mood of the Board Room event Goldsmith lands a few blows - although it's safe to say its a business friendly crowd.
And Goldsmith is unashamedly pro-business.
I ask him what really distinguishes him from Robertson - who's fiscal prudence and enthusiasm for issues like productivity see him continue to rate pretty well with business leaders.
The fundamental difference is that National has a greater respect for the contribution business makes, he says.
"I think there is a tendency in the Government parties to see it as big business, all about profit, there to be taxed and milked," he says "I certainly have the view that surviving in business is very hard."
National Party finance spokesman Paul Goldsmith. Photo / File
When he talks about what makes him qualified to run the nation's economy, Goldsmith cites his own experience running a business and also his time as a writer.
"My background was as a business historian. I spent my professional career before politics writing about New Zealand businesses and exporters who've figured out how to succeed," he says.
His last two books were on last on Bill Gallagher the businessman and inventor who took electric fences to the world and entrepreneur Alan Gibbs.
"I also wrote about the New Zealand economy and tax and history, and in the process I ran my own little business and experienced the challenges of paying provisional tax and GST and all those things," he says.
He absolutely puts blame for the current low levels business confidence at the feet of the Coalition.
There are three factors at play, he says.
He argues they've added costs (taxes and bureaucracy), they've created uncertainty (with things like Capital Gains Tax and other working groups) and then there has been "rank incompetence" around implementation of policies like Kiwi Build and in the infrastructure space.
"Put those three things together and it adds up to a collapse in confidence."
These criticisms no doubt land with many in the business community, but what about the rest of the electorate?
Does Goldsmith accept that there was a desire to slow things down and address social issues which led to a shift in power in 2017?
"We had nine years of good progress and it was a close run thing at the election," he says. "I don't think we got it all wrong and we don't need to completely rebuild it."
"But now in opposition you've got to go back and say, what are things we need to emphasise more?"
So as well as looking at growing the economy and increasing income, he wants put an increasing focus on the costs New Zealand families face.
"A big part of that is discipline in the regulatory space," he says. "I don't think we were perfect in that area of government."
"If we could translate the same level of discipline that we have around spending money to passing new regulations that cost the economy and families money we could make a real difference."
Goldsmith doesn't want to be seen as entirely focused on the bottom line but he sees wealth creation as a key to social progress.
"It's not all just about money of course. Money doesn't create happiness but it does create opportunities," he says.
"When I think about what government is all about, it's raising our living standard and having a strong economy. It's also about preserving and enhancing what is special about this country …the quality of environment, relatively high level of social cohesion, our high trust, low corruption environment, all those things are equally important."
