Sir Peter told the Herald from Melbourne that he did not know what to say when he heard what Ms Fenton had said about him.
"I honestly don't know what to say, to be fair - and it's not often that I'm lost for words. I don't know what I've done wrong.
"I'm very disappointed and it's just taken the wind out of my sails. I'm absolutely gutted by her comments."
Sir Peter said he supported Helen Clark when Labour was in power and took her to Warriors matches.
"The National Party never complained that I supported Helen Clark," he said.
"I've done nothing with John Key that I didn't do with Helen Clark - I've been a little bit more vocal."
In response to Ms Fenton's claims that he was "sucking up to John Key big time", he said: "I thought freedom of speech was one of the things people went to Gallipoli for. But obviously in her opinion, it's not."
He also said it was unfair that she was taking her frustrations out on the 37 Mad Butcher stores which he sold in 2007, but still promotes.
Ms Fenton said she made the comments in the "heat of the moment" after hearing Sir Peter compliment Mr Key - but was adamant she was not calling for a boycott of the butchery chain.
"I have never ever suggested a boycott, or started a boycott or called for a boycott - that is simply not true."
Ms Fenton has since shut her Facebook profile down and said she wanted to make amends for her comments.
"It was a reaction and it was silly.
"But the last thing I wanted to do was offend people, that's not what I do," she said.
"It was an impulsive and silly thing to say."