New Zealand's first openly gay Cabinet minister is close to quitting Parliament because he is sick of being attacked as a "luxury-loving gay boy".

Labour's Chris Carter and his civil partner Peter Kaiser angrily hit back yesterday after being condemned for being among the worst offenders in the MP expenses scandal.

Party leader Phil Goff is tipped to demote Carter to the backbenches, after he topped the list of personal spending from the public purse during the term of the last government.

In an interview with the Herald on Sunday yesterday, Carter said: "I just have to make a decision: Do I want to keep putting up with this?

"Do you want to live your life with this stuff going on all the time? You know, I love being an MP. But there might well be a point soon where I think this is just not worth it."

Carter racked up thousands of dollars on ministerial credit cards while holding the education, ethnic affairs, conservation and housing portfolios.

But he said the public perception of him as living the high-life at the taxpayer's expense was grossly inaccurate - and he still drives a 1996 Suzuki Swift.

"That has been another hard thing - people trying to paint this image of these luxury-loving gay boys who are swanning around, flowers to the boyfriend," he said.

"I have lots of faults ... but arrogance, pride and love of luxury are not among them."'

He said TV3 political editor Duncan Garner had a "Get Carter" vendetta, claiming Garner told him 'I am going to f***ing well get you' during a row in an Air New Zealand Koru Club lounge nine months ago.

But TV3 head of new and current affairs Mark Jennings denied there was a grudge.

He said: "It's not unheard of for reporters to have robust exchanges with politicians.

"Carter is a serial offender on expenses ... it's Duncan's job to shine the light on that."

Carter's partner Kaiser said the expenses scandal had placed a strain on their relationship. He said: "On all sides of the house families and spouses find issues like this extremely difficult and there is a universal understanding of the pressures."