Labour says the $32,000 Deputy Prime Minister Bill English has paid back to the Government is not enough, and it will continue to attack him over the taxpayer-funded accommodation allowance he has claimed.

In a bid to end the ongoing controversy, Mr English yesterday gave up the allowance and paid back the money received since he became a minister again in November.

But senior Labour MP Pete Hodgson said there were still an issue over the $24,000 a year Mr English had claimed while an Opposition MP.

Mr English's declaration that Dipton in Southland was his "primary residence" let him get the out-of-town allowance even though he was living in a home in the Wellington suburb of Karori owned by a family trust.

"There's still a question about where his primary residence is when he's been living in Wellington all these years," Mr Hodgson said.

He said Labour would also continue to question Mr English about changes made to the deed of the trust when he was switching to the larger ministerial allowance, and Prime Minister John Key's involvement as minister responsible for Ministerial Services.

Mr Hodgson said Labour would wait to see if the Auditor-General - which began "preliminary enquiries" last week - ruled on the primary residence issue before demanding that more money was paid back.

Mr English and his family have been in the Karori home since 2007, and have lived in Wellington for many years, meaning any such demands could end up being for over $100,000.

But yesterday, he said he had done nothing wrong and Dipton was still his primary place of residence.

"I stand by the view my home in Dipton is my home. I live in the house I was brought up in with 11 siblings. It is on English Rd. We've been there for 120 years and it is not up to anyone else to decide whether that's home or not home."

He did not believe he should also pay back the money received while an Opposition MP, as Speakers Margaret Wilson and Jonathan Hunt had signed off the arrangement.

The controversy has been growing around Mr English's allowance and a One News poll last night showed 62 per cent of voters felt the issue was damaging his credibility, compared to 27 per cent who did not. Asked if Mr English had acted with integrity, 54 per cent said no and 30 per cent yes.

Mr English said the fact there was a poll showed how "out of proportion" the issue had become. He made it clear he was paying the money back to end the political fallout, not because he had done anything wrong.

Mr English has been caught up in the row since July, when it was revealed he was claiming a much higher ministerial allowance for living in the same house he occupied as an Opposition MP. He initially paid back the $12,000 difference, and yesterday said he had paid it all back - a total of $32,000.

He also revealed he had stopped taking the allowance when the controversy first broke, despite defending it in the two months since.

He said Parliament needed to examine how it catered for long-serving electorate MPs who spent most of their time away from home.