"I want to know how we're going to fix it. That's my first focus."
Ministry of Social Development chief executive Brendan Boyle had formally apologised to the minister for the error.
According to RNZ, Ministry of Social Development deputy chief executive Ruth Bound said most of the underpayments were less than $1000.
"Work is under way right now to work through options to address the error and do the right thing by people who have been underpaid," she told RNZ.
About 800 of the 286,000 people receiving the accommodation supplement were affected by the error, Ms Bound said.
Ms Tolley said she hoped to have all the wrongful payments addressed by October. But she warned that the cost of tracking down the underpaid beneficiaries could be larger than the cost of repaying them.
"I've had initial advice on this, but I'm awaiting further advice around the total number of people affected, the total amounts involved, and the best way to deal with any reimbursements.
"We will not be seeking any repayments."
The review which detected the incorrect payments also picked up five other areas "which could have some implications for repayments".
Ms Tolley said these were mostly minor mistakes which did not involve large amounts of money.
Labour Party social development spokeswoman Carmel Sepuloni said the Ministry of Social Development needed to take people's entitlements seriously.
She said there was a culture at Work and Income of "where success is based on how much you can deny people".
The Ministry of Social Development said it would report back to ministers in June on proposals to address the error.