He and acting Education Secretary Peter Hughes had begun talks with Datacom, the previous payroll provider.
"We're basically investigating setting up a contingency - I want to stress that would not be a simple solution.
"You would have to wind back a significant amount of what's been done and then go forward again, and I don't think anybody is under any illusion that that solution would be at least as painful as where we are in the short term."
He said that would not be a decision taken lightly.
Mr Joyce said the Datacom payroll system had also had problems and was part of the reason the Government pushed for an alternative.
Mr Joyce also confirmed Cabinet Ministers Bill English, Hekia Parata and Craig Foss had signed off on the Novopay system even though the system had "bugs" in it.
"In terms of the advice to proceed, all advisers were unanimous in their decision that it should proceed," said Mr Joyce.
When asked if Novopay was a "dog", Mr Joyce replied: "it's one with a few fleas."
Mr Joyce said those fleas, or "bugs" were going to be investigated during the month-long technical audit running in tandem with the ministerial inquiry.
The technical audit would incorporate the results of the audit currently being undertaken by Ernst and Young for the ministry.
Mr Joyce is to take the terms of reference for the ministerial inquiry to Cabinet on Monday.
The Government will fund new measures to fix Novopay. Mr Joyce could not say the exact amount being spent but it is believed to be in the millions of dollars.
The commercial and contractual issues between Talent2 and the Government would be put to one side until the ministerial inquiry and technical inquiry were completed.
Mr Joyce said the Government would then go through the contract to determine who was responsible for the cost.
"There is still some debate around responsibilities for different elements of the extra resource we're going to need"
"We don't want the inquiry to get in the way of fixing the system.
"The reality is there are no steps forward that don't involve some more pain for the administrators and the people who are working on the payroll," he said.
Mr Joyce stressed the technical audit and the ministerial inquiry were separate and he didn't want the inquiry to get in the way of fixing the system.
But whether the system could be fixed was a main part of the investigation and Mr Joyce could not firmly rule out scrapping Novopay entirely
He had met sector groups this week - principals, and staff in schools, unions and more announcements would be made in the coming weeks.
Mr Joyce could not say when school staff would be paid correctly, or when the problems with Novopay would be sorted out.
Education Minister Ms Parata said all the advice at the time suggested they should proceed.
"I think hindsight is a wonderful thing," she said.
Labour's acting education spokesman Chris Hipkins said Novopay wasn't ready to be implemented.
"They didn't ask fundamental questions about whether or not it was ready," said Mr Hipkins.
