"Ministers knew about this and knew that when action was required to have been taken by an investigative mind, which the Auditor General surely must have, it wasn't taken," he told Radio New Zealand.
"And now the Auditor-General himself, I believe, must stand down until a full-scale inquiry gets to the end of this matter and points blame where it should have laid in the first place."
Peters said he found it hard to believe Matthews did not know what was going on at the ministry. His party would not have approved his appointment if it had known about his lack of action to prevent the fraud, he said.
Labour MP Trevor Mallard represented Labour on the panel that approved Matthews. He would not comment on what information about Matthews' actions was made available to the panel.
But Labour leader Andrew Little said information had been made public since Matthews' appointment that Mallard was not aware of when Matthews was being vetted.
Little would not say whether his party would have opposed Matthews' appointment if it had the fully story about the fraud case.
But he said it should be investigated by the State Services Commission.
United Future leader Peter Dunne also wanted the commission to step in.
Prime Minister Bill English, speaking to reporters in Japan, said if there were any allegations that Matthews failed to act, it would be a matter for the State Services Commission.
But he said calls for Matthews to stand down were "a pretty drastic way of dealing with the issue".
"While there may be ongoing speculation about it, the issue does appear to have been dealt with satisfactorily.
"In fact, it demonstrated that the system is able to pick up these kind of problems."