If fewer than half of eligible voters take part in the first flag referendum the second should be scrapped, Labour say.
Parliament has this week debated the Flag Referendums Bill, which will set up the two referendums on the flag - the first to vote on an alternative design and the second for voters to choose between the alternative and the current flag.
Labour is opposing the bill despite leader Andrew Little's own desire for a new flag and its 2014 policy to start the process to secure that change.
Mr Little said that the reaction from the public meant the party did not believe the time was right to change the flag.
"Today we have added two Supplementary Order Papers to the flag referendum bill, one seeking to defer the referendum for five years and the second to stop the final referendum if voter apathy continues at its current level.
"New Zealanders all around the country have told us now is not the right time to change the flag. Almost no one turned up at public events to promote it, millions of dollars were wasted on websites and postcards and a celebrity panel of experts.
"And now John Key is continuing to push his pet project through despite overwhelming opposition."
Flag Minister Bill English hit back at Labour yesterday, saying it was Labour's 2014 policy to hold a similar process but it was opposing it because they do not like John Key.
He said Labour liked to pride itself on being "the owners of New Zealand" and its national identity.
"That may have been the case when Michael Joseph Savage was the Prime Minister, but in this intellectually bankrupt, unaspirational, modern Labour Party it does not own any aspect of our national identity."
The official Flag Consideration Panel is expected to release a long list of alternative designs from the 10,000 submitted next month before whittling those down to the four designs that will be voted on in a referendum in November.