The man, who said he had never done "anything like it before" said he did not do it to push his own flag preference on others, but believed that flying the flag alone was a "total manipulation" of the flag referendum process.
"I took the flag down as I disagreed about it in isolation," he said.
The man said the current flag should be flying alongside the Silver Fern flag, possibly with one underneath the other on the Elizabeth St flagpole, as opposed to the alternate days the council voted to fly the flags on.
"Not everyone goes past there every day," he said. "It's a first-world issue, but it must be affecting the voting process."
The bagged flag was later collected by Mayor Stuart Crosby (pictured) at the Papamoa address where the man said he left it.
Mr Crosby said the flag was "unharmed" and said the man had been "courteous not to deface it". "If there is no damage I doubt we will take any complaints to the police."
Mr Crosby said he was not surprised it had been taken down. "It was only a matter of time."
He said the reasoning behind flying the flag options on alternate days was to "raise awareness and encourage people to vote" in the referendum.
"It does not indicate a position at all."
Mr Cosby said the Silver Fern flag was gifted by businessman Peter Farmer, who also financed the flagpole in 2008.
Mr Farmer said he had given the flag to the city so locals could "see it flying in a local context".
"I'm very disappointed, it's a pity," Mr Farmer said.
Mr Farmer said he had recently seen the Auckland Airport flagpole which had both flags flying at the same time.
Although he supported the flags flying together like that at Auckland Airport, he was not sure if the Elizabeth St flagpole "could take the stress of two flags flying, especially when it is wet". He thought the Elizabeth St flagpole might be able to fly both flags for a short amount of time but said he was "not an expert" as to how long it could handle it.
"But personally, I've got no problem with the flag flying underneath," Mr Farmer said.
Bay of Plenty MP Todd Muller said the Silver Fern flag "deserves to be flying ... the alternative flag helps people of the Bay of Plenty make an informed decision."
According to Mr Crosby, the Silver Fern flag might not even fly again after today due given he had received two official requests about the legality of flying it.
"We're confirming with independent legal advisers on that issue," Mr Crosby said.
The resource consent of the flagpole only referred to the national New Zealand flag".
He said council had received internal advice that both flags could be flown but had sought independent advice on the issue last week, and would find out the legal position today.
The man who removed the flag said if it did go back up, he would "have to abide by it" but hoped there would be a "groundswell of opinion and it will change the council's mind".
