The former Manukau City Mayor told Radio New Zealand that it would have been very hard to be re elected to "one of the most all-consuming jobs in the nation".
He was returned to the post in 2013, just days before news of his extra-marital affair with council advisory board member Bevan Chuang broke.
He said the scandal greatly affected his public image.
"I think it certainly had an impact on the view of Aucklanders towards me."
Mr Brown said that after discussions with his wife Shan and three daughters, he decided nine years as mayor, including Manukau, were enough.
Former Auckland City mayor John Banks, who stood unsuccessfully against Mr Brown at the first Super City elections in 2010, said today would have been a very disappointing day for Mr Brown.
"His leadership was, at least, quirky but irrelevant and, probably at worst, missing in action.
"The bureaucrats and the non-elected boards(of the council-controlled organisations) are now well in charge and a law into themselves. The Auckland Super City administration and leadership is a shambles.
"Right now the city is going sideways and backwards," Mr Banks said.
Mr Banks has not ruled standing again next year.
The 68-year-old said in the absence of a younger, more energetic candidate putting their hand up there was a possibility he could stand.
He said Auckland needed strong, visionary leadership.
"I have the energy and passion to do this but I want to see where the interest lies with the public in standing up and wanting to be involved in the future of Auckland," he said.
Mr Banks said he did not want to see the Auckland mayoralty handed from one Labour Party candidate to another Labour Party candidate - a reference to Mr Goff replacing Mr Brown.
Labour MP Phil Goff who is declaring his candidacy for the mayoralty on November 22, said Mr Brown had worked hard to settle the structures of the new city.
"The mayor also has so much passion for the City Rail Link, which I also endorse, and that will be part of his legacy."
Former Auckland City mayor John Banks, who stood against Mr Brown at the first Super City elections in 2010, has not ruled standing again next year.
Mr Banks, aged 68, said in the absence of a younger, more energetic candidate putting their hand up there was a possibility he could stand.
Of Mr Brown, he said: "His leadership was, at least, quirky but irrelevant and, probably at worst, missing in action."
Deputy Mayor Penny Hulse said she was standing again but her focus was on getting the region's new rule book for growth, the Unitary Plan, into shape for council to approve in July or August.
Councillors told the Herald they were annoyed by getting word of Mr Brown's decision at 9am on Sunday - when a news release was sent out by the mayoral office.
This was described as "typical" of Mr Brown's aloof leadership style by Penny Webster who is the chair of the finance and performance committee.
"We've been telling him for months to put us all out of our misery so we can move on."
However, Councillor Linda Cooper feared party politics and populist causes would get in the way of building public confidence in the council.
