Race Relations commissioner Dame Susan Devoy has labelled Sir Peter Leitch "the least racist person I know".
Sir Peter - known as the Mad Butcher - is under fire for telling Māori woman Lara Bridger that Waiheke Island was a "white man's island".
Dame Susan said she knew Sir Peter and she did not think he meant to offend.
"I know he's the least racist person I know in the world and yet what he said was obviously taken as offence by that young woman," she told Radio New Zealand.
"But I wasn't there and I wasn't part of the conversation.
"It's grown real legs, hasn't it?"
She said Sir Peter often used light-hearted banter, which could be misinterpreted.
"The last thing he would have wanted in the world was to offend someone, I know that.
"Let's not forget he's done a lot of great work in terms of race relations in New Zealand - providing opportunity and building bridges between different cultures.
"I think it's generational and culturally different these days and he's probably licking his wounds today."
She said the mediation service at the Human Rights Commission was the best way to resolve a situation.
"We at the Commission launched a campaign about casual racism, getting people to stand up and address it.
"The thing to remind ourselves is it's good to have conversations about these issues but also to remind ourselves that we have to reasonable and rational in our discussions about it too.
"Everyone's entitled to have an opinion, but respect each other when you're doing that."
Bridger, 23, cried as she shared the incident on Facebook in a video now removed.
In a statement issued last night, Sir Peter said he was "extremely disappointed that a young woman had misinterpreted some light-hearted banter".
"I was joking with her group about not drinking too much because there were lots of police on the island. She said that she was tangata whenua and could do what she liked, and I responded with a joke about it being a white man's island also.
"When she later informed me she was offended by my comment I apologised unreservedly.
There is no way I can ever be accused of being racist."