"We have already taken smokers out of cafes, restaurants and bars and now we are being asked to ban them from smoking outside on the pavement.
"What would stop them from moving 10m down the street and smoking there, anyway," he said.
Mr Caffell said it was insulting to single out Maori, Pacific people and the mentally ill and the whole idea was "stupid".
" It would be the same as our ban on smokers in the town square, all they did was move to the fringes of the square and smoke there," he said.
Councillor Brent Goodwin was also against the proposal.
"I am saying this as a person who has smoked during my life." Deputy mayor Graham McClymont said business owners had been "hammered enough lately".
The ban could even put young female smokers at risk of being assaulted if they were out for an evening and had to move away from lighted areas into the dark to smoke.
A vote rejected giving support to Mr Smith's proposal with only two councillors - Jonathan Hooker and Mark Harris - supporting it.