Mr Chesterman, whose portfolio includes properties at the north end of Victoria St, said he wanted to avoid any further attacks from the community about having a perceived conflict.
He also said as tenants paid rates and not landlords - there would be no personal financial gain for him.
Mr Chesterman, Mrs Bell and Ms Westphal, who had obtained separate legal advice from the council's lawyers, said they didn't want to put the council at risk of any legal challenge by being involved and left the room.
Mrs Westphal said she was also doing it to protect her family's interests from being splashed on the front page of the newspaper.
Councillor Roger Hennebry, another commercial and residential property owner, bucked the trend by refusing to leave. He said legal advice had showed any conflict was marginal and felt all councillors around the table - with the exception of Dave Macpherson and Margaret Forsyth who lived out of the area - could also be put in the same category.
Hamilton mayor Julie Hardaker said it was disappointing they felt they had a perceived conflict. Her own view was there was no real conflict and the council was elected to represent the community and to do that with an open mind.
She said it was a shame only 10 of the 13 council would be present and councillor John Gower would also only be present for part of the hearing due to a clash with another council commitment. Councillor Angela O'Leary also pointed out her mother was a submitter. "I don't believe I have a conflict. I don't always listen to her."