"Many of us are paying about $3000 in rates, this is going to devalue our homes," one resident said.
"There are schools in the area and plenty of children that live here, it's a terrible idea."
While the residents understood the need for such a facility, they questioned its proposed location with more than 60 school children walking through the area to and from school.
"How can you be so irresponsible?" another resident asked.
Mr Irwin said the housing facility was about reintegrating men back into society.
He believes the men will contribute to the community they will live in and the hosting could increase property values in the area.
He said a staff member would have their own sleeping container, with the other five "primarily for people who have completed addiction treatment programmes".
"By having them there it will cement the intense learning they had over the treatment programme. It's an opportunity for clients to catch their breath and cement their learnings."
He said they have also identified another need within the Napier community which is people coming out of prison.
They would host the person for less than a week only if they have been in prison for less than two years, had no electronic monitoring, were low risk and not sexual offenders.
"We won't be housing no-hopers, they may just need some time out to get their heads sorted."
Mr Irwin said there had been consultation with neighbours but that was eight years ago.