"Places are getting expensive to rent and our clients are struggling to pay for their accommodation. The accommodation supplement has not gone up but costs have."
He said he knew former prisoners who were struggling so badly to make ends meet they were living on one can of baked beans a day.
"This leads to health problems, and they're suffering from depression and anxiety on top of that."
Tim Metcalfe from Jigsaw Whanganui said with Housing New Zealand staff being taken out of Whanganui it was difficult for clients to get the help they need.
"And it's very hard to get a state house these days - it's hard for clients to tick all the boxes."
Mr Metcalfe issued the MPs a challenge.
"You need to unhook yourselves from the neo-liberal economic model, because all that's doing is creating huge gaps between the rich and the poor. We're seeing the results of that every day."
Mr Twyford, who is Labour's housing spokesman, said the party planned a huge overhaul of housing if it became the government after the next election. This included building more affordable state houses, introducing a warrant of fitness for all rental housing, cracking down on property speculation and a nationwide roll-out of emergency housing.
The forum also discussed issues such as the lack of input by social agencies into government policy, funding, regional development and technological literacy.
While in Whanganui, the three MPs also met iwi representatives. Tomorrow, along with other Labour MPs, Annette King, Trevor Mallard and Ruth Dyson, they will visit Whanganui Hospital and Age Concern.
They will also meet management of Cavalier Spinners, who last week announced the plant will shed 68 jobs.