One of those concerned, 20-year-old Tessa Rankin, was camping with two others at her friend's house on New Year's Eve when they heard a noise outside around 8.30pm.
When they came out of their tent, they found the woman, wearing a dressing gown and slippers, eating their food, which was outside the tent.
They later discovered she had been drinking their vodka as well.
They tried to ask her name, but the woman only stared at them and mumbled something that sounded like, "You know my name".
Miss Rankin ran down the road to the rest home to see if she was from there and said the nurses immediately knew who she was talking about.
As they headed back a distance of about five houses to get the woman, the nurse said, "Oh, she's gone a bit further this time," according to Miss Rankin.
Ruth Rankin, her mother, said "[the woman's] an accident waiting to happen".
"I know that situation. They're very, very vulnerable if the wrong person comes up to them."
A man living on the street said he saw the woman wandering around about twice a week, and that there have been a number of clients of the rest home getting out over the years.
"They should have a bracelet or something that if she goes outside the gate it sets off the alarm," he said.
He worried that rest home clients might be wandering the streets in the dark and accidentally walk into the path of oncoming cars.
The general manager of Lady Joy Rest Home, Stephen Poynter, said the woman in question had not been wandering for two to three months and she had deteriorated only recently.
Everything a resident did was documented, Mr Poynter said.
If anything "out of the ordinary" happened, including a resident going wandering, it was written up in an incident report, as required by the Ministry of Health.
He said a full investigation had been done, and steps were being taken on Monday to see if the woman could be moved to another facility.
"It's all been done appropriately and in a timely manner," he said.
New Zealand Aged Care Association chief executive Martin Taylor said rest homes were not funded to put trackers on residents, and could not legally lock the doors or impede people's flow to and from the door.
He said it was the district health board's responsibility to assess the elderly person and decide whether or not they should be put into a rest home or dementia care unit. If the DHB decided they should be put in a rest home, this meant the DHB believed it was acceptable for the residents to wander, he said.
"A rest home means that any one of the residents can come and go as they please."
Mr Taylor said the goal was to give residents the maximum amount of freedom.
"That doesn't mean people don't wander away from rest homes and get lost," he said. "That's a reality of having people with dementia in rest homes.
"It's not the responsibility of the rest home to make sure that every one of those is tracked at every point of the day."
Whanganui District Health Board spokeswoman Andrea Bunn said, "Whanganui District Health Board are confident Lady Joy Rest Home has taken the appropriate action to ensure their resident receives the level of care required."
If rest home workers believed a resident should be put in a dementia care unit, they needed to contact the DHB and ask for another assessment, and if they said it was urgent, the case would be seen to within 48 hours.
