Edwin Linder, 42, was also killed in a CBD attack - in Mills Lane on August 3.
"It's more frightening for the older ones because they're the ones that are more vulnerable than everyone else - not as tough and strong and loud as they used to be," said one man.
He said he was often targeted, especially by nightclubbers in the small hours.
"I'll be sleeping in an alleyway and get a bunch of [drunks] coming up saying 'yeah here's a bum, that's what we need - a two-legged toilet, something to piss on'. That really sucks, especially when you're on your own and you can't do s*** because there's a whole bunch of them."
He said everyone who had slept on the streets would "definitely" have a similar story.
"It just sucks, watching our own lot getting wiped off," he said.
Another man said he knew Edwin Linder reasonably well, and used to help him out with food and clothes from time to time.
"We had some good times, me and Ed. There were some downfalls but he's gone now. He's in a better place."
Acting Detective Inspector John Sutton said several items of interest had been recovered from the scene of the killing.
Police wanted to speak to three Asian people - two men and a woman - thought to have been in the park at the time who may have witnessed the assault.
Others in the park saw Mr Harris fall to the ground and gave him first aid as they contacted emergency services.
Inspector Andrew Coster said Mr Harris had recently been living with a friend in Epsom, but before that he was living on the streets.
