One flatmate said Mr Oliver had moved into the house six months ago after moving from Tauranga, but he was originally from "up north".
The woman said Mr Oliver whom they had nicknamed "Teapot" would have turned 54 later this month.
"We nicknamed him Teapot because he had lost part of his arm in an accident... about 12 or 13 years ago...He had a great sense of humour, "one flatmate said.
On Friday afternoon Mr Oliver had decided to go fishing, she said
"He was a very keen fisherman and used to go fishing at least three or four times a week, and it was about 3pm he finally headed off to his usual fishing spot.
"But he came back about five minutes later to get a box of beers he'd forgotten, then settled in for several hours of fishing, and that was the last we saw of him."
The woman said she did not know there had been an accident until answering a knock on the door by police about 12.30am on Saturday.
"It's been the most horrible surreal thing that has happened in my life, and I still half expect him to walk in the door. It's been a real struggle to know what to do next, and you feel so powerless when you can't do anything but wait for police to tell us he's been found.
"[Monday] was the first time I could even face going down to the spot where his car left the road, and it still feels so unreal. It's the not knowing which is the hardest thing, and we have been trying desperately to contact his family without success."
Mr Oliver was understood to have two sisters, one older and the other younger, and his elderly widowed mother lived in a rest home near Whangarei.
"Not being able to say goodbye is extremely hard," another flatmate said.
"I have told by a clairvoyant that he has gone, but I still feel his presence all around the house, and we just want his body to be found so we can say our proper goodbyes."
Neighbour Mick Pyne said he had been searching the beach and shoreline ever since and every time the Coastguard vessels went out he also headed out hoping to be there when Mr Oliver's body was found.
"I'll keep searching as long as it takes to find him," he said.
Mr Pyne said it appeared from the police's orange road markings that Mr Oliver was returning home when he veered off the road, crashed down a bank and hit a large rock on the shoreline about 3m down.
Maketu Coastguard president Shane Beech said after two-and half-days of extensive sea and shoreline searches from Papamoa to Maketu, it was now a body recovery situation.
The spot in the river where the car was found and recovered could have been as much as 4.5m deep, he said.
"We were all quite surprised that the driver's body was not inside the car when it was lifted about 400m from the Kaituna River mouth," he said.
Yesterday the search was suspended due to "quite adventurous" weather conditions, he said.
Inspector Karl Wright-St Clair said the search was currently on hold, but police would re-assess the situation on a daily basis.
"We do believe Mr Oliver had been out surf casting on the Maketu beach and was returning home before this incident happened," he said.
Mr Wright-St Clair said the crash was still under investigation, and it was being treated as a motor vehicle accident but it was too early to speculate what may have happened.
However, two possible scenarios being explored were that Mr Oliver managed to escape or was ejected from the car but in the strong current he was washed out to sea, he said.
Mr Wright-St Clair said the both front windows of the vehicle were not intact when it was lifted from the river.
"But based on inquiries so far police believe Mr Oliver was the only occupant in the car, and our thoughts are with his friends and family at this time."
