"One time I went up to the end of the road to meet him and he walked out of the bush and said, 'You are the first person I've talked to for three months'," brother Ray said.
Ernie's niece Prue Meehan remembers how he started farming in the Moeawatea. He used to walk through it to go shearing at Moeroa upstream. Sid Hughes was farming leasehold land along the way, and Ernie asked if there was any going. Sid said he wanted to give up his 160ha and Ernie applied to a land court for it.
The court tried to discourage him, saying the land was too hard. Similar properties taken up by hopeful farmers after World War I had been abandoned. But Ernie was determined. He ring-fenced the leasehold block in nine months and freeholded it for 500.
He moved on aged 23 and left aged 83, to be buried there two years later. By that time he had bought up neighbouring properties and was farming nearly 2500ha. At its peak he had 5000 stock units, mostly sheep and a few half-wild cattle.
Pigs were a problem, because they killed lambs. In the early years, Ernie and Merle Sorenson, the female pighunter who farmed upstream at Moeroa, handed over pig snouts and tails to get a bounty for the pests.
"He was very competitive with Merle, and she got more than him," Prue said.
When Ernie first moved into the valley he had neighbours, a telephone and radio communication. A surveyed public road ran through his property, linking Waverley with Eltham - but it was never formed. Since then the tough farming life has cleared out all the valley's permanent residents.
Baz is one of Ernie's many relatives, and talked about him as we trampers walked. Ernie spent weeks at a time clearing scrub. Later, when he owned buildings to accommodate them, he employed teams of Fijians, too. He kept about 10 farm and pig dogs and had a total bachelor existence, first in a hut on the initial property and later in shearers' quarters downstream. He lived in one room, with everything within arm's reach, Baz said. There was no electricity and he cooked on an open fire, eating goat liver for breakfast and sweetening drinks with condensed milk.
"It was a pretty horrendous diet. If you visited Ernie it was considered rude if you didn't take something like fresh bread in to him."
Ernie enjoyed his own company, but he liked people too. He had a fund of stories and jokes and a "great roaring laugh," niece Melody Wallace remembers.
"He lived on his own but he probably had more visitors than I have. People were up there all the time, and they tended to look after him a little bit in his old age," Baz said.
Many of the visitors were hunters and four wheel drivers. Ernie's house was piled with books and magazines and he followed current affairs on radio and in newspapers. He liked a debate and people were impressed with his conversational style.
He also liked a drink. Neighbour Chris Bergman remembers sharing a big bottle of whiskey with Ernie and another man.
"By the time we finished we had had it, and Ernie was like he hadn't had a drop."
The bachelor farmer was an amazing worker and very practical. Because he didn't use a vehicle he had equipment stored in boxes at handy places all over his farm. He did all his own fencing and shearing, loved trees and planted a lot of them.
He built his own woolshed, with split manuka for the gratings. He also carved culvert tunnels through papa rock, famously going in with his pickaxe in the morning and knowing when to come out because his seven-hour candle had burned out.
"He loved the hills and the challenges they gave him. He used to say people who had smaller hills in the front country were lazy farmers," Prue said.
Baz called him "at one with his environment".
"He couldn't see why we would want to go overseas. He was happy with his lot, committed," according to Prue.
There was no wife or girlfriend during his time in the valley - "none of that nonsense" according to brother Ray - but fellow resident Anita McPherson looked out for him for a while, and her memorial and ashes are near his grave.
Ernie's funeral was a major affair, with a service at the little St Hilda in-the-woods church in the Waitotara Valley. He was driven back into his valley next day, and buried by the kauri tree he planted.
That spot was around the mid-point of a two-day walk for us trampers.
Our weekend began with a long drive to Moeroa, at the end of a dead-end road that winds 60km into the east Taranaki hill country from Eltham. As we left, we met none other than Merle Sorenson, farmer and possumhunter, in her eighties and riding a four-wheeler.
At the turn-off was a sign erected by the owner of most of that end of the valley, Chris Bergman. It said Kenmuir Station Wildlife Properties Hunting Park. An arrow pointed down a track into a scrub-covered valley.
Mr Bergman acquired part of the property after logging huge rimu trees on conservation land at Taunoka at the top of nearby Watershed Rd. He had bought the logging rights to the trees and there was nothing the Conservation Department could do to stop him. In late 2004 people were so upset about it they blocked a road to stop the logs getting out.
After much negotiation Chris agreed to swap rights to the trees for Moeawatea conservation land adjoining land he already owned there. He now has a luxurious hunting lodge and cottage on it, and "makes most people welcome". No longer physically involved in logging, his focus is now on bees. In January there were a lot of hives on Kenmuir Station. He paid for them to be helicoptered in and accommodated the beekeepers, in return for a percentage of takings from the manuka honey.
Merle Sorenson still hunts possums on his property, and he has others hunting deer, goats, wild cattle and pigs. Now living between Taranaki and Kerikeri, he said the Moeawatea was an awesome place and he spends a lot of time there.
We four walked downhill into his property on a 4WD track. We passed his hunting lodge and continued downstream beside the clear Moeawatea, with its outbreaks of limestone rock and small waterfalls.
As the Bergman property ended we came on a virtually untracked section of valley with a deep, narrow papa gorge. Hunters had left a rope to hold onto while scrambling down, but it was a slippery climb back up the other side.
After that came a section of track bordered by conservation land, with beehives on the grassed river terraces. The stream ran through gorges, and scrub-covered hills extended on either side. It had started to rain, and mist was creeping down the valleys.
We set up camp on a grassy flat and lit a fire. After dark the place was alive with possums, and Baz thought he heard a kiwi in the night.
Conservation land it might officially be, but this area was not being conserved. The land had been cleared and was reverting to a degraded forest.
"It's been suffering years of grazing pressure from goats, deer, pigs and farmed animals, with no boundary fences," Baz said.
For most of the second day we were walking a bulldozed track, past Ernie's grave and first shack, still open and usable, but messy. Then there was his last dwelling, the hut built after the shearing quarters he lived in for years burned down.
Nearby was the woolshed he built himself, and the former Annabel homestead, now repaired and named Opaku Lodge. We passed other shacks and shanties, the ruins of abandoned farmhouses, and finally reached the house where Rewi Alley and a partner farmed for six years in the early 1920s. It was restored and added to in the 1980s and is now a heritage site.
At the end of the valley road was The Boar's Nest, a farming base for the Woods family, and their private swing bridge across to the Whenuakura Valley. From it you can look upstream to where the Moeawatea joins the larger Whenuakura River. The rest of the walk was a long slog, in light rain, up the hill to Mataimoana Rd. Despite signs of occupation along the way we had not seen another person for the whole weekend.
Our trip was billed in the Tramper magazine as "A Walk on the Wild Side". But the valley only felt half wild.
It's accessible to vehicles half the time, and it's half-clear, half-bush, half-farmed, half-settled and half-full of introduced creatures - with the call of the Australian rosella as common as that of the long-tailed cuckoo.