Fast forward 40 years: Daisley didn't grow out of it and he's grateful for some of those old army mates who, like the shearers and rousies, farmers, truck drivers and chippies he's worked alongside, keep him inspired and humble.
This week, Daisley won New Zealand's richest writing prize, the inaugural $50,000 Acorn Foundation Literary Award for his second novel Coming Rain. Five years ago in Australia, he won the Prime Minister's Literary Award for Fiction with his debut novel Traitor and collected an $80,000 cheque.
It hasn't made him rich or even able to write full-time without additional casual work so, while he writes from 6am to 1pm every day and spends the afternoon walking or doing household chores, Daisley will sometimes drive a truck or cut scrub to keep the household budget in the black. His wife of 29 years, Sylvia, owns a clothes shop and sometimes he helps out there.
When he collected the Kiwi prize, announced at the 2016 Ockham New Zealand Book Awards on Tuesday, Daisley praised the judges for their bravery.
"I was absolutely and totally shocked," he admits, "because I am the only New Zealand thing about this book and I was up against people like Patricia Grace. I might live in Australia, but I grew up here. All my education was here, I served in the New Zealand army and the ideologies of this place have shaped me.
"It recognises a development in New Zealand literature - a New Zealand book can be about the wider world written by someone who no longer lives here. It's a real honour to have won."
Traitor, the novel that won Daisley his Australian award, is about a New Zealand soldier fighting at Gallipoli in World War I who befriends a Turkish doctor. It asks what would make a soldier betray his country? Like Traitor, Coming Rain is the story of an unlikely friendship but set in the Western Australia wheat belt in the 1950s.
I can see the beauty in the ordinary. Once you can do that, you see it's around you everywhere.
Daisley likes to find the beauty in the ordinary and both novels, plus the two he is now working on, are about men inhabiting worlds he knows.
"I've been blessed to live a life with a rich array of people that I have associated with and I can see the beauty in the ordinary. Once you can do that, you see it's around you everywhere."
The son of Ken and Lal, publicans who owned the Raetihi Tavern, Daisley left the army and returned to New Zealand to start the succession of casual jobs he's done most of his life: shearing, sweeping up the shorn wool, scrub-cutting, truck driving and construction. He met Sylvia while having a beer after work in the Whanganui Tavern.
When times got tough, the couple moved to Western Australia where there was more work and better pay. They've been there 25 years and raised five children - four sons and a daughter. When Daisley was made redundant at 34, they had enough for him to attend Murdoch University and get his BA in English and Philosophy. He went on to complete a PhD at the University of Western Australia.
"It was pretty tough, though," he says. "We did have to get a grocery parcel once ... "
He could have stayed and worked at the university but wanted to write so, with Sylvia's support and when his kids were young, he wrote during school hours and worked in the evenings. Eventually a back injury, accompanied by an enforced eight-month break, saw him finish Traitor.
It wasn't until then that Daisley had any sort of success and he's got a drawer full of rejection letters to prove it. But he's finally done what he always wanted and reckons his mum would be proud. He understands why she might not have wanted him to be a writer; in rural New Zealand and Australia, the ability to shear a sheep or cut scrub is more useful than being able to write a poem.
He says it's part of our DNA inherited from our pioneering forebears but, yes, Daisley believes we need to be more tolerant of those who'd rather read poetry than porn; write books rather than cut scrub.
"Part of the message in the books - if there's a message in there at all, that's something up to the reader to find - is to be tolerant and there's the notion of getting in touch with the more tender side of a man's nature.
"I'm a great believer in dialogue. We heal ourselves by talking; silence is another death. I have an army mate who returned from Iraq and was at a barbecue where some of the guys were together and they said hello and went round, 'How are you?', 'Yeah, good mate, good', and it got to him and he said, 'Actually I'm not good; I can't sleep at night and I'm having flashbacks', and it was like the floodgates opened.They were able to talk, get some support and heal themselves."
