He came to New Zealand, lived in a flat in Ellerslie and worked in Penrose, while his girlfriend Suzanna, soon to become his first wife, managed the roughest hotel in town where, he claims, the previous manager had been murdered.
"We loved it in New Zealand, there was a lot going on," he recalls. "We were particularly keen on discovering Maori culture and the arts, the textiles and the carvings, while the myths and legends which explained all aspects of the modern world, were wonderful. We didn't miss our own culture at all!"
But they returned to Britain and Suzanna, obviously made of tough stuff, brokered a peace between Penrose and his mother. It meant when Miller died of cancer, aged 70 in 1977, the two were friends.
Penrose has spent the past three decades conserving and sharing the work of his parents and is director of the Lee Miller Archives and the Penrose Collection at Farley Farm. He's written a handful of books about them but The Boy Who Bit Picasso was his first foray into writing for children.
"It came out of frustration because there was a spate of books about Picasso saying quite nasty and unkind things about him - dishing the dirt - and, yes, there were people he treated badly but this didn't chime with the Picasso I knew, although I knew him through a child's eye.
He says people are complex and the book, partly inspired by a story of a dog who ate a Picasso painting, is an attempt to show a man who was kind to small children and animals. Miro's Magic Animals, to be released this year, is a sequel of sorts, although a totally different story.
"I never knew how much fun writing for children was. After The Boy Who Bit Picasso, I wanted to do it again."
The Boy Who Bit Picasso
by Antony Penrose
(Thames & Hudson $ 22.95)
Miro's Magic Animals
by Antony Penrose
(Thames & Hudson $ 26.99)
Antony Penrose speaks today at the New Zealand Portrait Gallery in Wellington from 11.30am-12.15pm. Admission is free.