Some sloppy defensive work from New Zealand allowed Holzberger to tap home from close range as she couldn't believe the ball fell so fortuitously to her.
Following an improved defensive showing from Rhiannon Dennison and Emily Naylor, the Black Sticks started to assert themselves on the game.
Eventually that pressure told as the Black Sticks equalised mid-way through the first half via Sophie Cocks.
The Canterbury striker enjoyed a fine debut tournament with the national side during the Oceania Cup and that goal was just a continuation of her strong form during the week.
With the scores level for a majority of the second spell, skipper Kayla Sharland broke the deadlock with a bullet finish from a penalty corner as the Black Sticks went ahead 2-1.
Striker Gemma Flynn could have iced the game as she picked up a loose pass from Australia but with just Lynch in goal to beat, she couldn't find the finish.
Lynch came out nicely and shut down Flynn's options, which ensured the game headed in to a tense final 10 minutes.
When in doubt though, you go to your experienced players and Australian skipper Madonna Blyth blasted a goal from the top of the circle as she tied it up at 2-2 with about six minutes left.
The lottery of a shoot-out loomed during the final five minutes and New Zealand had a couple of chances but some sharp goalkeeping from Lynch kept the scores level.
From there it was over to Lynch again as she prevailed for her side during the shoot-out.