The Football Ferns will play either Portugal or Italy in the semifinals.
Erin Nayler pulled off a great save in the shooutout to deny Sarah Wijnants before substitute Daisy Cleverley slotted home to secure a memorable win.
"It certainly was nerve-wracking but it was a terrific performance by the players," coach Tom Sermanni said.
"In large parts of the first half particularly, I thought we played some of the best football we've played in a long time. The players then showed a huge amount of character and determination," he added.
"The fact that we kept our discipline, our organisation and our concentration really said a lot about the determination of the team to get a result. It was a great result and a really good performance."
New Zealand were the better side for most of the contest, creating several opportunities to score and keeping Belgium's much-vaunted attacking duo of Tessa Wullaert and Janice Cayman uncharacteristically quiet at the other end.
They earned the lead their endeavour had merited when Betsy Hassett released Chance and she squeezed an effort in off the cross bar.
With Hannah Wilkinson proving a handful up front and the game plan being perfectly executed elsewhere, the Ferns were looking comfortable as the final 20 minutes approached but they then found themselves with their backs to the wall when Bunge received her marching orders.
They could well have scored again though when Wilkinson placed an effort around Belgian goalkeeper Nicky Evrard but her shot bounced agonisingly against the far post.
Belgium's pressure finally told with their last-gasp equaliser and both sides then kept their cool in the shoot-out. Katie Bowen, Gregorius, Rebekah Stott, Rosie White and Meikayla Moore were all on target but New Zealand then passed up an opportunity to win when CJ Bott's effort was saved.
The Ferns would not be denied for much longer though, Longo finding the net before Nayler's heroics allowed Cleverley to finally complete the job.
"It's fantastic to be in this position as we're now only one game away from the final. We just need to see how the players have come through today and make sure we can get some fresh players on the field in the next game," Sermanni said.
"If we go about it in the same manner then hopefully we can get another positive result."
