Maybe it dates back to the War of the Triple Alliance from 1864-1870. Paraguay never, ever, give in.
Their footballing culture continues the tradition of that war (against Brazil, Argentina and Uruguay). It nurtures the Paraguayans' famed fighting spirit which has seen them qualify for the last four World Cups from the football hotbed that is South America.
That war was actually a terrible defeat but the victors were astounded by the Paraguayan defiance and determination to fight until the death. To this day it seems to translate onto the football field.
"We have often lost to Brazil and Argentina over the years but we have never, ever been afraid of them," leading goalscorer Oscar Cardozo, who played in this week's 0-0 draw against the All Whites, said. "There is no intimidation."
In terms of style, while they have undoubted technique, Paraguay also play with a physical tenacity probably matched only by Argentina. To add to the mystique, on the field the players all speak the native language Guarani, completely unintelligible even to their Spanish-speaking South American opponents.
But the Paraguayans have achieved well beyond their country's physical and economic indicators. Argentina and Brazil generally have two places locked up which leaves eight countries battling for two-and-a-half spots in qualifying for World Cups.
With a population of just six million people, Paraguay is dwarfed by much larger nations like Colombia (42 million), Peru (27 million), Chile (16 million) and Ecuador (13 million).
Cardozo emphasised the current level of expectation: "Going on what they have already achieved, they have to at least reach the quarter-finals - if not the semifinals."
So what is Paraguay's secret? The answer seems to be three-fold. They have a highly developed youth programme; their players are snapped up by European clubs; and then there is that fighting spirit.
On the streets of the capital Asuncion just weeks before the tournament, World Cup fever was everywhere but the All Whites were barely on the radar - after Italy and Slovakia, many locals couldn't recall the third opponent in Group F. But Jose Luis Chilavert could. Chilavert was honoured by FIFA as one of the top 10 goalkeepers of all time. The charismatic Paraguayan redefined the position, taking free kicks and penalties and scoring 62 goals across his professional career.
His pre-tournament words now seem quite prophetic: "New Zealand is known for rugby, that is for sure. But today in football you can't underestimate anyone. For the New Zealand players to go to a World Cup is like taking a small kid from a tiny village to Disneyland - it is everything and they will perform as such."
Juan Angel Napout, the president of the football association, said: "My colleagues at FIFA are always asking me, how do you keep qualifying so well?
"It is simple. We are a small country, not even close to being rich, but it is a country where there is football everywhere.
"Everything stops when Paraguay plays," he said. "Nobody is even going to be allowed to die."
He was only half joking.
Paraguay play Japan at 2am, Wednesday.