The last time to two sides clashed was at the same stage in last year's tournament when New Zealand won 39-26.
France held a 3-0 halftime lead before crossing for the only try of the game shortly after halftime when Romain Ntamack, son of French great Emile Ntamack, scored.
Louis Carbonel converted the Ntamack try and added two further penalties in the second half to extend the lead to 16-0.
New Zealand finally crossed in the 70th minute when first-five Harry Plummer scored in the left-hand corner before converting his own try.
Minutes later New Zealand looked set for a second before a fumble close to the line squandered a chance with numbers out wide.
French then held on in the closing stages to secure a famous win at this level.
"I'm immensely gutted at the result. But I'm very far from disappointed with the lads. My lads played with a lot of heart and a lot of pride," said New Zealand skipper Tom Christie.
"I think we were unlucky sometimes and, yes, some of our mistakes were at times our errors but I can't fault my boys. They [France] are an amazing outfit and when you make mistakes they will punish you."