In the fourth set Monfils, 30, broke Nadal and then consolidated to take a 4-2 lead.
A 14-time grand slam champion, Nadal admitted after the match he felt he was losing the upper hand against his unpredictable opponent.
"The match was tough," Nadal said.
"He had chances then and he started to serve huge.
"I keep fighting with a positive attitude and finally I played last few games good."
Down 3-4, 0-30, in the fourth, the Spaniard showed his grit to level and then fired a trademark backhand winner to break back.
In a run of eight points, Nadal wrested back control of the match with an error by Monfils sealing the win on the second match point.
"I'm very happy to be in the quarter finals in a grand slam again after a couple of years without being there is very special," said Nadal, who made a first round exit from Melbourne Park last year.
"Especially here in Australia where I feel a little bit like home."