Athletic would have gone into the sheds knowing they could well have been further behind.
Light rain made way for sunshine at the break and with it a change in fortune for Athletic as they started stringing passes together and visiting Rangers' territory on a more regular basis.
Jude Hiri showed some good touches and cashed in with a header from close range for the home side's opening goal in the 71st minute.
His goal lifted the Athletic side who put on a stunning defensive display over the next ten minutes to keep Rangers at bay.
And that effort meant all the more when Jordy Gilmore levelled the score at 2-2 seven minutes from time.
Both sides did have chances to find a winner in normal time.
Rangers won a series of four straight corners while Athletic's Ross Kinnerley had a one-on-one opportunity with the keeper on the whistle.
Athletic's deserved elation at earning extra time soon turned to disappointment.
Rangers' Sam De St Croix was brought down in the box just minutes into extra time and stepped up to slot the penalty before the visitors doubled their advantage just a minute later.
This was classic cup football, however, and a great strike from Ryan Holden in the final minutes kept the match alive as Rangers held on.
Though obviously disappointed, Athletic coach Jason King was pleased with how his side stood up to more fancied opposition.
"Really, really proud of the boys. They played hard and hopefully gave them a little bit more than they were expecting," he said.
"Everybody out there, look at them, they're all lying around, hobbling. I don't think anyone is coming off with energy to burn.
"You've also got to give credit to New Plymouth. They came down and they played hard and they played well but it's good to see that our lads can stand up against them.
"They've got big physical players and you look at the likes of Jude (Hiri) and Logan (Hughes) - all of 16 - it's pleasing to watch out there and if we take that spirit into our next game, god help Marist."
The second half comeback came through not trying too hard to force play, King said.
"I said to the boys 'you gotta put your body on the line, throw yourselves around' and there was a couple of times that literally came out. Literally they threw their bodies around and they all worked together.
"That doesn't mean New Plymouth didn't play well in the second half either but the luck went our way and the football gods smiled on us and we got that extra time."
King had to use one of his replacements early in the first half which didn't help when it came to playing an extra half hour.
"I think while the boys battled well and were unlucky to miss out.
"But I think the better team did win. They came down and played great football and our boys went out and battled our hearts out but couldn't upset them."