He also said it was irrelevant whether the women left from New Zealand or not.
"The fact of where they leave from is irrelevant because they're New Zealanders. If they're New Zealanders ... they may return to New Zealand and so we have to deal with those issues."
Green Party co-leader Metiria Turei called for an apology, saying Mr Key had lied by omission and in doing so had "cast a shadow of paranoia over Muslim women in New Zealand".
Mr Key rejected the Greens' accusation that the public had been misled.
"It is just a statement of fact that there are New Zealand women who are jihadi brides."
The minister responsible for the SIS, Chris Finlayson, said the only apology he would be offering the Muslim community would be to apologise on behalf of Ms Turei, "who started all this nonsense".
He told reporters at Parliament that the Green co-leader's "performance has been lamentable".
Asked to elaborate, he said: "I'm not going through it again because I would die of boredom," before adding, "What's the critical issue here in all this? New Zealand citizens.
Where they left from is an irrelevancy."
Asked whether he would meet with the Muslim Council of Women, who wanted to discuss the issue, he said he would "meet anyone, anytime to allay their fears". But that was unlikely to include an apology.
"You don't just go round handing out apologies willy-nilly for no purpose at all," Mr Finlayson said.