He had followed a 21-year-old woman to her home in Brooklyn on May 9, when the alleged assault occurred.
Mr Key said the Ministry for Foreign Affairs had formally asked for the diplomat's home country to waive diplomatic immunity, but it had declined.
"New Zealand's very strong preference was that he would have been charged. Effectively that sending country stopped us from doing that," he told reporters.
Under the Vienna Convention, diplomats could claim immunity to avoid arrest and detention.
The Government made a show of hauling in the diplomat's head of mission to the Ministry of Foreign Affairs last night "to make it clear how seriously it views this situation".
But the ability for the Government to publicly sanction the country was limited because the Wellington District Court made an order to suppress the diplomat's identity and other details.
The diplomat first appeared in the Wellington District Court on Saturday May 10, the day after the alleged incident, Radio New Zealand reported.
He then reappeared later that month, but no media were at the hearing.
Numerous media organisations, including the Herald have challenged the man's interim name suppression, but it was not yet known when a hearing into the issue would take place.
Mr Key said that as a signatory to the Vienna Convention, "our hands are effectively tied". But he expected the man's home country to investigate the charges.
The Government had been advised that the diplomat could still be charged for a crime committed in New Zealand, and that the offence was taken as seriously in that country.
Mr Key would not say what action could be taken if the investigation did not hold the diplomat to account.
He was also asked why the case was not made public sooner, and if the head of mission would have been called in if the Herald on Sunday had not publicised it.
He said a series of meetings had taken place at a senior level over the past month, and the home country was aware of New Zealand's expectations.
Green Party foreign affairs spokesman Kennedy Graham said the convention's rules on diplomatic immunity were not fit for the 21st century.
Labour Party leader David Cunliffe told TVNZ's Breakfast show the Government needed to be "a bit more upfront" about naming the country where the diplomat was from.
"To emphasise to that country that we expect a justice process to occur at least in their own country, if not extradition back to New Zealand."
Prime Minister John Key could say where the diplomat was from, Mr Cunliffe said.
"That would go some way in satisfying New Zealanders' need to know and I think New Zealanders deserve a full explanation about why it's not possible for the government to do more than that."
The government could also call for extradition because it was such a serious alleged crime, he said.
Former deputy Secretary of Foreign Affairs and Trade and a former ambassador Michael Powles told RNZ the Government had done all it could to hold the diplomat accountable.
"I can't think of anything else it should be doing.
"It is absolutely bound by this convention - it had to allow the diplomat to leave the country."
Even if the diplomat was fired, the person would still be protected from prosecution.
"They've got immunity in respect of this particular event.
"The immunity, bear in mind is not their immunity, the immunity is the government's immunity."
Police said the charges remained active and a warrant had been issued for the diplomat's arrest.
Diplomatic immunity 'archaic'
The father of a young woman who was killed by Chilean Ambassador Luis Lopez, said diplomatic immunity was only there for officials to break local laws.
Sacha Macfarlane, 20, died after Lopez, who had been drinking, crossed the centre line in his BMW and crashed into her car.
He claimed diplomatic immunity, left the country and was not prosecuted. Chile apologised for the incident in 2010.
Ms Macfarlane's father Kester Mcfarlane told Radio New Zealand the situation felt "pretty empty".
"Over the years I've come to the conclusion that the diplomatic immunity was meant to ensure reciprocal protection to diplomats by the Vienna Convention, but the abuse of privileges has now become so widespread, especially in the 30 years since Sacha's death, that it's just a convenience for diplomats to flout local laws of the country which they've been sent to."
The law was archaic, which protected people who were in a host country, who believed they were above the law, Mr Macfarlane said.
He believed the law should be scrapped.
"There's no case that I know of where it's been necessary for a diplomat in a host country, it always seems to be stacked up against the residents of a country where they've been posted."
