The Government Communications Security Bureau did not have a warrant to intercept communications in the lead-up to the raid on the Dotcom mansion in January.
Prime Minister John Key has revealed the GCSB was acting without his sign-off when it was working with police to locate people who were subject to arrest warrants.
He said he had not issued a warrant and was not briefed on the operation.
He said he believed it was a mistake rather than a deliberate flouting of the law, but the inquiry would investigate how it happened.
The first he had heard of it was on Monday.
Mr Key said the operation was not done under the watch of Governor-General Lieutenant General Jerry Mateparae, who was the director of the GCSB until mid-2011 when he was appointed governor-general.
Mr Key has requested an inquiry by the Inspector General of Intelligence and Security into the unlawful interception of communications by the GCSB, saying it had acquired communications without statutory authority.
The law that governs the GCSB allows it to intercept the communications of foreigners without a warrant in some circumstances - but it cannot intercept New Zealand citizens or residents' communications even when it has a warrant and both Dotcom and his co-accused Bram van der Kolk.
The inquiry into unlawful bugging by the Government's foreign intelligence arm relates to intercepted communications by Kim Dotcom himself and his co-accused Bram van der Kolk, as well as their families.
The Prime Minister today announced that he had asked the Inspector General for an inquiry into unlawful interception of communications in the Dotcom case by the Government Communications Security Bureau.
Dotcom's lawyer Greg Towers confirmed the interceptions in question related to Mr Dotcom, Mr Bram van der Kolk and their families, all of whom were New Zealand residents.
"At the end of the day, until we get the outcome of the inquiry we really don't know to what extent there has been illegality. We know it's been illegal because they've admitted it."
The legal team had known about the issue since August, when it came up in a High Court hearing into the legality of the raids on the Dotcom mansion. Mr Towers said they could not comment further because the matter was still before the Courts.
"It will be interesting to see how this plays out with the High Court proceedings."
The Crown had filed a memorandum in the High Court advising of the illegal interceptions, which were to help locate people who were subject to arrest warrants.
Mr Key said he had requested the inquiry by the Inspector-General of Intelligence and Security into the circumstances of unlawful interception of communications of certain individuals by the GCSB.
Mr Key said the Crown filed a memorandum in the High Court in the Megaupload case advising the court and affected parties that the GCSB had acted unlawfully while assisting the police to locate certain individuals subject to arrest warrants issued in the case.
The bureau had acquired communications in some instances without statutory authority.
The Prime Minister was told about the matter by the director of the GCSB on September 17 and referred the bureau's actions to Inspector-General Paul Neazor.
The inspector-general is an independent statutory officer with the power to enquire into any matter related to a government intelligence agency's compliance with the law.
Mr Key expressed his disappointment that unlawful acts had taken place.
"I expect our intelligence agencies to operate always within the law. Their operations depend on public trust.''
Mr Key said he has also asked the inspector-general to recommend any measures he considered necessary to prevent such acts from happening again.
"I look forward to the inspector-general's inquiry getting to the heart of what took place and what can be done about it.
"Because this is also a matter for the High Court in its consideration of the Megaupload litigation, I am unable to comment further,'' he said.
Labour leader David Shearer said the alleged illegal spying amounted to a "shocking breach" of New Zealand law.
He said Mr Key needed to explain a claim he had not heard of Mr Dotcom until the day before a police raid was carried out at his Coatesville mansion.
Mr Key would have been called on to authorise monitoring carried out by the GCSB, Mr Shearer said.
"While it's been revealed that 'some' bugging was done illegally, it is not credible to think that other monitoring by the agency was not signed by the Prime Minister before the raid was carried out.
"This is not about national security. This is about John Key's own word and whether he has told the truth to New Zealanders."
New Zealand First leader Winston Peters said Mr Key had failed in his responsibility to oversee the GCSB and the New Zealand Security Intelligence Service.
His actions showed a "mega disregard" to his role and his actions amounted to "dereliction of duty", Mr Peters said.
"The taxpayers of New Zealand are about to pay a fortune to Mr Dotcom because of the Prime Minister's cavalier disregard for his responsibilities.
"Mr Key appears not to care about the security of New Zealanders, and is instead more concerned with photo opportunities, three-way handshakes at rugby games, and prancing down fashion runways."
Green Party co-leader Russel Norman said the actions of the GCSB should be concerning to all New Zealanders.
"A number of New Zealand agencies, including the police, have bent over backwards to assist the United States in prosecuting Mr Dotcom," said Dr Norman.
"Government agencies should not, however, break New Zealand's laws to assist another country.
"The Kim Dotcom case doesn't appear to constitute a threat to New Zealand's national security, so it is hard to understand why the GCSB considered it should be involved at all."
Mr Norman said the inquiry into the illegal spying needed to be thorough and transparent, with findings made public at its conclusion.
Shortly after Mr Key made the announcement this afternoon, Dotcom tweeted: "The NZ equivalent of the CIA has spied on me UNLAWFULLY''.
Last week the Court of Appeal reserved a decision on whether Dotcom would receive disclosure from the US Government in its case against him.
The US government had lodged an appeal against a ruling that required them to disclose all evidence in its internet piracy case against him in order for a fair extradition hearing.
Dotcom, Finn Batato, Mathias Ortmann and Bram Van Der Kolk are defending accusations of copyright infringement and money laundering.