The intelligence agency databases holding some of the most personal information about New Zealanders breached basic standards for protecting those secrets, a new report has found.
An inquiry by the Inspector General of Intelligence and Security found the lack of basic security standards was well-known for years at the NZ Security Intelligence Service.
Even though it was well-known little was done, meaning there was no formal protection for the four databases containing information about those wanting security clearances.
That included personal information such as financial details, medical histories, relationship secrets, substance abuse or even sexual preferences that might emerge during the vetting process for those accessing classified information.
The report from the Inspector General Cheryl Gwyn is the latest in a series of reviews exposing the NZSIS and its partner agency, the Government Security Communications Bureau, as shambolic and operating outside expected standards - and even the law.
The reviews followed the revelation in 2012 that Kim Dotcom and a host of others were illegally spied on, and have led to wholesale change across the intelligence agencies.
Gwyn's report released today shows the NZSIS brought in new systems to streamline vetting but had a requirement to get all four systems "accredited" by the GCSB before they went into operation in 2009.
If it couldn't get the GCSB to sign off on the use of the systems, it was obliged to get a temporary waiver - and to make sure that each system logged who was using it and for what purpose.
Instead, nothing was done - and, aside from minor security tweaks, it stayed that way until Gwyn's office started investigating two years ago.
Gwyn found the problem was known from the moment the systems were installed when the GCSB "raised a broad range of security concerns".
Gwyn's report said the concerns were such that they were discussed between the directors of the two intelligence agencies.
"The NZSIS did address some of those concerns but put the two systems into operation without certification or accreditation."
An external review of the other two systems also recommended the NZSIS get the systems accredited "but that recommendation was not acted upon".
Gwyn said it was difficult working out why the NZSIS had made decisions in a certain way because of the lack of proper record-keeping - a fault identified and corrected after an earlier review.
"The available records indicate that concerns were raised within the NZSIS, as well as by the GCSB, over potential security noncompliance and vulnerabilities and remedial steps were proposed, though these did not proceed."
By 2010, the steps proposed to fix the security flaws had gone from "urgent" to "business as usual" and eventually were cancelled in 2014.
Since putting the programme of accreditation and accreditation into place in 2015, there have been significant security upgrades for the systems.
Gwyn also sought to discover what controls there were around accessing the information, in line with the requirement that access to the highly sensitive information be logged.
She found it was only possible to see who had looked at the records for one of the four systems.
Two of the systems had no way of showing who had accessed the information and it was technically challenging to extract data from the third system.
During this time, Gwyn noted, the leaks by NSA's Edward Snowden and the hacking of 22 million personnel records from the United States' Office of Personnel Management showed the heightened risk the agency faced.
NZSIS director Rebecca Kitteridge said improvements had been made and all four vetting systems were certified and accredited.
"There has been no indication to date of any breach or compromise of vetting information. NZSIS vetting staff take the responsibility of holding personal information seriously."