David Fisher: Just how bad were our spies?

Prime Minister John Key has stepped away from directly overseeing the agencies. Photo / Mark Mitchell

John Key has opened up the spy agencies to public scrutiny in a way which we have never seen in New Zealand.

We know more now about what they do and even how they do it.

We know how the two agencies are managed, in that the GCSB and NZSIS both have top-flight lawyers in charge.

There will always be those who say we don't know enough. For those people, we now have improved oversight of the agencies. This also happened under the Prime Minister's watch as minister in charge of the agencies.

The new Inspector General of Intelligence and Security Cheryl Gwyn - another superb lawyer - has been a breath of the freshest air.

Mr Key has since stepped away from directly overseeing the agencies, which is a further liberation. It seems right that the most powerful weapons of state should sit with someone whose role is to objectively challenge his Cabinet colleagues.

Now, even at a ministerial level, the SIS and GCSB answer to a lawyer, this time Attorney General Chris Finlayson.

In terms of oversight and public disclosure, we are heading into an era unparalleled in our history. Citizens now have more ability to see and have explained the tasks done in their name. Again, it might not be enough but it is considerably more than we have had before.

That's where we have come to, three years after Mr Key had to admit Kim Dotcom and one of his co-accused had been illegally spied on by the GCSB. He also had to apologise - a concession which must have been galling.

That single event appears to be the point at which the Prime Minister stopped taking at face value the assurances given by the intelligence agencies, and began a programme for reformation which is huge in its scale and largely behind closed doors.

For all the comparative openness, it is unlikely the public will ever know the truth about how far adrift our intelligence agencies wandered. As a broad indication, consider the fact that respected senior lawyers with strong state experience now sit at all significant levels of the intelligence community. When you're unsure about the law, you need lawyers.

But there have also been reports which paint a picture of the state of New Zealand's intelligence services, past and present. None are individually explicit in their descriptions of how bad it was but the collective run of reports gives an impression of the intelligence community as an isolated part of government, lost to the public they were serving, changing purpose and shape under a cloak of secrecy.

There is a pattern which flows through these reports, whether it be NZSIS boss Rebecca Kitteridge's investigation into the GCSB, or the Performance Improvement Framework reviews or the most recent Gwyn report. Every reviewer is insistent staff at these important and powerful agencies all had good intent, but that they were left operating in frameworks not quite right, or in some cases flat out wrong.

It is almost tragic the intelligence services sought out such eager and patriotic staff only to subject them to a bureaucratic acid trip.

Kitteridge's report was incredible for the simple fact the bureau was so adrift of the law. She found signals intelligence staff faithfully following an internal "bible" showing how the GCSB Act worked on the job. Unfortunately, their "bible" was wrong and instead of telling them how to follow the law, it misled them and saw them breaking it.

And that was only the start. The legal and operational problems were symptoms of an organisation with a dysfunctional structure, bunkered and crippled by near-pointless internal secrecy modelled on larger foreign partner agencies because it couldn't look for role model guidance domestically. Trying to be the approximately 40,000-strong NSA with just 300 people is ridiculous.

These patriots worked on in this, willing to sacrifice all for the bureau and New Zealand in a structure that eventually meant their own capability and competence was compromised. Kitteridge found the bureau to be a place where poorly performing staff wouldn't be moved on partly because of misplaced loyalty, party for fear of leaks and also because it just took so long for the SIS to get a security clearance for the replacement. What a mess.

Finding a problem doesn't fix it - a few months after Kitteridge's report came out, the GCSB ran an operation spying on those competing against Trade minister Tim Groser for the World Trade Organisation's top job. The wisdom of that judgment is currently on with the IGIS.

The reviews didn't improve. State Services Commission performance commentary on the intelligence community suggested it would be useful if the intelligence community had a direction, recommending it "clarify the national security priorities". Then, in another report, it said the intelligence community needed to change the way it worked with "urgency as there is a huge amount of change to be undertaken".

By 2015, the path forward for the intelligence sector was still unclear. The followup inquiry into the Department of Prime Minister and Cabinet, which coordinates intelligence work, reported the need for "significant and necessary system improvements across all the agencies of the wider intelligence and security sector".

Two years after Kitteridge's report, DPMC had just got to the point of hiring someone to "dimension the task". DPMC's own issues with intelligence material were also outed in the report, with the biggest tell being the line which stated: "Issues arise in providing secure intelligence material or policy advice to the Prime Minister." If there is anyone you should be able to be classified material with secret urgency, it's the Prime Minister.

More clumsiness this week - the NZSIS own compliance regime was exposed as poor at best. Sure, it did its best, but not according to any explicit processes. The Inspector General's report into the agencies was also released. It showed the NZSIS had forgotten to send her copies of the first two warrants ever issued allowing video surveillance in people's homes, as it was required to do. Given the concerns expressed publicly over the intrusive nature of the new powers, it is no surprise Kitteridge found herself apologising to her minister.

By comparison, the GCSB fared well. Is it fixed? Possibly, at a basic level. There's much more to having a functioning intelligence community than simply having one part which doesn't break the law any more.

It often takes a massive mistake to reveal a deeply hidden problem. The repercussions of the Dotcom/van der Kolk blunder have been a shockwave which have shaken the intelligence agencies to their core.

John Key's response to a community which cultured problems and surprised him with embarrassment was to throw back the curtains, hire his own directors and to put serious lawyers in charge. There are those within horrified at their new reality, which includes the need to "inform the public of what we do and why it is so important", as one recently disclosed report stated.

Accountability and transparency make the intelligence community one in which all New Zealanders have a stake, by simple virtue of it being more accessible.

But it doesn't solve the problems which still exist - and there's no surety about where it's heading. There are well-founded concerns over the intelligence community's strategic direction (whether it has one) and the blurring of its capabilities into law enforcement.

There will be no inquiry looking back to find how our agencies strayed from the path. Opposition parties keep calling for one. Will their first witness be former Prime Minister Helen Clark? They'd likely prefer the current Prime Minister, who ran the agencies for four years until the Dotcom disaster was exposed. And can you even call Governor General Sir Jerry Mateparae, who was head of the GCSB?

No, that inquiry is never going to happen. Not unless a bombshell emerges. Will Cheryl Gwyn find the agencies helped the CIA's rendition and torture programme?

Until then, we have this current review - narrow and resolutely forward-looking. It's likely to recommend law changes to the NZSIS and GCSB, but will do nothing to clearly explain the need for the law change.

This is the question which needs to be answered - what should the agencies be doing? If their job is "keeping New Zealand society secure, independent, and free and democratic" how can it best achieve that? Among other things, it was confusion about the GCSB's reason for being which led it into forbidden territory.

If we're all clear about the path on which the intelligence community is heading, surely there's far less chance of those agencies accidentally straying into the wilderness.