Finding ways to dive into Big Data

Big Data is considered a key part of social reform. Photo / Getty Images

When sentencing someone convicted of a crime, the judge adds a couple of years because of the laws they will likely break on release.

This is the future of justice - a sentencing system in which Big Data is used to predict crimes which haven't been committed, in the same way the "precogs" in the Tom Cruise film Minority Report could know who was going to commit murders.

As science fiction as it sounds, exactly that type of scenario is the focus of a new government-formed think tank called the Data Futures Partnership, which has been formed to act as New Zealand's guide to a data-ordered future.

The sentencing scenario is playing out in Pennsylvania. In New Zealand, says the new Data Futures Partnership chairwoman, Dame Diane Robertson, we already use data to estimate reoffending for parole.

As government accrues more data and finds way to link huge pools of data together, how exactly do we want to allow it to rule our lives?

Big Data - as these pools of information are called - will be massive. The partnership was formed after Finance Minister Bill English led the establishment of a working group that looked at how data could be shared by private and public sectors.

The extraordinary pool of public-owned data is considered a gold mine for private companies, which can monetise it, but the Government needs society's permission to make that happen.

This is called the "social licence" - one of the many tasks on Dame Diane's list of jobs. "It's day one," she says. The outgoing Auckland City missioner had only just stepped into the role when she met the Weekend Herald yesterday. Until now, her experience with data was the collection and analysis of information about people in need.

Big Data is considered a key part of social reform. Eventually, it will predict those most likely to suffer social ills as children, those educationally deprived, those in households where violence is most likely to emerge. Early interventions can be staged - even before the problems emerge.

The "social licence" will be "what we see as acceptable in intervening in people's lives", says Dame Diane. There will be enough data to fully analyse the benefit system and philanthropic contributions to those living in poverty. "Are we making the best use of our money?"

The partnership is also looking to the private sector use of the data. The huge pool of public information is a gold mine to the private sector. In straight dollar terms, innovation stemming from data brought in $2.4 billion to New Zealand in 2014. If we were geared towards taking advantage of data like Australia does, it would be $4.8 billion a year.

Don't get it? Welcome to the new "rich-poor" divide, says Dame Diane. "The new included-excluded," she adds - those who understand how Big Data affects their lives and those who do not.

The Cabinet paper released on the subject included a large section on Open Government - completely redacted.

Data mining

• Prisoners in Pennsylvania face having the chance of their reoffending impacting on the amount of time they spend in jail under a new plan. Those less likely to reoffend will be released sooner, while those judged to be a risk face a longer sentence.

• In the US, credit card customers have had their credit limits cut after a data analysis of shopping patterns showed they had shopped at the same stores as people with poor credit histories. Statistically, it made them a greater risk.

• Your car registration costs different amounts depending on the likelihood of accidents happening to people driving the type of car you own. The idea of the Big Data crunch was to reward those who pose the least risk.