Kevin Spacey plays Frank Underwood in the award-wining series 'House of Cards'. Netflix used big data to help with its decision-making around casting.
Big data has been a big buzzword for the past several years. Rapid changes in technology are creating a new frontier for computing and innovation, one which has the potential to revolutionise everything from healthcare and security to education and business. The applications are seemingly limitless and evolving every day.
New Zealanders have been pioneers in the data space for decades, with Ross Ihaka and Robert Gentleman developing R, a programming language and software environment for use by statisticians in data mining and analysis.
The pair developed the language while at the University of Auckland. It first appeared in 1993 and remains one of the most used data mining solutions globally.
But overall, New Zealand business has been slow out of the gate.
"I think there's a lot of opportunity for us in that space to leverage our roots, plus, with what's happening with big data internationally, to create some really interesting economic opportunities for New Zealand," says Stephen England-Hall Hall, CEO of Loyalty NZ. "New Zealand as a country is in a really unique position because we're highly connected and already familiar with data being part of our decision-making process. So the opportunity for us, from a big data point of view, is pretty vast."
"There's an abundance of data there, there's some ideas around how you could leverage it, but there's still some questions around how you get value from it," adds Craig Richardson, managing director of Wynyard Group. The company, founded as an offshoot from Christchurch software firm Jade, has emerged as one of the early superstars in the big-data space.
Wynyard uses an innovative approach to advanced analytics, data mining and crime science for use in risk management, investigations and intelligence. Founded in 2012, Wynyard already boasts a client list of 400, featuring governments, Fortune 500 companies and national infrastructure providers. It is recognised as a thought and market leader in the industry at a global level.
The market has been quick to back the latest cab off the Kiwi high-tech BIG
New Zealand businesses are only just starting to figure out data is more than ... two analysts sitting in a basement somewhere doing some programming.rank, with the company's share price reaching a high of $3.21, a significant jump from the $1.15 offer price when it listed on the NZX last July. Wynyard estimates the market for the services it provides is worth $5.3 billion globally. But they are competing with the likes of IBM and Thomson Reuters for market share.
Richardson credits the New Zealand Police as innovators with the way they are using data, describing their thought leadership and use of data to prevent crime as "some of the best in the world". The police first showed evidence of their big data acumen with the implementation of Signal at the 2011 Rugby World Cup, using purpose-built software to mine social media for information. Police Director of Intelligence Mark Evans previously told the Herald that the software is used to help police "identify and analyse social media feeds relevant to crime and public safety," including the time and the location.
"Social media occurring in the public domain is just another public space and police need a presence in that space in the same way we need to be seen in neighbourhoods and on the roads," said Evans.
Following the Rugby World Cup, the software has been used in emergency response, crime detection and at big events. In a recent incident, Signal proved the diversity of its applications, being the first to detect a plane in distress by picking up on the Twitter postings of one of the passengers. New Zealand Police have notably deployed Signal during the Royal Tour. It has been used to monitor the London Riots and the 2012 London Olympic Games.
Despite the success with big data in New Zealand in academia and at public-private partnership levels, businesses have not been in any hurry to jump on the bandwagon.
A survey of New Zealand and Australian IT professionals by ISACA revealed only 22 per cent were confident their businesses had a policy with regards to big data and only 45 per cent were confident in their businesses' ability to ensure effective governance and privacy of big data.
"We are a bit behind the curve, which isn't necessarily a bad thing," says England-Hall. "A lot of money has been wasted on the wrong stuff. ROI (rate of return on investment) at the moment is about $1 for every $1.50 of investment, but people have spent a lot of time hunting down the wrong paths which haven't really been well thought through.
"New Zealand businesses are only just starting to figure out and learn that data is more than a basket analysis. It's more than two analysts sitting in a basement somewhere doing some programming. It's a sophisticated way of driving business performance and it's going to become a boardroom topic."
Talent Shortage
At the forefront of New Zealand's issues with data is the severe lack of available talent domestically. Though the data scientists New Zealand has are seen as "some of the very best," hiring more is proving to be a serious challenge for businesses.
Loyalty NZ's England-Hall estimates New Zealand is going to be short of 1800 data scientists by 2017. "Where are they all going to come from and how are we going to ensure that New Zealand maintains a foothold internationally in this space?" he asks. "At the moment it's really throwing the net as wide as we can and seeing what we reel in. Domestically there's a very small number of people so we have to look everywhere." That situation doesn't seem to be set to improve at an adequate rate either.
The University of Otago will, for the first time, offer a Master of Business in Data Science this year after an "examination of market needs and opportunities," says programme coordinator Michael Winikoff. "We considered a number of potential foci for the Masters degree, and in the end felt that Data Science was the most promising area, in terms of meeting a real need, but also in terms of being an area where we had sufficient critical mass of relevant expertise on staff."
The University of Auckland offered a Master of Professional Studies in Data Science for the first time last year, tailored for those already in the workplace. The initial programme ran with five students but interest for this year's intake is said to be "significantly higher" as the university looks to build a more flexible schedule to fit around people with professional commitments.
Data sourcing
Data today comes from all types of sources, from traditional numeric data in databases and spread-sheets, through to email, social media posts, videos, geo-location based data from mobile phones, transactional data from invoices and deliveries, even real time data from sensors - part of the internet of things. The cumulative amount of data collected is doubling every 18 months and as storage and processing power become more efficient and cost-effective, the potential for what could be achieved continues to expand.
Virginia Rometty, CEO of IBM, told the Economist that by one estimate there will be 5200 gigabytes of data for every human on the planet by 2020. "Powerful new computing systems can store and make sense of it nearly instantaneously. A new generation of 'cognitive' systems, built for big data, can keep up with the flood because they aren't programmed, they learn - from their own experience and from our interactions with them," she said.
Globally, big data is being used to address a plethora of issues in a wide variety of industries. American heavyweight Xerox uses complex algorithms in conjunction with personality tests and big data sets to hire call centre employees.
Netflix are using big data sets to support decision-making around creation of original programming, leading them to license BBC mini-series House of Cards for a remake. They correlated fans of actor Kevin Spacey and director David Finch to fans of the original series, leading to the casting decisions for the Emmy Award winning series.
Mt Sinai Medical Centre in the United States is using big data to analyse the entire E. coli genome sequence, including in excess of one million DNA variants, to study why some strains develop resistance to antibiotics.
Definition
Big data refers to the exponential growth and availability of both structured data and unstructured data. As more information is created and collected, software is able to organise the data before algorithms process it and decipher stunning insights. As the data becomes more diverse and information-rich, the more valuable and meaningful the results generated can be.
Comvita shares fall 7% after mānuka honey company forecasts a net loss of more than $20m.