We live in the age of "Big Data". More and more data is being collected about us by both governments and businesses. This data is being used to make decisions which affect many aspects of our lives.
Dr Cathy O'Neil, chief executive of ORCAA (O'Neil Risk Consulting and Algorithmic Auditing) and author of the New York Times bestseller Weapons of Math Destruction was in New Zealand to speak at Statistics New Zealand's "Informed decision-making through the ethical use of data" Data Summit.
New Zealand Herald Data Editor Chris Knox caught up with O'Neil at the Summit.
Q: What sort of decisions about our lives are based on data collected about us?
In the United States the answer is pretty much every bureaucratic decision. What high school you go to, what college you get into, what kind of job can you get, whether you get a raise, whether you get a promotion, whether you get fired, whether you can get credit cards or loans or housing opportunities or insurance.
In fact, health insurance is a huge part of the current landscape of big data. One life insurance company just made a rule that it will only insure people that carry around a FitBit. Car insurance companies will force you to have data surveillance in your car - to see how you brake.
Q: What has led to this being the situation in the United States?
It's an arms race in pretty much every industry I just mentioned - especially insurance. Insurance is a really good example, because if you think about different pools of insurance they are competing with each other. The more they can cull high risk people off of their own policies on to someone else's policies, the more profit they can make. They are competing for more information on their customers.
Q: My initial thought is recording how people brake doesn't necessarily seem like a bad thing - what is your point of view?
I have a lot of points of view. It definitely interferes with the idea of privacy - in terms of confidentiality. They also track your location. What if you want to go somewhere and not be followed? This affects the very idea of autonomy, period. Time of the day is also tracked - they could demote you because you work nights. They could also demote you for braking for good reasons - like if there is a kid in your neighbourhood who played street hockey. Is that really something that should raise your premiums?
The larger point is - what is insurance? What is the social contract that insurance is supposed to represent? And the answer is very obviously that it is protection for the public against catastrophe. But if insurance companies are working very hard to get rid of the most risk-prone people it is not actually accomplishing that.
Going back to people who work nights. It is believable that there is a correlation of people who work nights and people who get in an accident. But that doesn't mean everyone who works nights is going to get into an accident.
As a society we need people to work nights and they need to be able to get car insurance.
That's exactly right. That goes back to the social contract. If they are specifying which people to stop covering it ends up undermining its own original goal.
Q: Can these systems be constructed in ways that avoid these problems?
Certainly. But there have to be laws. The law that is relevant to the health insurance situation is that you can't charge people more for a pre-existing condition. I would go further, and say that what we really need to have is a stronger law that says you cannot be charged more for a future condition.
Insurance was working well, essentially because health insurance companies didn't know what they didn't know. And now they are starting to know that stuff and it's a disaster.
Q: You have only been in New Zealand for two days. Is there anything that has caught your eye in this space?
Two things. First, I think the social contract here is much stronger and I'm kind of wishing I lived here. Second, my impression is that you take these questions of data governance and ethics pretty seriously. You are, as a community, striving to do it right and it is very exciting to see and it is almost sufficient to make me lose my cynicism for the next day.