Facial recognition in supermarkets hailed a success, tensions rise amid the ongoing conflict in Lebanon and can your boss really force you back to the office?
Significant caveats have been flagged over live facial recognition in the Privacy Commissioner review;
Review of supermarket trial found to be lawful, to have protected people’s privacy and to have reduced serious offending;
Serious data gaps found in trial that recorded 226 million scans of suspect faces.
What was the facial recognition trial?
Foodstuffs North Island - operators of Pak’N Save and New World - wanted to introduce facial recognition technology into 25 stores for a six-month trial period.
As the Privacy Commissioner’s office said at the time,facial recognition “will be used to scan and make a biometric face template of each customer as they enter trial supermarkets to see if they match a watchlist of people identified as causing harmful behaviour”.
The supermarket company met with the Office of the Privacy Commissioner (OPC) and took advice on how to implement facial recognition. The oversight report from the Office of the Privacy Commissioner said “we recommended that (Foodstuffs NI) should run a trial of (facial recognition)”.
To back up that recommendation, the OPC opened its own formal inquiry on April 4 2024 to cover the trial, which ran from February 8 2024 until September 7 2024.
Foodstuffs-owned Pak'nSave and Woolworths-owned Countdown. Photo / NZME
The OPC said: “It is hard to overstate the privacy implications of a technology that, if widely deployed in supermarkets, would capture images and process the faces of millions of New Zealanders going about their daily lives.”
That’s a valid observation. Over the six-month trial period it scanned 225,972,004, clearly including repeat shoppers. Foodstuffs NI has around 4 million customers a week. A failure rate of one in a million would still be 200 people a year who have wound up on the wrong end of live facial recognition.
What’s the problem it is trying to solve?
Foodstuffs NI and other retailers have spoken of the increase in retail crime in recent years, particularly with harm to staff and customers. Foodstuffs NI says it faced 5124 retail crime incidents in the first three months of 2024. Nationally, the estimated loss through retail crime is said to be $2.8b.
It’s not just the shoplifting, though. Aggression and violence towards staff was also a major motivator. As the OPC said, there was an overlap between the two with intervention over shoplifting incidents often leading to aggressive or violent behaviour.
Live facial recognition offers a number of benefits, advocates say. Deterrence is the first - those who know live facial recognition is operating are said to be less likely to offend if they know they will be detected and potentially banned from shopping there. And for those already on a watchlist, they know their images have been captured and face a high risk of being identified if they enter the store.
Foodstuffs trialed live facial recognition across 25 stores in the North Island. Photo / File
If that doesn’t put someone off, live facial recognition refines the human element by speeding up detection. Rather than monitoring of CCTV cameras by store security workers operating off memory, it provides a match with such speed that - Foodstuff NI says - it gives staff an average of four minutes of extra time to decide how to respond.
That could include making a plan to deal with someone known to be problematic which could include calling police for backup.
How does facial recognition work?
When someone entered one of the Foodstuffs NI supermarkets in the trial, a camera would capture their face from which a computer would create a mathematical model of the person. That would be done by measurements of the shape and position of facial features like your eyes, mouth or nose. The mathematical model is called a “biometric template” and is compared against images of the person that had been previously captured.
The result will be a “comparison score” expressed as a percentage that measures the likelihood the image of the person captured entering the supermarket is the same person who’s image is held, for example, in the store’s watchlist of people previously identified as a risk by store security.
In the case of Foodstuffs NI, the percentage score initially used was 90%. OPC considered that was too low and responsible for matching the wrong images. It has recommended Foodstuffs NI increase the percentage to 92.5%. No further issues emerged after that change.
What happens when it doesn’t work?
This is a developing technology that comes with flaws although those are decreasing as time passes. OPC’s view was that the type of system is a critical choice for those seeking out facial recognition technology with wide variation in quality between products on the marketplace.
Issues which have emerged across platforms are higher error rates matching women and similar issues matching people with darker skin, largely because lighter shades are used to capture the biometric markers. “Those error rates are likely to be higher if the system has not been trained for a particular population,” says OPC.
Users across the world have attempted to iron out these discrepancies by training systems on the specific population on which it is being used. And the OPC says there is “no specific training data set for the New Zealand population”.
Dr Karaitiana Taiuru spoke of concerns Māori would be used as guinea pigs for facial recognition companies. Photo / RNZ
OPC said “it is unclear that these improvements have affected error rates for Māori and Pacific peoples”. It also notes greater negative effects for “populations that are already more subject to disadvantage, surveillance, profiling, or being labelled as ‘persons of interest’”.
It is unclear that these improvements have affected error rates for Māori and Pacific peoples
Māori flagged a specific concern. A Māori interest group set up by the OPC, which is about to publish a position paper on biometrics, spoke against live facial recognition in supermarkets. Issues that arose included Māori already being subjected to over-monitoring and that a disproportionate focus on individuals will lead to active bias against a group.
Also, there are real tikanga Māori concerns about the taking and storage of biometric information. The body parts captured are often considered tapu, including tā moko, mataora or moko kauae.
Is it a silver bullet solution to retail crime?
No, a human element is critical. The OPC emphasised the need for humans to be involved despite the ability of the technology to scan hundreds of millions of people each year. “The technology is only part of the picture.”
Live facial recognition needed to be used alongside a security strategy - “that is, whether and how staff respond if the system triggers an alert”, says the OPC. And “how” is a key question as the OPC points to the intent behind the technology to reduce harmful events - and that depends on how the human element responds.
The OPC also offered a reminder: “It is also still primarily the role of the police as a well trained and equipped organisation with specific statutory powers, to reduce and respond to harmful incidents in stores.”
It offers other cautions. The live facial recognition will identify only those on the store’s watchlist, and that the previous alleged behaviour of those on the watchlist may not adequately predict current or future behaviour. It warns that each separate store needs to make a decision on the worth it will get from such a system - they all need to decide whether this technology is “right for their communities”.
The Office of the Privacy Commissioner raised repeatedly in the facial recognition report that police are the responsible agency to deal with theft and other crimes.
Also, the trial was limited to those who showed “serious harmful behaviour”. Those who committed “low-level shoplifting” were one example of those the OPC says should not be on the watchlist.
But did it work?
The OPC says yes, “a live facial recognition system was an effective way to reduce serious repeat offending during the trial period” although warned the use of live facial recognition and increased contact with customers could create new opportunities for violence.
Foodstuffs NI lead lawyer Julian Benefield pointed to a review of the trial which found 100 serious harm incidents had been prevented (half because trespassed people did not enter the store) and a 16% reduction in harm as a result. There was also a 54% increase in trespass being detected.
That’s not where it ends because the OPC did have concerns at the lack of data produced by the trial. It was told the number of captured images (including repeats) was 225,972,004 but it didn’t know how many “unique faces” had been scanned. It tried to square that number through checkout transactions but the end result was not enough to understand the extent of the privacy impact.
The scanned images and data related alerts, interventions and mismatches were not broken down by skin tone which made it hard to evaluate any bias towards people of colour.
The OPC also raised issue with the way the trial stores self-selected and the quality of the “control” information from stores not using the technology. Those in the trial provided good data while the control stores “became disengaged” which led to “inconsistencies in reporting and comparatively very low incident counts”.
“This affected the overall data quality. In short, it was clear to us that the data need to be treated with a degree of caution.” Its own inquiry, along with Foodstuffs NI’s independent review - hired at the urging of the OPC - got the OPC to a place where it was “confident in our conclusions” that the technology was effective at reducing harmful behaviour, had privacy safeguards to protect people and that Foodstuffs NI was complying with the Privacy Act.
What is expected of those using live facial recognition?
The vibe of the OPC report was one of an expectation that any store using this technology will invest in a framework that will avoid, or at least minimise, the downsides of “overcollection, scope creep, surveillance, misidentification and bias”.
The framework the OPC recommends has humans at its centre. Every step of the way, humans are making key decisions and carrying out ongoing reviews.
In the trial, the live facial recognition system was disconnected from Foodstuffs NI’s usual reporting tool - presumably the Auror crime reporting network. Each live facial recognition system was specific to its location, rather than available across other stores. All access was logged automatically and subject to regular review. It also came with strong record keeping to keep track of how the system was working.
Foodstuffs has been told use of live facial recognition technology requires constant monitoring.
Particularly, the OPC says Foodstuffs NI needs to actively monitor the use of live facial recognition but also the environment it is used in. While initial results from the trial show success in reducing serious crime, “there is no evidence about whether it will continue to be effective or justified in the longer term”.
It also includes making sure the language associated with those on the watchlist is specific to reflect the “serious” alleged offending that led to their inclusion. If triggered by a trespass order, they were urged to drill down into what led to the trespass as not all would qualify for inclusion in a facial recognition watchlist.
Once you’re on a watchlist, how do you get off?
The OPC was strict on how a watchlist operated - as was Foodstuffs NI in its trial. The OPC found the watchlists to be “of reasonable quality and carefully controlled”. There was a separate watchlist for each store, apparently so as not to shut an alleged offender out of all places locally that might sell food.
The watchlist could only be added to by staff with specific training. As noted earlier, inclusion on the list was for serious matters - not “low-level” shoplifting. The OPC defined serious as “physical and verbal assault, violent and threatening behaviour and higher value theft”. To be added to the watchlist, it required two staff members to confirm that the required level had been met.
When it came to keeping people’s information stored, those who didn’t register an alert were instantly deleted. Those who did would be kept for a maximum two years with three months for accomplices.
The watchlist also had those who were not allowed to be added such as children or young people under 18, the elderly and those with “known mental health conditions”.
Will police use the same technology?
Police have so far ruled out the use of live facial recognition. In a recent assessment - August 2024 - it said “the overall risks currently outweigh the potential benefits in the policing context”. It offered the view that it couldn’t proceed until there was better knowledge around the ethical, legal, privacy and security elements of its use.
That’s not to say police officers don’t use facial recognition it because they do - but not as a live tool. Rather, it uses it to match images of unknown offenders on a range of databases to see if a match can be made.
With the publication of the largely positive OPC inquiry, Minister of Police Mark Mitchel and Police Commissioner Richard Chambers were enthusiastic. The statements made by each were effusive about its use by retailers rather than reflecting on any police use. Mitchell talked up crime reduction figures touted by Foodstuffs NI while Chambers enthused about technology as “one of the biggest opportunities we have as a country … when it comes to fighting crime”.
Chambers comments focused on retailers and how best they might use facial recognition for “deterring, detecting and resolving crime”.
The OPC’s reminders of the primacy of police in this space and the police’s enthusiasm for retailers to adopt the technology is an interesting contrast.
Will we see more of this across other stores?
Almost certainly. Live facial recognition technology is highly likely to be commonplace across New Zealand retail spaces.
That’s definitely the aim of Sunny Kaushal, spokesman for the Ministerial Advisory Group for the Victims of Retail Crime.
Justice Minister Paul Goldsmith, centre, with Associate Justice Minister Nicole McKee and Ministerial Advisory Group chair Sunny Kaushal. Photo / Ben Dickens
The advisory group is working up a report for Minister of Justice Paul Goldsmith on the subject although Kaushal has made it clear where its thinking is at. ”Retail crime is a $2.8 billion dollar problem and is paid for by every retailer and customer in New Zealand. Retailers need access to every tool that can help them to keep themselves and their businesses safe - including facial recognition technology."
“Privacy is important, but we need rules that recognise the right of every New Zealander to be safe from violent crime at work.”
Goldsmith was effusive about the Foodstuffs NI trial following the release of the OPC report, calling it great news for those subject to retail crime. “I expect our Ministerial Advisory Group will continue to look at this technology as an option to be used more widely and engage with the sector on it.”
What comes next?
The OPC is preparing to publish its guidance on biometric systems. That will include facial recognition but also consider other means by which people are identified and tracked, from eye scanning through to computer analysis and tracking of the way people walk.
And Goldsmith is expecting to receive a report from the retail advisory group. As indicated above, the sector is bullish over a wider rollout of the technology.
David Fisher is based in Northland and has worked as a journalist for more than 30 years, winning multiple journalism awards including being twice named Reporter of the Year and being selected as one of a small number of Wolfson Press Fellows to Wolfson College, Cambridge. He joined the Herald in 2004.
Sign up to The Daily H, a free newsletter curated by our editors and delivered straight to your inbox every weekday.
