We already have deepfake AI tools to imitate everyone from Tom Cruise to Jacinda Ardern in social media videos.
Now Zuru co-founder Nick Mowbray is warning that an AI version of himself was used in an attempt to “recreate me” and dupe Zuru’s chief financial officer,Christian Pellone, on a Microsoft Teams video call.
“This is unbelievable. The sophistication is something everyone needs to be aware of. Mind-blowing,” Mowbray warned his followers on LinkedIn this morning.
Mowbray — who told the Herald his AI fake was trying to dupe his CFO into transferring cash — posted screenshots of messages leading up to the appearance of the fake video version of himself, plus a screengrab of Pellone’s explanation.
“When I clicked on it, Nick was there, talking and gesturing to his ear, like ‘can’t hear you’.
Pellone was sufficiently suspicious of the messages to take screenshots of the WhatsApp message exchange and the Teams invite, “but not of the AI because I thought it was real ... It wasn’t a caricature or animation of Nic on Teams. It WAS Nick!”
A screengrab of the messages shows that Mowbray displays a mobile phone number, but it is not his regular WhatsApp number, but Pellone said it could him off-guard because it had a (+64) suffix for NZ and Mowbray’s picture. More, the call was timing was “Late at night so I would be groggy and confused from being woken up, but not so late as to be unrealistic.”
Pellone said in a later debrief, “It was not an avatar or an animation. It was absolutely Nick, wearing clothes that he would typically wear: a cool T-shirt and designer jeans.”
He wasn’t sure how the would-be fraudsters had managed to stitch the AI video into the Teams call (Microsoft has been asked for comment).
In follow-up comments to the Herald, Mowbray said his AI imitator asked his CFO to transfer money — using text because he said he couldn’t get off mute (presumably a jape because the AI wasn’t sophisticated enough for a real-time voice exchange).
But “in the text exchange, he started using language that didn’t sound like me”, Mowbray told the Herald.
The real Nick Mowbray. Probably.
Pellone is now building a case study for the Zuru team “to raise awareness and point to red flags”.
He messaged Mowbray, “I couldn’t distinguish between real you and Teams you. Like it WAS you.”
Agencies such as Netsafe and Cert NZ (the Government’s Computer Emergency Response Team) have already warned that AI has made it faster and easier for fraudsters to create new scams, and eliminate giveaways like bad English or typos. Now, Mowbray — or at least his AI evil twin — has flagged a new, more sinister turn.
The outgoing government passed no AI-related legislation, in contrast to the EU’s AI Act (due to come into force by the end of this year) or US President Joe Biden’s series of Executive Orders on Artificial Intelligence, and neither of the major parties had AI policy going into the election.
A study funded by the Law Foundation noted the rise of deepfake video, but said it could be dealt with under existing laws. Deepfake-specific legislation would risk undermining freedom of expression, it said.
Chris Keall is an Auckland-based member of the Herald’s business team. He joined the Herald in 2018 and is the technology editor and a senior business writer.
