Serious technology behind the silly screen of Snapchat's filters

Snapchat's facial recognition technology is so accurate at times it's scary. Photo / Pinterest
Snapchat's facial recognition technology is so accurate at times it's scary. Photo / Pinterest

Snapchat has made itself a market niche with its filters, or lenses, that can create all sorts of surreal digital contortions of your face and surroundings - even able to switch faces between two people.

And although their practice is quite silly, the engineering behind the filters is quite serious indeed. Developed by a Ukranian tech company called Looksery which Snapchat acquired for $150m they have gone to a lot of effort to put those ridiculous augmented animations on your face.

Snapchat are understandably secretive about their technology but website Vox lifted the lid on its basic premise.

Their technology taps into the growing market of computer vision, which are apps that use pixel data from a camera in order to identify real world objects - such as when you deposit bank cheques using the camera on your phone, or how self-driving cars avoid running over people.

But the technology goes a lot further than that. To identify all the features and expressions of a face Snapchat trained the system using thousands of faces that were manually marked with all the facial points: borders of the lips, nose, mouth, ears etc. So effectively the app makes a point mask of your face and shifts that mask around to match an individual face based on the data it is getting from your camera at 24 frames per second.

The final step is to create a mesh from the point-mask, a mesh that can move with you or trigger an animation when you make certain facial expressions, such as panting like a dog.

And there are more implications to this facial recognition technology than just creating funny faces. Snapchat has even touted a move to sponsored filters which integrate ads into the selfies you take and share with your friends - think of Nike tick eyebrows!

There's also a creepy side to the technology where facial databases are being built without legal impediemnts by the FBI and Facebook, according to Vox.

- NZ Herald

Get the news delivered straight to your inbox

Receive the day’s news, sport and entertainment in our daily email newsletter

SIGN UP NOW

© Copyright 2017, NZME. Publishing Limited

Assembled by: (static) on production bpcf03 at 23 Feb 2017 20:49:46 Processing Time: 557ms