Weta helps to 'clone' facial expressions

New technology will allow even more lifelike effects in movies. Photo / Supplied

A Weta Digital researcher, Iain Matthews, has joined university academics in Britain and the United States to develop a way of "cloning" facial expressions.

The computer scientists have challenged assumptions about human behaviour and what influences that behaviour when people communicate by tracking the expressions of individuals during live conversations on a video conference, and mapping the facial movements to models of faces.

Dr Matthews' work, published in the peer-reviewed journal Language & Speech, showed the facial expressions and head movements could be manipulated live to alter the apparent expressiveness, identity, race, or even gender of a talker.

"These visual cues can be manipulated such that neither participant in the conversation is aware of the manipulation," said the researchers, led by Dr Barry-John Theobald, of the University of East Anglia.

The work done by the team, which also included Professor Steven Boker of Virginia University and Professor Jeffrey Cohn of Pittsburgh University, is being trialled by psychologists in the United States to challenge pre-conceived assumptions about how humans behave during conversations.

Researchers have previously shown that speakers will move their head differently when speaking to a woman than when speaking to a man.

The new software has helped show that this difference is not because of the conversational partner's appearance, but because of the way they move.

"If a person appears to be a woman but moves like a man, others will respond with movements similar to those made when speaking to a man," the researchers said.

"Spoken words are supplemented with non-verbal visual cues to enhance the meaning of what we are saying, signify our emotional state, or provide feedback during a face-to-face conversation," said Dr Theobald.

"Being able to manipulate these properties in a controlled manner allows us to measure precisely their effects on behaviour during conversation.

"This exciting new technology allows us to manipulate faces in this way for the first time. Many of these effects would otherwise be impossible to achieve, even using highly skilled actors."

It is likely to have application in the entertainment industry - such as the films and animated video games in which Wellington-based Weta has carved out a leading role - where life-like characters might be required.

Dr Matthews, formerly of the Robotics Institute at Carnegie Mellon University in Pittsburgh, has been working with Weta since 2007 on face and eye tracking for movies, and performance-driven facial animation.

He has a special credit for post-production work on the ground-breaking 3D movie Avatar, to be released by director James Cameron in December.

Weta Digital was founded by director Peter Jackson and Richard Taylor in 1993 and runs its own set of supercomputers.

Last month, it began a working group, Transfx, to bring advanced research together with movie production headed by Sebastian Sylwan, who said one goal was to get computer graphics "closer to reality while maintaining artistic control".

Weta's team writing computer programmes for visual effects will expand to about 20 people, including advanced computer graphics researchers and students, and about half of them will be recruited specifically for the new work.

The paper, "Mapping and Manipulating Facial Expression", was funded by the British Engineering and Physical Sciences Research Council and the US National Science Foundation.

- NZPA