Apple Watch: ToyTalk

The Winston Show in the sci-fi chapter.

The last developer I was able to visit in San Francisco was the little ToyTalk shop. Around 50 per cent staffed by ex Pixar people, the company is focussed on creating animated characters that actually talk back to kids, involving them in games and scenarios that change according to the spoken input they get. The skills base here isn't just animation, sound recording and coding: various ToyTalk staff members have voiced all the characters, too.

If Talking Tom immediately springs to mind, this is a different company completely. As clever and engaging as Talking Tom (and all its offshoots) is, they just mimic back what you say back to you with a funny voice.

Toy Talk is a whole different proposition, and it's created by a bunch of very dedicated people. The first major offering was The Winston Show but don't let thoughts of a certain NZ politician dissuade you. This Winston Show is presented like a TV series on iPad, except that the host talks directly to you. He talks back. Kids have real conversations with the animated characters. This adds a magical new dimension of make-believe play for children aged 6-8. The Winston Show has six animated, conversational characters and the player is included in the play, thanks to the iPad camera, and gets to wear dozens of costumes to join in the fun across four different worlds. It's a talk show on iPad and the characters talk back. As Oren says, this is "role playing and make believe for real." And it's free, and carefully developed: The Winston Show was voted 2013's Most Innovative Kids' App.

A second product from ToyTalk (various other things are in development) is SpeakAZoo, in which animals from vampire bats to dolphins also talk back. This is also free, becoming available two months ago.

There are 20 original characters in four habitats ranging from the Serengeti, forests, tropics and underwater.

The talking is all handled by internet connections, with the answers supplied by server-based audio routines. Both apps capture audio recordings necessary to create the play experience, but ToyTalk is very serious about safety and privacy. To put you in control of your child's recordings (and as outlined in applicable US privacy laws) ToyTalk requires parents to create an account for secure log-in and management of a family account and all the data associated with it.

ToyTalk requires parents to create an account for secure log-in and management of a family account and all the data associated with it.

ToyTalk was founded by Oren Jacob, who left his role as CTO at Pixar after working at the studio for over 20 years. Oren did a brief stint as Entrepreneur In Residence at August Capital, then announced he was off to start ToyTalk: talking for entertainment. As he put it to us, the idea came from his daughter. She was playing make believe with various toys, pretending to use Oren's iPhone to talk to them, and she said she only wished they could talk back...

Martin Reddy is co-founder (he has a PHD in Artificial Intelligence). Reddy formerly led Pixar's software development teams on several movies, including Finding Nemo and WALL-E.

The first result of ToyTalk's efforts was the Winston Show, which basically offers a bunch of animated skits. Winston might ask what your favourite colour is. ToyTalk has recorded clever retorts, and if they don't fit, offers bits of trivia around the subject instead.

At ToyTalk's backend, missed answers are noted. If enough kids say the same answer, ToyTalk knows they've missed something, and adds new response dialogue accordingly. There are several thousand lines of dialogue on tap.

All these voice responses are uploaded to, and processed in, the cloud, so the apps require constant internet connections, please note. Check out the video teaser here.

Kids might find all this magical, but it's the underlying technology that interests developers - some is being (or has been) patented by ToyTalk. The apps use artificial intelligence and voice recognition to build a framework, solid steps towards full-on conversational technology that may, one day, have all our machines talking back to us. The ToyTalk tech learns from every response it gets, using a proprietary technology called PullString to design a two-way conversation via animation, recording and the response process. PullString is a professional audio tool for dialogue,
The Winston Show launched with six hours of content but has had more hours added since. It's currently running at 17 hours. The ToyTalk team is constantly working to add in more, while developing new products. Across the two titles over 50 characters have already been built, all with different animations, voices and responses.

ToyTalk has a creative half that produces the work‚ writes the stories, creates the animations, voice acting, audio recording - and the tech half, which makes it all work. We also met Frank Clary, ToyTalk's senior sound engineer, who showed us some of the sound work involved.

SpeakAZoo i's currently only available in Australia, the US, Canada and New Zealand. The reason for the limited geographical spread so far is that non English-speaking countries need far more than just translation. Responses have to be believable, although ToyTalk staff members are interested in the idea of ToyTalk's apps in countries where English as a second language might benefit from the game-based, fun learning environment that Winston and SpeakAZoo provide. The third party (and secret) speech recognition engine handles a lot of different English dialects and accents - even ours - but heavily accented kids' English might need some testing. Educational uses are also possible, but ToyTalk's Head of Communications Christine Schirmer pointed out they would need considerably input from educative professionals and child development experts before this aspect could be embarked upon seriously. Like other staffers, she has an impressive resumé: she has worked at Amazon, Digg, Facebook and Netflix, but started her career at Apple where she helped to launch the iPod, iTunes and over 20 global retail stores.

Everything above the speech recognition has been built by ToyTalk; companies have been coming to them to see if they can use aspects of the tech for various other apps.

Kids' stuff? At the consumer level, sure. Behind that: brilliant and exciting.