Xbox Kinect: Hack to the future

By Charles Arthur

With eight million controllers sold in just two months, Microsoft's Kinect has proved wildly popular with gamers. But its open source software is inspiring hackers to put the technology to even more inventive use, reports Charles Arthur.

Tom's mates didn't have the heart to tell him the glowing gloves were the real cause of his low scores. Photo / Supplied
Tom's mates didn't have the heart to tell him the glowing gloves were the real cause of his low scores. Photo / Supplied

Christopher Baker spent the day after Christmas as a VJ - video jockey - for a warehouse party. Among the items being used was a Microsoft Kinect controller, normally used to play Xbox 360 games such as Kinectimals or Kinect Sports. But Baker wasn't playing games: the system was rigged up to a computer running software to interpret the movement data being gathered by the system. The software Baker had written monitored how the clubbers were moving and used that to affect the sound, creating a feedback loop between dancers and sound. "It was popular with the clubbers because they could interact directly with the sound and were immersed in the experience," he says.

Baker is one among hundreds of people who have embraced the Kinect not for its potential in gaming (though that is substantial), but because, unusually for an Xbox accessory, its outputs can be put to different - and in some cases revolutionary - uses. In fact, the Kinect may turn out to be the device that ushers in an age of smart machines that can be rigged together and react to humans in useful ways.

Even on its own, the Kinect is an impressive piece of technology. As Microsoft's tagline goes: "You are the controller". It uses an infrared sensor with cameras to sense the movements of players in front of it. Stand in front of a Kinect linked to an Xbox 360, wave your hand and a hand icon on the screen moves too. Start playing a game and it will know if you've jumped for a spike shot in beach volleyball, hit a topspin or thrown a jab in boxing. No wires, nothing - the machine knows. It's an experience that is at first strange and then exhilarating.

This has made the $229 Kinect an incredible success story. It has sold eight million units worldwide since its release in November, making it the fastest-selling consumer electronics item ever, faster than the iPad (the previous record-holder) and its predecessor, the DVD player.

The Kinect solves a problem that has troubled computing experts for years: how to give machines useful vision. Ron Forbes, of Microsoft's interactive entertainment business, says it needed a different approach: "We knew early on [in the Kinect's development] that we had to invent a new way of approaching this problem, one that works like the human brain does." The infrared projector and sensor incorporated into the front of the device turn the world into shades of black (far away) and white (close), which it scans at 30 frames per second and analyses to find the moving parts of the image, by comparing differences between frames. These data are fed into a system tuned to recognise parts of the human body, guessing what moving part is a hand or foot or leg and estimating the movements of each.

The data are then fitted around a schema of a 20-joint skeleton (hands, wrists, elbows, shoulders, feet, ankles, knees, hips, plus two points in the lower back, and the neck and head), allowing you and a partner to be tracked in 3-D, across a range of 1 to 3.5 metres, and an angle of 57 degrees horizontally and 43 degrees vertically. Games designers have built sports, fitness and dance games where gestures decide what happens, but it's the "hacks", not the games, that have people like Baker so excited. The open-source software developed to interpret Kinect output - to a standard USB plug - has already been used for dozens of projects: a team at a US university has created a miniature helicopter that flies itself ; another has made a "virtual piano" on the floor (you play it with your feet); the multiplayer role-playing game World of Warcraft can now be played simply by using gestures, thanks to the University of Southern California's Institute for Creative Technologies, which has written a special software "toolkit". The institute has bigger plans for the future: medical games to help people regain the use of their limbs after a stroke, indoor exercise games and so on.

Even more impressive is the work done by a team at the Massachusetts Institute of Technology (MIT), which amplified the hand detection to recognise hands and fingers and then linked it to a program for scrolling pictures, giving something of the feeling of Minority Report, the film in which Tom Cruise waved computer files and videos around in his search for crime suspects. Watching the MIT video (which is on YouTube - search for Kinect Hand Detection) is eerie if you have seen the film - it's as though the future is unfolding before your eyes.

Yet at first Microsoft seemed unwilling to let people dig under the surface of its software, the company seems to have initially misunderstood what sort of hacking was going on and thought that people were opening up the innards of the machine to get at its proprietary (and highly valuable) chips and software, rather than simply playing around with its output.

But a fortnight after its launch, Alex Kipman, the Xbox's "director of incubation", was insisting the Kinect was open by design and that Microsoft was excited about the idea of people designing new applications for it. At that, the floodgates opened. The possibilities are endless.


Get the news delivered straight to your inbox

Receive the day’s news, sport and entertainment in our daily email newsletter


© Copyright 2017, NZME. Publishing Limited

Assembled by: (static) on production bpcf05 at 26 Apr 2017 12:29:34 Processing Time: 396ms