Microsoft Kinect opens the 'Recognition' door

Reading this article, on the surface you'll think of us as boys playing with toys.

The actual thing is, Microsoft's new Kinect XBox games controller offers so much opportunity in the retail and business space that we've already thought of uses we could put it to inside Tesco.

Why? Well it's the ability to program it to recognise the people in front of the Kinect device, and possibly, if my imagination takes flight for less than 10 seconds:

• Augmented Reality: Wrap clothes around the body so a customer could understand how an item of clothing looks on them.
• Recognise the 'body signature' through a mix of visual signals (and sound) and thus identify a particular person - and possibly greet them.

So here in Tesco.com R&D we're going to try and create a proof-of-concept that uses the Kinect as a form of person-recognition or play a part in augmented reality.

In its raw state, the incoming data from the Kinect device is itself fascinating in revealing how it goes about identifying body shapes in 3D. First here's Ben Martin (@realbenm) and myself standing in front of the Kinect (photo by Rob D'Amico):

...and here is what it looks like on the computer screen. Note how the Kinect has identified the two humans separately and colours them... oh wait...three humans - that's Rob taking the pic and being identified in the process! The Kinect is just above the laptop screen with a glowing red light (click for larger image).

Although not easy to spot in the photos above, notice that Rob's hand is slightly in front of Ben and his body slightly in front of my leg - but Kinect has correctly kept identifying Rob and not become confused. 

Also notice that the colours are brighter for the hands and arms that are closer to the Kinect device - that means it's detecting our shapes in three dimensions. Clever stuff! If we know how far away any pixel of the body is away from the device, we could use this information to accurately wrap bitmap images such as "clothes" around the body.

Just as fascinating is the fact that Kinect is also able to calculate skeletal geometry. In the image below, not only has it identified three separate people, but on two of them it has managed to create a wireframe skeleton, showing arm, torso, hip and leg positions and movements. Although not shown, the accuracy is maintained if one arm is passed in front of someone else's body image.

So we're up and running. Ben is going to develop this R&D project into a proof of concept which we'll install down in the atrium of Tesco.com HQ so people can have a play when he's done. It'll be interesting to see what he comes up with!

We're using a somewhat unauthorised software development kit right now but I spoke to Microsoft today and they say an official Kinect SDK is in the pipeline and will get a copy to us as soon as they can.

I get a feeling they're rather delighted with the journey we intend to take this project on...!