Photos: Microsoft Kinect pushes natural UI frontier

In the second generation of Microsoft's Surface touchscreen technology - Surface 2.0 - the table itself is one large camera, so it can recognise objects such as this paint brush, and also identify the direction of users' hands and finger orientation.

Here, a Microsoft researcher uses a handheld digital camera to take a series of 2D images of a statue.

The images of the statue are then automatically converted and rendered into a photo-realistic 3D model of the statue, which can be manipulated on screen. The system does not require a high-end camera to generate the 3D model, even working with a mobile cameraphone.

Here, in a project known as Mirage Blocks, a stereoscopic projector and the Kinect camera are used to render real objects digitally in a 3D space. The virtual objects could then be used for remote collaboration on projects.

This photo-realistic digital talking head is the fruit of another Microsoft research project. The head can be animated in real time so it appears to speak the text that is being typed into the dedicated text box on the left. Microsoft envisages using the technique to create more believable avatars in future.

This demo from Microsoft's Applied Sciences Group shows how touchscreens could evolve. Instead of touching a surface, the user makes gestures above a retro-reflective screen which has a camera and a projector pointing down at it.

Computer-vision algorithms are then used to interpret individual gestures - such as the closed-finger loop pictured above - enabling the user to write, paint or control navigation on top of the screen.

Microsoft envisages various NUI technologies being used in tandem in future. Here, Microsoft's Mundie shows a conceptual demo of various work and social NUI technologies organised into spherical hubs to illustrate how the various techs could be displayed and accessed together.

Microsoft expects mobile phones to be used as the remote control to organise the various work and social hubs, as shown above.

At the Mobile World Congress (MWC) trade show earlier this month, Microsoft CEO Steve Ballmer showed a video of a smartphone running its Windows Phone 7 OS being used in conjunction with the Kinect gaming peripheral - so that multiple players can participate in the same game across different devices.

"2011 is shaping up to be a very fast-paced year," said Ballmer at MWC. "It should be a great year for the development of intelligent mobile devices and the value that they add to society."