Photos: Microsoft Kinect pushes natural UI frontier

Computer vision set to change how we interact with all our machines, says Microsoft...

Microsoft has been showcasing how its Kinect computer-vision technology is pushing beyond the boundaries of gaming. Computer vision refers to a research field exploring how computers can extract data from images. Just as our eyes and brain sense and inform us about our immediate environment, a computer equipped with a camera and computer-vision software can identify objects or sense movements in its vicinity.

During a tech demo day at its Seattle offices last week, Microsoft demonstrated how computer vision and Kinect-style natural user interfaces (NUIs) could change how people interact with machines and screens in future. In February, the company announced it will be launching an SDK for Kinect to enable third-party developers to create software for a Kinect motion sensor plugged directly into a Windows PC - potentially bringing NUIs to a monitor near you soon.

"GUI plus NUI is the way people will interact with computers in future," said Craig Mundie, Microsoft's chief research and strategy officer, in a statement.

Pictured above is one of Mircosoft's NUI research projects - called Avatar Kinect - which enables Kinect users' avatars to mimic the head movements, speech and facial expressions of their human owners. Participants can also hang out with other Avatar Kinect users in virtual environments and shoot animated videos to share online, said Microsoft.

In the second generation of Microsoft's Surface touchscreen technology - Surface 2.0 - the table itself is one large camera, so it can recognise objects such as this paint brush, and also identify the direction of users' hands and finger orientation.

Here, a Microsoft researcher uses a handheld digital camera to take a series of 2D images of a statue.

The images of the statue are then automatically converted and rendered into a photo-realistic 3D model of the statue, which can be manipulated on screen. The system does not require a high-end camera to generate the 3D model, even working with a mobile cameraphone.

Here, in a project known as Mirage Blocks, a stereoscopic projector and the Kinect camera are used to render real objects digitally in a 3D space. The virtual objects could then be used for remote collaboration on projects.

This photo-realistic digital talking head is the fruit of another Microsoft research project. The head can be animated in real time so it appears to speak the text that is being typed into the dedicated text box on the left. Microsoft envisages using the technique to create more believable avatars in future.

This demo from Microsoft's Applied Sciences Group shows how touchscreens could evolve. Instead of touching a surface, the user makes gestures above a retro-reflective screen which has a camera and a projector pointing down at it.

Computer-vision algorithms are then used to interpret individual gestures - such as the closed-finger loop pictured above - enabling the user to write, paint or control navigation on top of the screen.

Microsoft envisages various NUI technologies being used in tandem in future. Here, Microsoft's Mundie shows a conceptual demo of various work and social NUI technologies organised into spherical hubs to illustrate how the various techs could be displayed and accessed together.

Microsoft expects mobile phones to be used as the remote control to organise the various work and social hubs, as shown above.

At the Mobile World Congress (MWC) trade show earlier this month, Microsoft CEO Steve Ballmer showed a video of a smartphone running its Windows Phone 7 OS being used in conjunction with the Kinect gaming peripheral - so that multiple players can participate in the same game across different devices.

"2011 is shaping up to be a very fast-paced year," said Ballmer at MWC. "It should be a great year for the development of intelligent mobile devices and the value that they add to society."