An LCD screen with multitouch and off-screen gestural control
Researchers at the Massachusetts Institute of Technology have created a working prototype of a bidirectional LCD (captures and displays images) that allows a viewer to control on-screen objects without the need for any peripheral controllers or even touching the screen. In near Minority Report fashion, interaction is possible with just a wave of the hand.
The BiDI Screen, as it's called, is based on LCD technology in which arrays of optical sensors are interlaced with a panel's pixels to detect multiple points of contact with the surface. This enables touch screen interaction. But to get the screen to see the world in front of it, the researchers displaced the sensor layer of photodiodes behind the liquid crystal layer. The LCD screen works double duty, switching between display mode and capture mode in real time. In display mode, the LCD functions as normal with backlight and liquid crystals modulating to produce an image. When in capture mode the screen serves as a pinhole array to capture the angle and intensity of light passing to the sensor layer and the backlight is disabled. By correlating data from multiple views across the sensor array, the system images objects (such as fingers) that are located beyond the display’s surface and measure their distance from the display. Computer software then enables gestural motion control of the on-screen objects.
The researchers designed the screen to operate within the limits of consumer off-the-shelf technology and managed to keep a thin form factor despite the additional layer of cameras. The approach resulted in an increased angular resolution as compared to other approaches that use a small number of cameras behind, to the side, or in front of the displays. such as Microsoft's Surface or Project Natal.
The research can potentially unlock a wide array of applications, such as in-air gesture control of everything from CE devices like mobile phones to flat-panel TVs, according to the MIT researchers.
Specifically, as higher-resolution video cameras and LCD screens become available, the researchers say their design should scale to provide high-resolution photographic images — enabling demanding videoconferencing, gaze tracking, and foreground/background matting applications.
The BiDi screen was presented at Siggraph Asia last week, in Yokohama, Japan.
Paper: BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields (9-pages, PDF)
BiDi Screen, 3D gesture interaction in thin screen device from Matt Hirsch on Vimeo.