This exact paradigm was invented and patented by Apple several years
ago. Not sure how these MIT researchers can do much with this
commercially, as a result.
Also, you appear to be confusing things. You say:
"arrays of optical sensors are interlaced with a panel?s pixels to detect
multiple points of contact with the surface. This enables touch screen
interaction. But to get the screen to see the world in front of it, the
researchers added a sensor layer of photodiodes behind the liquid
crystal layer."
These "optical sensors" in the first sentence ARE the "photodiodes" in
the second. All they did is displace this array a short distance, allowing
a greater depth of focus, and thus allowing for imaging of items not in
contact with the screen. AFAICT, they are not discrete arrays of
photodiodes.
And this is essentially Apple's patent, to interlace the components of a
CCD array into the active LCD matrix, to make a bidirectional monitor.
Good luck MIT.
Researchers at the Massachusetts Institute of Technology have created a working prototype of a bidirectional LCD (captures and displays images) that allows a viewer to control on-screen objects without the need for any peripheral controllers or even touching the screen. In near Minority Report fashion, interaction is possible with just a wave of the hand.
“This is a level of interaction that nobody’s ever been able to do before,” says Ramesh Raskar at MIT’s Media Lab, who created the prototype along with colleagues Matthew Hirsch and Henry Holtzman, as well as Douglas Lanman at Brown University.
The BiDI Screen, as it’s called, is based on LCD technology in which arrays of optical sensors are interlaced with a panel’s pixels to detect multiple points of contact with the surface. This enables touch screen interaction. But to get the screen to see the world in front of it, the researchers displaced the sensor layer of photodiodes behind the liquid crystal layer. The LCD screen works double duty, switching between display mode and capture mode in real time. In display mode, the LCD functions as normal with backlight and liquid crystals modulating to produce an image. When in capture mode the screen serves as a pinhole array to capture the angle and intensity of light passing to the sensor layer and the backlight is disabled. By correlating data from multiple views across the sensor array, the system images objects (such as fingers) that are located beyond the display’s surface and measure their distance from the display. Computer software then enables gestural motion control of the on-screen objects.
The researchers designed the screen to operate within the limits of consumer off-the-shelf technology and managed to keep a thin form factor despite the additional layer of cameras. The approach resulted in an increased angular resolution as compared to other approaches that use a small number of cameras behind, to the side, or in front of the displays. such as Microsoft’s Surface or Project Natal.
The research can potentially unlock a wide array of applications, such as in-air gesture control of everything from CE devices like mobile phones to flat-panel TVs, according to the MIT researchers.
Specifically, as higher-resolution video cameras and LCD screens become available, the researchers say their design should scale to provide high-resolution photographic images — enabling demanding videoconferencing, gaze tracking, and foreground/background matting applications.
The BiDi screen was presented at Siggraph Asia last week, in Yokohama, Japan.
Paper: BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields (9-pages, PDF)
BiDi Screen, 3D gesture interaction in thin screen device from Matt Hirsch on Vimeo.
