An LCD screen with multitouch and off-screen gestural control

An LCD screen with multitouch and off-screen gestural control

Summary: Researchers at the Massachusetts Institute of Technology have created a working prototype of a bidirectional LCD that allows a viewer to control on-screen objects without the need for any peripheral controllers or even touching the screen. In true Minority Report fashion, interaction is possible with just a wave of the hand.

SHARE:
TOPICS: Hardware
6

Researchers at the Massachusetts Institute of Technology have created a working prototype of a bidirectional LCD (captures and displays images) that allows a viewer to control on-screen objects without the need for any peripheral controllers or even touching the screen. In near Minority Report fashion, interaction is possible with just a wave of the hand.

"This is a level of interaction that nobody's ever been able to do before," says Ramesh Raskar at MIT's Media Lab, who created the prototype along with colleagues Matthew Hirsch and Henry Holtzman, as well as Douglas Lanman at Brown University.

The BiDI Screen, as it's called, is based on LCD technology in which arrays of optical sensors are interlaced with a panel's pixels to detect multiple points of contact with the surface. This enables touch screen interaction.  But to get the screen to see the world in front of it, the researchers displaced the sensor layer of photodiodes behind the liquid crystal layer.  The LCD screen works double duty, switching between display mode and capture mode in real time. In display mode, the LCD functions as normal with backlight and liquid crystals modulating to produce an image. When in capture mode the screen serves as a pinhole array to capture the angle and intensity of light passing to the sensor layer and the backlight is disabled. By correlating data from multiple views across the sensor array, the system images objects (such as fingers) that are located beyond the display’s surface and measure their distance from the display. Computer software then enables gestural motion control of the on-screen objects.

The researchers designed the screen to operate within the limits of consumer off-the-shelf technology and managed to keep a thin form factor despite the additional layer of cameras. The approach resulted in an increased angular resolution as compared to other approaches that use a small number of cameras behind, to the side, or in front of the displays. such as Microsoft's Surface or Project Natal.

The research can potentially unlock a wide array of applications, such as in-air gesture control of everything from CE devices like mobile phones to flat-panel TVs, according to the MIT researchers.

Specifically, as higher-resolution video cameras and LCD screens become available, the researchers say their design should scale to provide high-resolution photographic images — enabling demanding videoconferencing, gaze tracking, and foreground/background matting applications.

The BiDi screen was presented at Siggraph Asia last week, in Yokohama, Japan.

Paper: BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields (9-pages, PDF)

BiDi Screen, 3D gesture interaction in thin screen device from Matt Hirsch on Vimeo.

Topic: Hardware

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

6 comments
Log in or register to join the discussion
  • Apple patented this approach several years ago

    This exact paradigm was invented and patented by Apple several years
    ago. Not sure how these MIT researchers can do much with this
    commercially, as a result.
    Also, you appear to be confusing things. You say:
    "arrays of optical sensors are interlaced with a panel?s pixels to detect
    multiple points of contact with the surface. This enables touch screen
    interaction. But to get the screen to see the world in front of it, the
    researchers added a sensor layer of photodiodes behind the liquid
    crystal layer."

    These "optical sensors" in the first sentence ARE the "photodiodes" in
    the second. All they did is displace this array a short distance, allowing
    a greater depth of focus, and thus allowing for imaging of items not in
    contact with the screen. AFAICT, they are not discrete arrays of
    photodiodes.

    And this is essentially Apple's patent, to interlace the components of a
    CCD array into the active LCD matrix, to make a bidirectional monitor.

    Good luck MIT.
    SpiritusInMachina
    • crApple fanbois take a hike

      First, I'm sure the folks at MIT are aware of crApple's faux patents and have modified their design appropriately.

      2nd, crApple isn't the end-all originator of all ideas. They steal from others just like the rest of the industry. crApple recently patented a 3D tracking idea that Johnny Lee @ Cornell U demo'd a couple years ago. So, I guess it's ok for crApple to ignore prior art, but if anyone else dare step near crApple's pile then watch out!
      chefp
      • They even tried to steal the Apple Logo from

        The Beatles, Jobs claimed he was unaware of the fact, even though at one time he had said he had every Beatles album ever produced. Apple always tries to keep anyone from using their thing, while trying to take any idea they can get their hands on from others.
        mjolnar@...
        • To bad you don't have a clue what you are talking about

          You might want to research your internet memes before you post them
          as declarative statements. You might find out that what you accept as
          fact has no validity. Case in point your idiotic claim above. That is NOT
          what Jobs said. Nor would it have mattered, as:
          1) the trademark was not even REMOTELY similar, beyond the obvious
          fact that they were apples, and
          2) the issue was worked out to the satisfaction of both sides.
          SpiritusInMachina
      • haxorz

        First, using stupid terms like crApple and fanboi means you automatically
        fail. Hell, why not go the whole distance and use leet? The degree of
        immaturity of your post is matched only by its stupidity.
        Faux patents? They either have a patent or they don't. They do.
        More importantly, no one ever said Apple is the "originator of all ideas."
        They are, however, the originator of THIS idea. Regardless of what you
        feel they may have stolen from somewhere else (which is entirely
        irrelevant to the current issue) they did not steal this one. That said, I
        challenge you to post ANY empirical data to support your asinine claim.
        SpiritusInMachina
  • RE: An LCD screen with multitouch and off-screen gestural control

    Thanks DeuseXMachina,

    I edited the text to reflect that the sensor layer was displaced and not added. Apple also filed a patent for a display that changes based on the user's perspective..adding to their raw material for future products.

    Best,
    Chris J.
    christopher_jablonski