Windows 10 of the future? Microsoft works on mind control for apps

You still need gestures, but mind-controlled menu selection could make you far more efficient, says Microsoft.
Written by Liam Tung, Contributing Writer

Video: Microsoft expands network to create AR and VR content with a new mixed reality studio

Microsoft has been working on brain-computer interfaces for years. But now it has a patent for a brain-sensing system that could improve the experience of apps when using gesture-based systems on augmented-reality headsets and PCs.

The patent, called ' Changing an application state using neurological data', draws on research into using electroencephalography (EEG) nodes to measure brain activity.

EEG is typically used in a clinical context, with a band placed around a person's head. But Microsoft and others see potential for EEG in consumer tech. Microsoft's patent was filed in May 2016 and awarded this week.

The technology it describes could fit very nicely with Microsoft's HoloLens headset, and though it's not the first to blend EEG with gesture controls, it is thinking of practical ways EEG can enhance computing.

Researchers at UC Berkeley in 2015 created the Mindscape VR using a Muse EEG unit and an Oculus Rift, though it mainly demonstrated the ability to use thoughts to move digital objects in a VR scene.

As per the patent's name, Microsoft is concerned with changing application states using the mind. It explains this approach in the context of 3D modeling software, where actions like click and drag result in different operations depending on an application's state at the time the input is received.


Though HoloLens would lend itself to the mind-control system, Microsoft says it could be used on a PC, laptop, smartphone, or any piece of computing hardware.

Image: Microsoft

For example, click and drag could select and move an object, draw a line, or define the edge on the application.

Download now: IT leader's guide to deep learning (free PDF)

But when a certain action isn't available in a particular application mode, the user needs to go to a menu and switch application states. Users could simply think their way to a new application state rather than manually doing it.

Additionally, in an age where AI is making apps more predictive, Microsoft sees its system detecting when a user needs to change state to complete the next step of a task.

"In some embodiments, the state changes are performed with detected neurological user intention data associated with a particular operation of a particular application state, and changing the application state so as to enable execution of the particular operation as intended by the user and without requiring a user to manually change the application state," the patent says.

"In some embodiments, the application state is automatically changed to align with the intended operation, as determined by received neurological user intent data."

A hint that Microsoft has a HoloLens-like system in mind is its suggestion that the system works in tandem with gesture controls, rather than necessarily using brain waves to replace gestures.

"The computer system detects first neurological user intention data generated when the user is performing the particular gesture to actuate a first operation, and detects second neurological user intention data generated when the user is performing the particular gesture to actuate a second operation."

Microsoft reckons the combined gesture and mind-controls would facilitate efficient application state switching and an overall productivity boost. For example, users could bypass manual menu selection to do different tasks that rely on the same gestures in different contexts.

Among hardware considerations, the main processor will need a 'state machine', a 'state training module' and training profiles located in the system's RAM.

The state machine would receive neurological user intention data from the sensor headband, and translate that into instructions for changing an app's state. The training module also detects brain data and links user intent with application actions.

Microsoft also looks at crowdsourcing neurological data from a network of users, which would enable better predictive features.

Though HoloLens appears to lend itself to Microsoft's mind-control system, Microsoft says it could be used on a PC, laptop, smartphone, or indeed any piece of computing hardware, including but not necessarily the cloud.

Previous and related coverage

Windows Phone 2.0? Microsoft toys with foldable handset design with split camera

Microsoft eyes fat cameras for skinny phones.

Microsoft patent hints at dual-screen foldable tablet

Could these hinges offer Microsoft a new entry into the mobile market?

Microsoft foresees AR tracking your keys, milk, entire life (CNET)

A patent filing by Microsoft hints at tech that would automatically keep tabs on everyday objects so you don't have to.

Microsoft: Here's how Mixed Reality will aid information workers and boost digital transformation (TechRepublic)

At Microsoft Future Decoded, the tech giant explained how mixed reality can help workers and companies achieve more.

Editorial standards