Scientists read minds with brain imaging tech

Scientists have worked out how to reconstruct images a person is viewing by studying their brain activity.

The breakthrough, which the University of California at Berkeley scientists say represents a significant step in the ability to decode what is going on inside the brain when exposed to rapidly changing visual stimuli, was announced on Thursday.

https://www.youtube.com/watch?v=nsjDnYxJ0bo Credit: The Gallant Lab, UC Berkeley What a person sees can be reconstructed using an algorithm and a database of imagery

"This is a major leap toward reconstructing internal imagery," Jack Gallant, a neuroscientist and co-author of the Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies study, said. "We are opening a window into the movies of our minds."

By studying changes in the brain when it was watching clips from movies, the scientists were able to develop an algorithm that could help them decode the observed image.

Three subjects spent several hours watching two sets of Hollywood movie trailers while functional Magnetic Resonance Imaging (fMRI) machines monitored activity in their brains. Scientists studied how blood flowed through the brain's visual cortex and used the data to create a three-dimensional simulation of the brain made up of volumetric pixels — voxels.

Data from the first set of trailers was fed to a computer program which learned to associate the visual patterns of the movies with the activity going on in the subjects' brains. Scientists then studied the data from the second set of trailers and fed it to an algorithm which reconstructed the images it associated with the brain activity by trawling 18 million seconds of random clips from YouTube. The clips did not include any of the images used in the trailers and the output bore a striking resemblance to what the Hollywood clips had portrayed.

"Reconstructing movies using brain scans has been challenging because the blood flow signals measured using fMRI change much more slowly than the neural signals that encode dynamic information in movies," the researchers said. They were able to surmount this problem by building a simulation of how the neurons were expected to react and then combining this data with the study of how blood moved through the visual cortex.

In the future, the techniques could be used to give a visual proxy for dreams and other internal mental images of other people, the scientists said. "As long as we have good measurements of brain activity and good computational models of the brain it should be possible in principle to decode the visual content of mental processes like dreams, memory, and imagery," the academics wrote on the Gallant Lab at UC Berkeley's website.

More work needs to be done before this is possible. "It is currently unknown whether processes like dreaming and imagination are realised in the brain in a way that is functionally similar to perception," the scientists wrote. "If they are, then it should be possible to use the techniques developed in this paper to decode brain activity during dreaming or imagination."