Weizmann AI Rebuilds Images From Brain Scans, and Predicts Brain Activity
Michal Irani and colleagues trained a model on fMRI data from eight people who each viewed about 9,000 images, technologyreview.com reports.
Researchers at the Weizmann Institute of Science have developed an AI tool that reconstructs what a person is looking at from brain scans, and can also predict a person's brain activity based on what they are viewing. The work was led by Michal Irani and colleagues, according to technologyreview.com.
The team trained an AI model on data from eight people who were each shown around 9,000 images inside a high-resolution fMRI scanner. The brain decoder uses two branches: one predicts the structure of the viewed image, while the other predicts its content. A diffusion model then produces the reconstructed image from those two sets of predictions.
The researchers also trained an encoder that works in the opposite direction, predicting brain activity from an image. Used together with the decoder, the two tools improved each other's performance.
Years of work on reading the visual brain
Neuroscientists have spent years trying to reconstruct what people see using fMRI; early attempts produced blurry images, and other researchers had previously collected scans from volunteers shown hundreds of images in scanners. Other teams have built tools to recreate images from brain-scan data, but Irani says those are not accurate enough.
Quick answers
What does the Weizmann Institute's AI tool do?
It reconstructs what a person is looking at from brain scans, and can also predict a person's brain activity based on an image they are viewing.
How was the model trained?
It was trained on data from eight people who were each shown around 9,000 images in a high-resolution fMRI scanner.
How does the brain decoder produce an image?
One branch predicts the image's structure and another predicts its content; a diffusion model then produces the reconstructed image from those predictions.