Rafael Yuste is in his early sixties and bears a more than passing resemblance to Pablo Picasso—if Picasso had worn glasses and had a trim white goatee. Speaking succinctly and methodically, his accent rich with Spanish inflections, he told me about an experiment he had carried out in his lab at Columbia on the brains of mice, and specifically on that part of the cortex that responds to vision. His mentor had been the Swedish neuroscientist Torsten Wiesel, who won a Nobel Prize for his research into how our visual systems process information.
“He discovered by chance that the strongest stimulus is a pattern of high-contrast dark and light bars.” He held up one hand and waved his fingers back and forth. “If you imagine my fingers were bars of light surrounded by complete blackness—if I move my fingers in front of your eyes, that fires up your whole visual cortex.”
To begin with, they used these moving images to train the mice. The bars were projected onto a computer screen in front of them, and when they moved up and down, it was a cue to take a drink from a tube of water. When they moved from side to side, they were to stop drinking. The researchers used a sophisticated laser system to monitor brain activity through the mouse’s skull—identifying exactly which neurons were firing when it was looking at the projected images. “We can see the neurons that are encoding the visual stimulus,” Yuste explains.
Having cracked this neuronal code, Yuste’s group used a second holographic laser system to project a series of points inside the mouse’s brain, with each point activating the very same neurons that represented vertical or horizontal moving bars. “The killer experiment was to turn off the screen,” Yuste says. “Just like when you are playing the piano, you use different fingers on particular keys. So, we are playing the images on the cortex. And when we play them, we make the mouse behave in the way we want it to.” When the team implanted images of bars moving up and down, the mice licked the water. When they implanted images of bars moving side to side, they stopped licking.
In effect, they had read the mind of the mouse, identified exactly what was happening in its brain when it viewed the images—and then used that data to make it see things that were not there.
“The way that the mouse licks the spout when he sees the image that we implanted is identical to when he sees the image with his own eyes. And I mean the same number of licks, the same duration of each lick, the same delay until he starts licking. So, as far as we know, he cannot tell the difference. He thinks that these things are real in front of him.”
It was a clear demonstration, Yuste said, of the power of this new technology—that they could “manipulate the mouse like a puppet” and make it do one thing, or do another, depending on which image they put into its brain.
“And what we can do in a mouse today we can do in a human tomorrow.”
Over the past two decades, researchers using functional magnetic resonance imaging (fMRI), which tracks the iron in the hemoglobin supplying oxygen to neurons, have been building up increasingly detailed maps and inventories of the mammalian cortex. Thanks to huge advances in machine-learning artificial intelligence—computer algorithms that are able to sort through enormous amounts of information and use statistical methods to make classifications and predictions—fMRI scans can now be used to identify everything from depressive thoughts to the nuanced feelings of envy and schadenfreude. Other algorithms have been able to accurately piece together reconstructions of movie clips watched by subjects, just by analyzing their brain scans; or have detected, in probing the brain activity of swing voters in the US presidential election, responding to photographs and videos of presidential candidates, which candidates provoked anxiety or even disgust, and which elicited positive responses or feelings of empathy.
In just the last few years, neuroscience researchers have progressed from decoding images and emotions as they play across the cortex to sounds, words, phrases, and even language. In 2023, in a remarkable demonstration of this emerging technology, a woman called Ann Johnson, who had been paralyzed for 18 years by a brain-stem stroke, was able to speak again through the insertion of a grid of 253 electrodes onto the surface of her brain, which translated her neuronal signals into sentences, in real time, at a rate of 78 words per minute (just about half the speed of standard conversation). The research team at the University of California, led by neurosurgeon Edward Chang, had combined this brain-computer interface with an animated avatar of Johnson’s head, which spoke in her own voice, as reconstructed from a recording of a 15-minute toast she had given at her wedding. Just as the avatar’s mouth spoke Johnson’s words as she thought them, so its expressions were similarly influenced by the nuances of her brain activity, which turned her thoughts about facial gestures into displays of emotion—from smiles to pursed lips and frowns.
“They unlocked her,” Yuste said. “They cloned her mind in a computer. Well, not her whole mind, but this language part. And when they did it, Eddie”—Chang, the study’s lead neuroscientist—“called me up and said, ‘I cannot sleep.’ Because he realized all the power and all the perils. This is incredible for patients that are paralyzed. But imagine you put this on a person for other reasons. There is great responsibility. Look what we have in our hands. We just built you a machine that can decode your language. And in 10 years, we’re going to give you a machine that can interfere with your thoughts the way we do it in mice today.”
All brain-reading technologies work on the same basic principles: They first record the behavior of neurons when a person is engaged in a particular function, such as speech, language, vision, concentration, and so on, to isolate and interpret where this behavior is happening—predominantly expressed through electrical fields, waves, or pulses—and then work out what it means.
The more invasive the recording equipment, the richer and more detailed the data. Surgical interventions are at the vanguard of neuroscience and remain very rare—fewer than 100 people on the planet have brain-computer interfaces like Johnson’s embedded beneath their skulls. Yet almost inevitably, a concerted trickle-down effect is occurring. In the summer of 2023, a team at the University of Texas demonstrated that they could use fMRI to translate brain scans into words and sentences, after subjects listened to 16 hours of the storytelling podcasts The Moth Radio Hour and The New York Times’ Modern Love to train an AI model. When the subjects then listened to new podcasts, the algorithm was able to convert the gist of what they heard, as it manifested in their brains, into words, phrases, and sentences that roughly captured the stories. As the team’s lead computational neuroscientist, Alexander Huth, put it in an interview with Science, “Our thought when we actually had this working was, ‘Oh my God, this is kind of terrifying.’”
Now noninvasive, wearable brain scanners are beginning to proliferate beyond the lab, making their way into our workplaces and, through the vast global consumer market, into our homes too.
In conversation with The New Yorker in 2021, Jack Gallant, a professor in cognitive neuroscience at Berkeley whose work is focused on assembling a “complete functional atlas of the human brain,” talked, in a brief aside, about a possible future technology that he described as “a thinking hat.” He imagined companies paying people $30,000 a year to wear the hat, which could incorporate video-recording glasses along with a variety of sensors, to produce brain data on everything the wearer saw, felt, heard, and experienced as they went about their everyday life. The scientific logic was obvious—just imagine the incredible volumes of n