Neural engineering - movement with first thought / Miguel A. L. Nicolalis

The idea that paraplegics will one day be able to control their limbs with the power of thought alone is no longer just a Hollywood fantasy

The billions of viewers of the opening game of the soccer World Cup, which will be held in Brazil in 2014, may remember it not only for the goals scored by the Brazilian team and the red cards received by its opponent. On that day, my lab at Duke University in the US, which specializes in developing technologies to control robotic limbs using electrical signals from the brain, plans to set a milestone in overcoming paralysis.

If we manage to overcome the enormous obstacles that still stand before us, we hope that a paralyzed boy (or girl), who will take the field wearing a robotic body suit and the competing national teams by his side, will kick the opening kick. This suit, which we call an "exoskeleton", will wrap his legs. His first steps on the field will be carried out by the instruction of movement signals that originate in the boy's brain and which will be transmitted, wirelessly, to a computer unit the size of a laptop that he will carry on his back. The computer will translate the electrical signals from the brain into digital motor commands, so that the exoskeleton can stabilize the kicker's body weight and then coordinate a series of forward and backward movements with the robotic legs that will guide him on the cut grass. As you approach the center of the field, the kicker will imagine the foot coming into contact with the ball. Three hundred thousandths of a second later, signals are fired from the brain to the exoskeleton's robotic leg to squeeze under the ball of skin and launch it up Brazilian-style.

Credit: Camp Ramier

This scientific demonstration of a new and revolutionary technology will prove to billions of viewers that brain control of machines has gone beyond the limits of laboratory demonstrations and futuristic discussions and has reached a new era - an era in which tools actually provide mobility to patients with limited mobility due to injury or disease. We are making progress in collaboration with entities in Europe and Brazil towards a technology that will communicate between the brain and mechanical, electronic or virtual devices, perhaps as early as the next decade. This development will restore mobility not only to victims of accidents and war, but also to patients with ALS (also known as "Lou Gehrig's disease"), Parkinson's and other diseases that interfere with motor actions such as reaching out, grasping, walking and producing speech. Neuroprosthetic devices, or brain-machine interfaces, will allow scientists to achieve much more than just helping the disabled. They will enable exploration of the world in revolutionary ways, giving healthy humans the ability to increase their sensory and motor skills.

In this futuristic scenario, voluntary brain waves, the alphabet at the base of human thinking, will operate robots of all sizes, remotely control airships and perhaps even allow the sharing of thoughts and feelings between humans, in a brain-based collective network.

Machines read minds
The lightweight body suit, intended for an as yet unselected kicker, is still in the development stages. A prototype is currently being built in the laboratory of my good friend and colleague, Gordon Cheng, at the Technical University of Munich. Cheng is also one of the founders of the Walk Again project - an international non-profit collaboration between Duke University's Center for Neural Engineering, the Technical University of Munich, the Swiss Federal Institute of Technology in Lausanne, and the Edmond and Lily Safra International Institute for Science The brain in Natal, Brazil. In the coming months, this international project will be joined by several new members, including renowned universities and research institutes from all over the world.

The project draws on nearly two decades of pioneering work done at Duke University on brain-machine interfaces, research that itself grew out of studies conducted in the 60s, when scientists tried to connect to the brains of animals and see if it was possible to feed a neural signal into a computer and use it to operate a mechanical device. In the 90s and the first decade of this century, my colleagues and I at Duke University developed a method to implant hundreds of hair-thin and flexible sensors, called microwires, into the brains of rats and monkeys. In the past twenty years, we have shown how these flexible spikes are able, once implanted, to detect tiny electrical signals, called action potentials, and produced by hundreds of single nerve cells located in the frontal and parietal cortex - the areas that define a huge brain circuit responsible for creating voluntary movements.
For a whole decade of animal experiments, this interface was used to translate signals from the brain to create movement in robotic arms, hands and legs. The important breakthrough occurred in 2011, when two monkeys in our laboratory learned to control with their minds a virtual computer hand, which touched objects in the virtual world but also returned a feedback signal of "artificial touch" directly to each monkey's brain. With the help of the software, we were able to train the animals to feel the touch of virtual fingers that are directly controlled by their brains.

The Walk Again Association, with the help of an international team of neuroscience researchers, roboticists, computer scientists, neurosurgeons and rehabilitation specialists, began to use these findings, which emerged from the animal studies, and pave an entirely new way to rehabilitate and train severely paralyzed people using brain interface technologies - machine to regain full physical mobility. In fact, the first steps of our future ceremonial kicker will take place in the "automated virtual environment cave" - ​​an advanced virtual reality chamber that has screens installed on all its walls and the floor and ceiling. Our candidate, who must be young and lightweight at this stage of the technology, will wear XNUMXD glasses and a headgear that detects brain waves non-invasively (using electroencephalography (EEG) and magnetoencephalography). He will immerse himself in the virtual environment that will attack him from all sides and learn to control the movements of a virtual body by the power of thought alone. Slowly and gradually, the movements of this body will become more complex, until they reach the level of precise control needed to walk on uneven surfaces, or to open the lid of a virtual jam jar.
connect to the nerve cells

The mechanical movements of the actual exoskeleton are more complicated to perform than the movements of the virtual body, so the technology and training will have to be more complex. To control the robotic limbs it will also be necessary to implant electrodes in the brain. We will not be satisfied with placing the implant under the skull, but we will also have to increase the number of nerve cells that it will "read" at the same time throughout the cerebral cortex. Many of the sensors will be located in the motor cortex - an area in the frontal lobe that is related to the production of motor patterns, which are transferred in healthy people to the spinal cord, from which other nerve cells that coordinate the work of the muscles come out. (Some scientists believe that it would be possible to achieve such an interaction between the brain and the muscles even through a non-invasive method for recording brain activity, such as AEG, but this goal has not yet been achieved in practice).

Gerry Lehew from my group at Duke University has invented a new type of sensor: a receiving cube that, when implanted in the brain, is able to pick up signals from a three-dimensional area of ​​the cerebral cortex. Unlike previous brain sensors, which are built as flat arrays of electrodes whose tips detect electrical nerve signals, Lehio's cube sends thin micro-sensors up, down and sideways out of a central channel.

The current version of our cube contains up to 1,000 recording microwires. Since each wire is capable of recording signals from four or six individual neurons, each cube is capable of receiving the electrical activity of 4,000 to 6,000 neurons. Assuming we can implant a few such cubes in the frontal and parietal cortex, the areas responsible for decision-making and high-level control of movement, we can simultaneously sample tens of thousands of neurons. According to our theoretical software model, such a layout is sufficient to create the flexibility of movement necessary to operate a bipedal exoskeleton and restore independent movement to our patients.

To handle the flow of data coming from these sensors, we are also making progress in developing a new generation of neural chips specially adapted for a specific task. Such chips would be implanted in the patient's skull along with the microelectrodes and extract the raw motor commands required to operate a full exoskeleton.

Of course, the signals received from the brain have to be transmitted to the prosthetic limbs. Tim Henson, who recently completed his doctorate at Duke University, built a 128-channel wireless recording system equipped with chips and sensors that can be implanted into the skull. The system is able to transmit recorded brain waves to a remote receiver. The first version of these neural chips is now being successfully tested on monkeys. In fact, not long ago we watched the first monkey to operate a 2012/XNUMX brain-machine interface that wirelessly transmitted signals from the brain. In July XNUMX, we submitted a request to the Brazilian government, to receive permission to use this technology on humans.

In our future soccer kicker, the data from the recording systems will be transmitted wirelessly to a small computerized processing unit that will be in a backpack. Many digital processors will run various software algorithms, which will translate motor signals into digital commands that can control the moving parts, or motors, which will be scattered over the joints of the robotic suit - hardware parts that determine the position of the artificial limbs in the exoskeleton.

brain power
The commands will allow the wearer of the exoskeleton to walk, slow down, speed up, bend down or go up a flight of stairs. Some of the postural adjustments of the prosthetic hardware will be handled directly by the electromechanical circuits of the exoskeleton, without any neural input. This space suit-like garment would remain flexible and still provide structural support to the body - a replacement for the human spine. The brain-machine interface, which will take full advantage of the combination between control signals from the brain and electronic reflexes coming from the motors, will allow our world cup kicker - so we hope - to move at will in the full sense of the word.

The kicker will not only move using the exoskeleton, but will also feel the ground on which he will step. The exoskeleton will restore a sense of touch and balance with the help of microscopic sensors, which will sense the magnitude of the force generated as a result of a certain movement, and transmit the information back to the brain. The kicker should be able to feel the contact made between the toe and the ball.

Our ten years of experience in building brain-machine interfaces show that as soon as the kicker begins to interact with the exoskeleton, his brain will begin to integrate the robotic body as an actual extension of his body perception. The accumulated experience from this continuous feeling of contact with the ground and the positioning of the robotic legs will allow it to move with fluid steps, on the court or on a sidewalk. All stages of the project require strict and continuous testing in experiments on animals before it is possible to switch to using them on humans. Also, all procedures must be approved by the regulatory authorities in Brazil, the USA and Europe to ensure proper scientific and ethical control. Despite all the uncertainty, and the short time needed to complete the first public demonstration, the simple idea of ​​such an ambitious goal resulted in an almost unprecedented awakening of interest in science in Brazilian society.

Remote control
The opening kick of the World Cup – or, if we miss the deadline, a similar event such as the Olympic and Paralympic Games in Rio de Janeiro in 2016 – will be more than a one-off stunt. A hint of what will be possible to achieve with the help of this technology can be obtained from a two-stage experiment in monkeys that has already been completed. In 2007, our research team at Duke University trained rhesus monkeys to walk upright on a treadmill while measuring the electrical activity of more than 200 cortical neurons. At the same time, Gordon Cheng, who was then at the ATR Labs for Intelligent Robots and Communications in Kyoto, Japan, created an ultra-fast internet protocol that allowed us to transmit the neural data stream directly to Kyoto. The data was fed into the electronic controllers of the CB1 humanoid robot. In the first half of this world-wide experiment, Cheng and my group tested previously developed algorithms for translating thoughts to control robotic arms. The experiment showed that these algorithms can translate patterns of neural activity involved in bipedal locomotion into the walking of a pair of mechanical legs.

The results of the second half of the experiment were much more surprising. While one of our cashiers, Idoya, was walking on the treadmill in Durham, North Carolina [home of Duke University], the brain-machine interface we created transmitted a continuous stream of her electrical activity data through Cheng's Internet connection to Kyoto. The CB1 robot recognized the motor commands and started walking too, almost immediately. At first he needed a little support at the waist, but in later experiments he began to move independently in response to commands produced by the Kopa's brain on the other side of the world.

What's more, even when the treadmill stopped and Idoya stopped walking, she was still able to control CB1's leg movements in Kyoto, simply by observing the moving legs through real-time video capture and thinking about every step CB1 had to take. She continued to generate the brain patterns necessary to make the robot walk, even when her own body was not performing the motor action. This demonstration of a transcontinental brain-machine interface showed that a human, or a monkey, could easily transcend space, force, and time and free the brain's commands from the physical limitations of the biological body in which the brain resides and send them to a man-made device located far from the thought that produced the original movement. .

From these experiments it appears that with the help of brain-machine interfaces we can operate robots in environments that humans will never be able to reach with their bodies. Our thoughts could operate microscopic surgical tools inside a human body, or control a humanoid robot trying to fix a leak in a nuclear power plant.

The interface will also be able to control tools that exert much more force, or much less, than our body exerts, thus breaking the everyday limitations on the force a person can exert. The connection between a monkey brain and a humanoid robot has already eliminated the limitations of the clock: Edoya's mental trip across the world took 20 milliseconds - less than the time it takes her to move one of her shoes.

The work we have done with the monkeys gives us not only an inspiring vision for the distant future, but also confidence that our plan is feasible. As of the time of writing this article, we are waiting to see if the International Football Federation (FIFA), which is responsible for organizing the ceremony, will agree to our proposal to let a young man paralyzed in his legs participate in the opening ceremony of the 2014 World Cup games. The Brazilian government, which is also waiting for FIFA's answer, supports our request in principle.

There are bureaucratic and scientific difficulties facing the realization of our vision, but I can't stop imagining what it will be like, when three billion people will watch a short and historic march on the green grass of the field - when a paralyzed young Brazilian man will stand up, walk again under his (or her) will, kick a ball and score a goal Unforgettable for science in a country that specialized in this great game.

About the author
Miguel A. L. Nicolelis is one of the pioneers in the field of neural prosthetics. He is a professor of neuroscience at Duke University School of Medicine, and one of the directors of the university's Center for Neural Engineering.

Summary
Brain waves can control the movement of a computer marker, robotic arms, and soon also an entire suit: an exoskeleton that will allow the paralyzed to walk and perhaps even move lightly.
Sending signals from the external cerebral cortex for the purpose of producing movement in the exoskeleton is an advanced product of several bioelectrical technologies, which have been perfected in recent years.
The opening match of the 2014 FIFA World Cup in Brazil will serve as a stage to demonstrate the use of a mind-controlled exoskeleton, if a disabled youngster kicks the ceremonial kick-off as planned.

Timeline
The long road to mind-controlled prostheses
Replacement limbs have been known for thousands of years, as a rational response to the need arising from war wounds, birth defects and other injuries. Today, the technology is so sophisticated that it is possible to control an artificial organ using electrical signals transmitted directly from the brain.
1500-1000 BC
The first historical mention
A sacred Hindu book, written during this period, mentions Vishpala having her leg amputated after an injury in battle. The leg was replaced with an iron prosthesis, which allowed her to return to the army.
Fourth century BC
An ancient find
One of the oldest artificial organs discovered, a copy of which is shown here, was found in excavations in southern Italy in 1858. It was created around 300 BC, from copper and iron, and was probably intended for a person whose leg was amputated below the knee.
Century 14
cannons and amputations
The appearance of gunpowder on the war fronts in Europe greatly increased the number of wounded soldiers. In the 16th century, Ambroise Pare, the royal physician to several French kings, developed techniques for attaching upper and lower limbs to casualties, and brought back the use of ligatures to stop the flow of blood.
1861-1865
The American Civil War
The war between the North and the South caused many amputations. One of the stumpers was Brigadier-General Stephen Joseph McGroarty. Extensive government funding and the availability of anesthetics, which allowed for longer surgeries, improved prosthetic technology during this period.
1963
Primitive brain interface
Jose Manuel Rodríguez Delgado implanted a remote-controlled electrode in the caudate nucleus, deep inside a bull's brain, and made it stop running with the push of a button. This device preceded modern brain-machine interfaces.
1969
Pioneering experiments
Eberhard Petz of the University of Washington performed an experiment in which monkeys were trained to activate electrical signals in the brain to control the activity of a single neuron, which was measured using a metal microelectrode.
The 80 years
listen to brain waves
Apostolos Georgiopoulos of Johns Hopkins University discovered a pattern of electrical activity in the motor neurons of rhesus roach monkeys that appeared when they turned their hands in a certain direction.
The early 90s
Connecting
John Chapin, now at the University of the State of New York, and Miguel A. L. Nicolalis developed a technique that allowed the simultaneous recording of dozens of separate neurons using permanently implanted electrodes, thus paving the way for research on brain-machine interfaces.
1997
Improved movement
The C-LEG knee prosthesis appeared, which is controlled by a microcontroller, and which in its current version allows the user to activate customized settings for activities such as cycling.
1999-2000

positive feedback
The labs of Chapin and Nicolais published the first description of a brain-machine interface, activated by the brain activity of rats. The animals experienced the movement through a visual feedback signal. A year later, Nicolais' lab published the first study in which a monkey controlled the movements of a robotic arm using only brain activity.
2008-2012
Blade Runner
Oscar Pistorius [known as the Blade Runner] did not meet the criteria for the 2008 Summer Olympic Games, but won big in the Paralympic Games, reaching the semi-finals in the 400 meters at the 2011 World Athletics Championships in Daegu, South Korea. [Pistorius participated in the London Olympics in 2012 and the Paralympic Games that followed - the editors].
2011
The monkey thinks, the virtual character executes
Nicolalis' team at the Center for Neural Engineering at Duke University showed how a monkey is able to use thoughts to manipulate the movements of a virtual character (avatar).
2012
From my brain to my robotic arm
John Donohue of Brown University and his colleagues have shown that a person with a brain implant is able to manipulate a robotic arm to pick up a drink using the BRAINGATE neural interface system.

More on the subject
Controlling Robots with the Mind. Miguel AL Nicolelis and John K. Chapin in Scientific American, Vol. 287, no. 4, pages 46-53; October 2002.
Cortical Control of a Prosthetic Arm for Self-Feeding. Meel Velliste et al. in Nature, Vol. 453, pages 1098-1101; June 19, 2008.
Beyond Boundaries: The New Neuroscience of Connecting Brains with Machines-and How It Will Change Our Lives. Miguel Nicolelis. St. Martin's Griffin, 2012.
2014

Cyborg opening kick
Nicolais' laboratory intends to produce an exoskeleton for a disabled young man, who will kick the opening kick of the soccer World Cup in Brazil.