Abstract
Objective. Present day cortical brain–machine interfaces (BMIs) have made impressive advances using decoded brain signals to control extracorporeal devices. Although BMIs are used in a closed-loop fashion, sensory feedback typically is visual only. However medical case studies have shown that the loss of somesthesis in a limb greatly reduces the agility of the limb even when visual feedback is available. Approach. To overcome this limitation, this study tested a closed-loop BMI that utilizes intracortical microstimulation to provide 'tactile' sensation to a non-human primate. Main result. Using stimulation electrodes in Brodmann area 1 of somatosensory cortex (BA1) and recording electrodes in the anterior intraparietal area, the parietal reach region and dorsal area 5 (area 5d), it was found that this form of feedback can be used in BMI tasks. Significance. Providing somatosensory feedback has the poyential to greatly improve the performance of cognitive neuroprostheses especially for fine control and object manipulation. Adding stimulation to a BMI system could therefore improve the quality of life for severely paralyzed patients.
Export citation and abstract BibTeX RIS
Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Introduction
During the last decade, considerable progress has been made in research on upper extremity cortical neuroprostheses. Different cortical areas have been utilized as sources for providing control signals for neuroprosthetics which typically control computer cursors or robotic limbs. For these neuroprosthetic applications, neural signals can be recorded from motor cortex to provide continuous control of trajectories (Serruya et al 2002, Taylor et al 2002, Carmena et al 2003, Santhanam et al 2006). More cognitive neural signals can be extracted from posterior parietal cortex (PPC) to provide both goal and trajectory information (Musallam et al 2004, Hwang and Andersen 2009, 2012, Hauschild et al 2012). However, most neural prosthetics research has focused on closed-loop control in which vision is the feedback signal to the subject for computer cursors (Serruya et al 2002, Taylor et al 2002, Hochberg et al 2006, Kim et al 2007, 2008, 2011, Truccolo et al 2008, Simeral et al 2011) and robotic devices (Wessberg et al 2000, Carmena et al 2003, Aaron et al 2006, Velliste et al 2008, Hochberg et al 2012). The almost complete absence of somesthetic information provided by current upper extremity prostheses severely limits their usability, particularly for the on-line control of robotic hands for grasping and object manipulation (Fagg et al 2007, Johansson and Flanagan 2009, Lebedev et al 2011).
One approach for providing missing information from the prosthetic's contact with objects is sensory substitution whereby an intact sensory system such as vision, hearing or cutaneous sensation elsewhere on the body is used as an input channel for information related to the prosthesis (Riso 1999). But none of these sensations feels natural and subjects must learn to translate and utilize input that is not direct (Marasco et al 2011). For amputee subjects, natural sensation of missing limbs can be provided by stimulating peripheral afferent nerves in the limb's stump with intrafascicular electrodes (Dhillon and Horch 2005, Horch et al 2011) or cuff-like electrodes (Tyler and Durand 2002). For subjects who have had targeted reinnervation surgery, sensation of the limb can be provided by touching the part of the skin which is reinnervated as a consequence of surgically redirecting nerves that once served the lost limb (Kuiken et al 2007). However, none of these techniques will work for quadriplegic patients who have damage at a high level of the spinal cord. A plausible alternative is to directly stimulate the neurons in the corresponding intact somatosensory cortex which normally receives sensory signals from the limb. This stimulation is direct in the sense that sensors on the robotic limb provide input to topographically matching locations in the somatotopic map in cortex, with primary somatosensory cortex (S1) being a suitable target (Libet 1982, Ojemann and Silbergeld 1995).
Studies have demonstrated that somatosensory percepts can be elicited by both epicortical stimulation (Penfield and Boldrey 1937, Penfield and Rasmussen 1950, Penfield and Jasper 1954, Libet 1982, Richer et al 1993, Ojemann and Silbergeld 1995) or intracortical microstimulation (ICMS) (Romo et al 1998, 2000, Fitzsimmons et al 2007, O'Doherty et al 2009, 2011, 2012). Compared to epicortical stimulation, ICMS provides a more viable option for restoring sensory capacities. ICMS employs penetrating microelectrodes which produce more punctate activation than surface contact electrodes (Cogan 2008). Microelectrode arrays can be chronically implanted into the cortex and stay functional for a long period of time (Hathaway and McKinley 1989, Rousche and Normann 1999, Santhanam et al 2006, Parker et al 2011, Torab et al 2011, Berg et al 2013). Also, as mentioned above, using microelectrodes allows stimulation of a small volume of tissue, which should, with a sufficient number of electrodes, improve the selectivity and spatial resolution of functional responses compared to macroelectrode alternatives.
Two groups of scientists have demonstrated preliminary evidence of ICMS in non-human primates (NHPs) being useful for providing somatosensory feedback for an upper limb neuroprosthetic. Nicolelis and his colleagues reported the operation of a bidirectional BMI that provided artificial tactile feedback to rhesus monkeys through ICMS of S1 (O'Doherty et al 2011). In that study, control signals were derived from single unit activity recorded from primary motor cortex and were used to control a virtual-reality arm in a 2D environment. Artificial texture of different objects was conveyed to the animal via different ICMS patterns which were found to facilitate perception and minimize the detrimental effect of stimulation artifact on recorded brain signals. The results of two studies from Bensmaia's group showed ICMS patterns could be precisely tuned to provide somatosensory feedback in an intuitive way (Hathaway and McKinley 1989, Berg et al 2013). One study implemented a somatosensory prosthesis which could intuitively convey information about contact force to the subject (Berg et al 2013). The other study developed approaches to intuitively convey information about contact location, pressure, and timing through ICMS (Hathaway and McKinley 1989). Both studies provided the evidence that sensory experience induced by ICMS was comparable to that caused by mechanical stimuli.
Here we present data from a study designed to show that tactile sensation of a virtual prosthetic limb can be fed back to a NHP subject to guide the movement of a virtual prosthetic limb via ICMS in Brodmann's area 1 (BA1) of primary somatosensory cortex. We tested the somatosensory feedback in scenarios in which vision was either not helpful or was only partially helpful for performing a task. We also compared performance of tasks using somatosensory feedback versus tasks using auditory feedback as a sensory substitute. Additionally, we explored the possibility of closing the loop for the cortical neuroprosthesis by coupling stimulation evoked somatosensory feedback with real-time brain control of the prosthetic limb. In this case, high-level cognitive signals from the PPC (Musallam et al 2004) were employed as control signals for the virtual prosthetic limb. Our previous work in PPC showed brain control with spiking data (Hauschild et al 2012). This current study for the first time adds stimulation feedback and also uses broadband multiunit activity (MUA) for brain control. This study demonstrates that the movement of a virtual prosthetic limb can be controlled by signals recorded from PPC while ICMS artifacts are filtered out.
Methods
Approvals
We obtained approval for the animal use protocol in this study from the Caltech Institutional Animal Care and Use Committee. All experimental procedures are in compliance with the guidelines of the National Institutes of Health Guide for the Care and Use of Laboratory Animals.
Prosthetic system with a somatosensory feedback loop
We developed a neural prosthetic system consisting of a motion tracking system, cortically implanted electrode arrays, a neural data processing system, a neurostimulation system, and a virtual modular prosthetic limb (vMPL) which runs in a three dimensional (3D) virtual reality environment (VRE) (figure 1). All tasks were designed in the VRE environment and were presented to the animal in stereoscopic 3D via shutter glasses. The monkey controlled the vMPL either by motion of his own right limb through a motion tracking system (trakSTAR, Ascension Technology Corporation, Milton, VT) or by decoded brain signals through a neural data processing system (indicated by the dashed arrow in figure 1). Specific circumstances determined by the task (e.g. the virtual hand touching a target object) triggered the neurostimulation system to stimulate the monkey's somatosensory cortex (S1). The monkey used the system in a total of 119 study sessions (over a period of 13 months). The components of this system are described in detail in the following sections.
Figure 1. Schematic diagram of the neural prosthetic system. Collisions of target objects and the virtual hand in the VRE triggered the neurostimulation system to stimulate the monkey's somatosensory cortex. The prosthetic limb was driven by either neural data or motion tracking data depending on the task.
Download figure:
Standard image High-resolution imageImplantation of electrode arrays
One Utah electrode array (UEA) (CerePort Array, Blackrock Microsystems, Salt Lake City, UT) was implanted in S1 and used to convey electrical stimulation currents generated by the neurostimulation system for somatosensory feedback. The UEA array consists of 100 microelectrodes (1.5 mm in length) arranged in a 10 × 10 grid on a 4 mm × 4 mm silicon base that is 0.25 mm thick. Each microelectrode is insulated with Parylene-C polymer and is electrically isolated from neighboring electrodes by non-conducting glass. Each microelectrode has a tip that is coated with sputtered iridium oxide film, allowing for stable neural recordings as well as electrical stimulation. Of the 100 electrodes, 96 are wire bonded using 25 μm gold alloy insulated wires collectively sealed with a silicone elastomer. The wire bundle is potted to a printed circuit board with epoxy, the printed circuit board is inserted into the Patient Pedestal (percutaneous connector), and then the Patient Pedestal is filled with silicone elastomer. Two fine platinum reference wires are also attached to the Patient Pedestal.
Four microwire-based electrode arrays (Floating Microelectrode Arrays (FMAs); MicroProbes for Life Sciences, Gaithersburg, MD) were chronically implanted in the PPC to record neural activity from cortical neurons. Neural data were then transmitted to the neural data processing system where they were processed to decode movement intentions. The FMAs (figure 2) each contain 34 microwire electrodes (1.4–7.1 mm in length) uniformly arranged in a 4 mm × 1.8 mm alumina ceramic base that is <0.9 mm thick. Thirty two of these electrodes are used for recording and the remaining two provide a within-array reference. Each microwire recording electrode is insulated with Parylene-C while each reference electrode is uninsulated. All 34 electrodes are bonded to Omnetics connectors housed within a titanium percutaneous connector using Parylene-C polymer insulated 25 μm gold wires that are collectively sealed within a silicone elastomer. The Omnetics connectors are affixed to an in-house designed Percutaneous connector with epoxy and the Percutaneous connector is sealed with silicone elastomer. The FMA pedestal is designed and manufactured to be biocompatible (titanium/silicone base, small circumference wound margin, and rounded legs with flush mounted screws) (Huang et al 2008).
Figure 2. Side view of a 32 channel FMA array (left) and a 128 channel percutaneous connector which can connect to up to four FMA arrays (right). The arrays on the right are embedded in low temperature wax that is melted away prior to insertion.
Download figure:
Standard image High-resolution imageAseptic surgery was performed according to Caltech-approved IACUC protocols. A biocompatible titanium head holder (Gray Matter Research, LLC) for stabilizing the head was initially affixed to the skull prior to the array placement (Adams et al 2007). One UEA array and four FMA arrays were then implanted stereotaxically using pre-surgery anatomical magnetic resonance imaging scans to guide the implantation (figure 3). Two percutaneous connectors, one for stimulation and one for recording, were affixed to the skull with bone screws and acrylic. The UEA array was implanted in the hand representation of Brodmann's area 1 located in S1. The four FMA arrays were implanted in PPC (two in area 5, one in the parietal reach region (PRR) and one in the anterior intraparietal area (AIP)). The UEA array was inserted with a pneumatic inserter (Blackrock Microsystems, Salt Lake City, UT). The FMA arrays are inserted using a custom, vacuum based stereotaxic inserter (Rizzuto et al 2006).
Figure 3. Top view surface reconstruction of an MRI image of the monkey cortex with superimposed approximate array placements. The yellow lines indicate the central sulcus (CS) and intraparietal sulcus (IPS). The image was obtained after head holder implantation (a strong distortion artifact of the four legged head holder can be seen in the top half of the image) but before array placement.
Download figure:
Standard image High-resolution imageReceptive field mapping
During the first days post implantation we determined which UEA electrodes would be used for stimulation via an initial mapping procedure. We mapped the sensory receptive fields of the multiunits recorded from each electrode. Mapping was done manually while the animal was awake. The animal was trained to remain still while we manipulated his extremities and gave him liquid reward at regular intervals. The animal's hand was systematically probed with a cotton swab while the MUA on each electrode was observed to determine the respective receptive fields (figure 4). If a recorded multiunit cluster was modulated while we were probing the hand (i.e. brushing and poking) we narrowed down the probing area. All multiunits we characterized increased their firing rate when their receptive field on the skin surface was touched. We noted specificity for individual fingers and the palm but used the part of the hand which elicited the strongest response for coloring purposes in figure 4.
Figure 4. Somatosensory map of the receptive field locations for the UEA array. Electrode locations were colored differently according to the positions on the monkey's hand that elicited the strongest response when touched (left). Light gray squares indicate unspecific activity not related to touching the hand and dark gray are electrodes that were considered not usable because of impedance measures being out of specification (note that the electrodes in the four corners were reference electrodes). For most of the stimulation experiments we used the three red framed electrodes in the bottom left of the diagram. Corresponding locations on the monkey's hand (right).
Download figure:
Standard image High-resolution imageNeurostimulation system
A neurostimulation system was used to generate stimulation currents which were delivered through the UEA. The system consists of a neurostimulator (CereStim 96, Blackrock Microsystems, Salt Lake City, UT), a control switch (CereStim Switch, Blackrock Microsystems, Salt Lake City, UT), and a control PC. The neurostimulator is a 96-channel programmable current generator equipped with three current generator modules (0–215 μA output current range; ±3.5 to ± 9.5 output voltage range; 4–5154 Hz frequency range). Thus the neurostimulator is capable of producing three concurrent stimuli from any three of the 96 channels. These stimuli are biphasic, charge-balanced pulse trains with adjustable timing and magnitude parameters. In all experiments we used biphasic stimulation (cathodic first) with a maximum current never exceeding 100 μA and a maximum frequency of 300 Hz. The control switch is designed to switch between stimulation and recording modes and was only used in stimulation mode for this study. When in stimulation mode, the control switch passes currents from the neurostimulator to the UEA. The control PC sends signals to the neurostimulator to configure, start and stop stimulation.
Neural data processing system
Neural data recorded from the FMA arrays was processed via the neural data processing system which used a Cerebus 128 channel Neural Signal Processor (Blackrock Microsystems, Salt Lake City, UT) and a PC for decoding. The Neural Signal Processor acquires incoming data at 30 kHz with 16 bit resolution and transmits the data to the decode PC. The decode PC processes and decodes neural signals in near real time to provide direct control of the vMPL arm.
vMPL
The vMPL is a virtual replica of a physical robotic arm (Modular Prosthetic Limb; MPL). Both the virtual and real MPLs were developed by the Applied Physics Laboratory at John Hopkins University (JHUAPL, Laurel, MD). The vMPL is intended to closely resemble a real human adult's upper arm and has 17 degrees of freedom for joints extending from shoulder to individual fingers, which allows it to perform complex reach movements and dexterous manipulations in 3D. For this study we did not use the full flexibility of the vMPL, but restricted it to an 'endpoint' control mode. In this mode only the 3 degrees of freedom which control the position of the center of the palm of the hand are used to move the arm in 3D Cartesian space. In all experiments the monkey was restricted to control the endpoint and not the hand posture or individual fingers. The hand was shaped to form a fist to make collision detection easy and consistent. Control of the arm was either achieved by tracking the monkey's hand position or by the output of a software decoder which used the recorded brain signals to predict the intended movements of the monkey. The vMPL was displayed using Unity3D (Unity Technologies, San Francisco, CA) on a separate display PC.
Behavioral task
Experimental apparatus
A male macaque monkey was trained to make arm movements within a computer-generated, 3D virtual environment. A schematic of the experimental apparatus is shown in figure 1. In all experiments, the primate was seated upright in a plastic primate chair with head constrained to the chair via a skull-mounted head holder (Gray Matter Research, LLC). Vision of the animal's real arm was blocked by a mirror which projected the virtual environment displayed on a top mounted monitor to the monkeys' eyes. The monkey wore shutter glasses (NVIDIA, custom modified) which allowed each eye to see only its corresponding image of the scene to create the illusion of a stereoscopic 3D image.
The behavioral task was implemented as a Simulink model (The Mathworks, Natick, MA) and executed on a PC running a real-time OS (xPC, The Mathwork, Natick, MA). The custom program created experimental flow logic to control the state of the VRE. It monitored all behavioral events, delivered reward, controlled the timing for displaying virtual targets and the vMPL on the display PC, determined target-vMPL collision and triggered a custom C++ program in the neurostimulation system to start stimulation.
Task description
Handbag task
The 'handbag' task was designed to examine whether ICMS could be used as an additional feedback channel. The task starts with the presentation of a blue center target (CT) which is manifested as a cube of edge length 3 cm in front of a gray screen ('handbag') 34 cm horizontal and 30 cm vertical on the display. The monkey had to align the vMPL with the CT and keep touching it for one second to initialize the task. The CT then disappeared and a target (T1) of the same size was randomly presented at one of four possible locations in a plane 3 cm behind the handbag (arranged on an 8.5 cm by 7.5 cm rectangle) and therefore invisible to the monkey. The monkey then had to move the vMPL into the handbag (as shown in figure 5 top row) and search for the target. When found, the vMPL's hand had to touch T1 for 1 s to indicate that he found it. T1 had to be acquired within 12 s after task initialization. The time limit for this task was determined empirically during the training phase. We had to make sure that it was long enough to allow for a sufficient high success rate in the control condition (see below) and to keep the monkey motivated to perform the task. If the target was acquired and held for one second within that time the monkey received a liquid (water) reward. The monkey controlled the vMPL via motion tracking (figure 1). To assess the benefit of this channel we used four different task conditions which provided different types of feedback information when any of the targets (CT or T1) were touched.
Figure 5. The vMPL reaching into the handbag (image sequence; top row). Schematic task progression (bottom row). After the center target appears (bottom left; blue square; ct) in front of the handbag (gray screen) the vMPL hand has to touch and hold the target for 1 s. The center target then disappears and a target appears behind the screen (dotted square; T1) not visible to the monkey. The vMPL then has to reach inside the handbag and probe for the target. When found the target has to be held for 1 s.
Download figure:
Standard image High-resolution imageControl condition: In the control condition, no feedback of any kind was given when a target was touched. In this case the monkey could only solve the task by moving slowly and accidentally staying long enough at the target location. By using this condition as a baseline we could further examine how much each additional channel of information would improve the performance.
Sound condition: When a target was touched a 1000 Hz sine wave sound was played as long as the vMPL's hand and the target touched. This condition was another control to investigate how effective somatosensory stimulation is versus another sensory modality.
Stim condition: When a target was touched, stimulation was triggered and stayed on as long as the vMPL's hand touched the target. We stimulated simultaneously on three selected electrodes (figure 4 left). The stimulus was a 300 Hz biphasic square wave (pulse width 200 μs; phase gap 53 μs) with amplitude of 80 μA. Stimulation parameters and electrode selection were based on the initial mapping and training phase. Only a subspace of all possible combinations of stimulation parameters and electrode selections was tested. Once we evaluated a few robust combinations we did not further investigate others.
Stim + Sound condition: This condition was a combination of the sound and stim conditions in which the sound was played and the stimulation was triggered as long as the vMPL's hand touched the target.
Match-to-sample task
The match-to-sample task was designed to examine whether ICMS could provide other information besides contact. In this task, the animal was required to identify one of two objects in the handbag based on their stimulation frequency (figure 6). As in the handbag task, the vMPL was controlled via motion tracking. The CT in this version of the task served as a template and elicited one of two possible stimulation frequencies, 150 or 300 Hz, when touched. All other stimulation parameters were identical to those in the handbag task. At the beginning of each trial, the CT was presented in front of the handbag, as in the handbag task. Touching and holding the CT for one second initialized the task. The CT disappeared and two target objects (figure 6; T1 and T2) were placed within the handbag at two different locations out of four. If touched, one elicited a 150 Hz stimulation and the other a 300 Hz stimulation. The task of the monkey was now to probe the targets by searching and touching them with the vMPL's hand and then to hold the one matching the frequency of the CT for one second. Each of the two targets randomly appeared in a different location selected from the four corners of a square in a plane behind the screen. If the monkey held at the wrong target for one second or he could not find and hold at the correct target within 18 s, the trial was aborted and the monkey was not given a liquid reward. The additional time needed for probing was taken into account by using a longer time-out period of 18 s.
Figure 6. Schematic task progression for the match-to-sample task. The central target appeared (ct) and when the vMPL touched it a sample ICMS stimulus was applied (either 150 or 300 Hz). After holding the ct for 1 s it disappeared and two targets (dotted squares T1 and T2) were positioned within the handbag. If touched one would trigger ICMS with 150 or 300 Hz. The monkey then had to find, touch and hold the correct target, i.e. the target which would elicit the same stimulus as the ct.
Download figure:
Standard image High-resolution imageBrain control task
The brain control task was a simplified version of the match-to-sample task in which the vMPL was controlled using decoded brain signals from the PPC. No handbag was used and the animal could see all the targets in the VRE. The trial was initiated by the vMPL's hand touching the CT for one second, like in the handbag task. Touching the CT elicited ICMS just as in the handbag 'stim' task (300 Hz; all other stimulation parameters were the same as in the handbag task). After the CT disappeared two identical yellow target cubes (figure 7; T1 and T2) were shown at two locations (not four as in the handbag task) left and right of the CT in a vertical plane 6 cm behind the CT. The location of the two targets was randomly picked from trial to trial. Only T1 elicited ICMS when it was touched by the vMPL's hand and T2 was a distractor. A timeout similar to the match-to-sample task was enforced here. If T1 was touched and held for one second within a timeout of 18 s, the trial was counted as successful and the monkey received liquid reward. If instead T2 was touched for one second, or the timeout was reached, the trial was aborted.
Figure 7. Schematic task progression for the brain control task. The main difference in this task compared to the previous ones is that the monkey's decoded brain signals were used to control the vMPL in this task instead of motion tracking. A central target appeared (ct) and the vMPL had to touch and hold it for 1 s. Then two identical looking yellow targets (T1 and T2) appeared in a plane behind the ct. When touched only T1 elicited ICMS (300 Hz). T2 was a distractor which did not elicit stimulation. T1 and T2 were randomly positioned on each trial (T1 left, T2 right or T1 right, T2 left).
Download figure:
Standard image High-resolution imageDecoding methods paired with stimulation artifact removal
Kalman filter based decoding algorithm
A standard discrete linear Kalman filter was used for continuous online neural decoding for the brain control task (Wu et al 2006, Gilja et al 2012, Hauschild et al 2012). The Kalman filter provides an efficient recursive method for estimating system state in real-time. The current system state was dependent on both the previous system state and system observation. In our study, the system state was defined as the kinematic state of the vMPL hand which included position, velocity and acceleration in three dimensions. In order to accommodate lack of apparent single unit activity (<3 stable units), the system observation was defined as the measurement of MUA. We manually selected 29 channels out of the 128 FMA channels based on signal quality (13 from the PRR array, six from the AIP array and ten from the posterior area 5 array). For each channel we calculated the mean power in three high frequency bands (300–2000 Hz, 2000–4000 Hz, 4000–6000 Hz). Thus a total of 87 neural features were used for decoding. Both the evolving system state and the relation to neural observation were approximated by a linear Gaussian model which provides an estimate of uncertainty and the coefficients which were readily learned from training data using a closed form solution based on Bayesian inference (Wu et al 2006). The discrete time interval between successive states (time bin size) was chosen to be 50 ms based on values reported in the literature (Cunningham et al 2011, Gilja et al 2012). Since neural activity was usually considered to precede hand movement, a uniform time lag of 100 ms was introduced between neural activity and hand kinematics.
Stimulation artifact removal
Retaining spectral features
ICMS delivered through the UEA is picked up by the FMA recording electrodes despite the physical distance between the electrodes. Because the stimulation artifact contaminates the recording, it must be removed before decoding. We used a simple filtering method for online artifact removal which was designed to work with power spectrum based features derived from MUA activity. ICMS has a different influence on each recording channel in terms of shape and amplitude of the artifact, but possesses the same temporal pattern. We used this property to detect the beginning and end of the stimulation period by focusing on one 'reference channel' which showed the largest and therefore easiest to identify artifact. We dealt with the stimulation artifact on the feature level to reduce the computational load for the on-line brain control task.
The artifact-removal filter was calculated at the beginning of each session from a five second interval containing both stimulation and non-stimulation periods. The sample data were divided into 50 ms bins, and the power spectrum of each bin was calculated. The filter was constructed as the ratio of the power spectrum averaged over all non-stimulation time bins to the power spectrum averaged over all stimulation time bins. Then, during online decoding, this filter was applied to the power spectrum of each channel prior to calculating the averaged spectral power features. Figure 8 illustrates an example of averaged power spectrum for non-stimulation time bins, stimulation time bins, and the resulting filter spectrum.
Figure 8. Example of spectral filter generation from one channel (5 s interval; channel 16; recording 20130403). The averaged power spectrum for stimulation (yellow; top left) and non-stimulation time bins (yellow; top right) is used to generate a filter template (blue; center). This filter is then used to remove artifacts in future instances (red; bottom left), which leads to a cleaned signal (red; right). Note that the three distinct peaks at 2400, 3000 and 3600 Hz in the averaged non-stimulation spectrum (yellow; top right) and filtered signal (red; right) are artifacts from the motion tracking system which are present continuously and therefore are not filtered out.
Download figure:
Standard image High-resolution imageRetaining spike features
A new method was also developed to retain spike features offline when there is a stimulation artifact. We made the assumption that the stimulation artifact present in a certain recording channel is nearly deterministic for fixed stimulation parameters, and so the exact shape and size of the stimulation artifact waveform could be modeled and used to build a template for that channel. The stimulation artifact could then be rejected by subtracting the template from the signal. To obtain the artifact template, we identified stimulation artifact waveforms in the band pass filtered signal (300–6000 Hz). The template was then derived by averaging across artifact waveforms. Figure 9 shows the result after rejecting the stimulation artifact from one channel. The residue not only retains spikes between stimulation artifact waveforms but also recovers spikes formerly masked by the stimulation artifact. However, we could not test this method online because the monkey had to be explanted due to an infection. Although we did not use spikes for brain control in the current study, this spike-based method will be useful for future brain-control/stimulation studies in which spikes are used for decoding.
Figure 9. Time series of data before stimulation artifact rejection (gray) and after stimulation artifact rejection (black). The template of stimulation artifact applied is illustrated by dotted line.
Download figure:
Standard image High-resolution imageResults
Handbag task
The utility of the ICMS-induced somatosensory feedback can be evaluated by the performance achieved in the handbag task. The task was tested for six study sessions (excluding training sessions). As shown in figure 10, we used three metrics to measure the animal's performance: success rate, average trial duration, and total number of touches before completing a trial. The success rate was determined by the number of successfully completed trials divided by the total number of initiated trials, i.e. trials in which the CT was held for one second. The trial duration was calculated using only successful trials, and was defined as the time from the first CT touch to the successful completion of the trial. The number of touches was defined as the number of times the target (T1) was touched by the vMPL, before a trial was successfully completed (so the minimum number of touches for a successful trial would be one). The 'touched' state was determined by the virtual environment's internal collision detection routines. We considered the performance better when the success rate was higher, trial duration was shorter, and the animal was able to identify the target location with fewer touches.
Figure 10. Performance of the handbag task measured by success rate (left), trial duration (middle) and number of touches (right) for four sensory feedback conditions. Significant difference according to a one-way ANOVA (Tukey–Kramer method for multiple comparison correction used) between different task conditions are noted by stars (* = 0.05, ** = 0.01 and *** = 0.001 p-value); numbers of trials recorded for the control condition, sound condition, stim condition and stim + sound condition are 281, 1750, 1668 and 858, respectively.
Download figure:
Standard image High-resolution imageAs expected, performance was lowest when no feedback was provided ('control' condition) (figure 10). Although there appeared to be no significant difference of success rate or number of touches between the 'sound' condition and the other two feedback conditions, we found it took the animal a significantly longer time to find the target in the handbag using sound alone. There was also no significant difference in number of touches between the 'control' and 'sound' conditions, but there were significant differences between the stimulation conditions ('stim' and 'stim + sound') and the 'control' condition. The combination of ICMS and sound did not appear to make a significant difference in performance compared with using ICMS alone.
Match-to-sample task
This task was tested exclusively for about one month (11 study sessions; excluding training sessions). We found that the success rate was significantly above chance (average performance 74.17% correct; one sample t test: p = 3.88 × 10−5), which indicates that most of the time the animal could distinguish between two different ICMS frequencies and use that information to detect different objects. Despite some fluctuations, the animal's performance improved gradually over the 11 study sessions as demonstrated by the increase in daily success rate and decrease in average duration (figure 11).
Figure 11. Daily success rate (left) and average trial duration (right) for match-to-sample task in 11 study sessions.
Download figure:
Standard image High-resolution imageBrain control task
In this task, which was tested for eight study sessions (excluding training sessions), we combined brain control while ICMS was applied during the course of the task. Successful completion of the task required both the ability to perceive ICMS and the ability to move the vMPL via brain control. Successful decoding was only possible by stimulation artifact removal during ICMS. We initially tried ICMS and online brain control without artifact removal but when ICMS was applied, the vMPL would drift off in a random direction making it difficult for the animal to perform the task. We therefore focused on brain control with artifact removal. The results show that after training the animal was always able to move the vMPL to the correct target with a success rate much higher than chance level (figure 12). To successfully complete a trial, the monkey had to probe the targets and hold the vMPL on the target if it elicited ICMS or otherwise move towards the other target.
Figure 12. Daily performance of the animal in the brain control task. Chance level was 50% since only trials were counted in which one of the two targets was touched for 1 s.
Download figure:
Standard image High-resolution image
Discussion
In this study, a real time system enabling simultaneous stimulating and decoding from the brain was developed for a closed-loop cognitive neuroprosthesis which could potentially be applied to quadriplegic patients. ICMS was delivered to the hand area of BA1 via chronically implanted microelectrodes. Movement parameters were decoded simultaneously from the PPC to control a vMPL. In our model animal, the rhesus macaque, we demonstrated that percepts can be successfully elicited by ICMS. To overcome the decoding challenges caused by stimulation artifacts, we implemented a frequency filter to effectively remove the stimulation artifacts online. We also developed a method for retaining spike features offline.
Stimulation
Our study demonstrates that ICMS of BA1 can provide useful percepts for closed-loop feedback control. In the handbag task, the subject was able to move the vMPL and use information provided by ICMS to find the object hidden in the handbag. The performance in the handbag task was significantly better when using ICMS feedback as compared to the control condition. In the match-to-sample task, the animal could learn to distinguish between two different stimulation frequencies and associate different frequencies with different objects, which suggests ICMS could potentially convey additional object properties to the brain. Although we cannot tell exactly what kind of percept the different frequencies elicited, the monkey could clearly discriminate them. It is possible that the monkey learned an artificial sensation since researchers have found that the brain can learn a new set of percepts after being exposed to ICMS over time (O'Doherty et al 2011). It is also possible that ICMS of BA1 may be able to provide close to natural sensations since epicortical acute simulation studies on humans have shown that electrical stimulation of S1 is able to evoke natural, painless percepts (Libet 1982, Ojemann and Silbergeld 1995). Furthermore, researchers have shown that natural signals to the brain can be mimicked if ICMS parameters are precisely tuned (Hathaway and McKinley 1989, Berg et al 2013).
One important factor which can impose constraints on the use of ICMS for neural prosthetic applications is safety. Prolonged ICMS can potentially induce some level of neural injury. Researchers evaluated the histological effects of prolonged ICMS on neural tissue and found that the amount of neuronal loss surrounding the electrode tips was dependent on the stimulation regime applied to the electrodes. (Agnew et al 1986, McCreery et al 1990, 2010). No neurological problems or deterioration in performance was reported for stimulation regimes adopted by most of the animal studies (Rousche and Normann 1999, Santhanam et al 2006, Fitzsimmons et al 2007, Parker et al 2011, Torab et al 2011). A recent study tested sensorimotor consequences of various stimulation regimes on three rhesus macaques. Except on the first day of stimulation when two ICMS-induced adverse events were reported (vocalization in one NHP and rhythmic contractions of the contralateral arm in both NHPs), no further adverse effects of ICMS were noticed after modifications were made to the stimulation regimes (Chen et al 2014). In the current experiments, no noticeable ICMS-related behavioral deficits were seen during the entire course of the study.
Decoding method
On-line decoding of movement parameters was performed using a simple standard discrete linear Kalman filter. The Kalman filter and its modified versions have been demonstrated to be effective in many off-line reconstruction and on-line continuous brain control studies (Wu et al 2003, O'Doherty et al 2011, Hauschild et al 2012, Hochberg et al 2012). All aforementioned studies utilized single unit activity as control signals. In this study, however, we used MUA as control signals in the brain control task due to lack of apparent and stable single unit activity. MUA reflects aggregate spiking activity of a number of neurons around electrode tip in a region smaller than LFPs but larger than single units, and reflects energy in high frequencies (300–6000 Hz) (Buchwald and Grover 1970, Legatt et al 1980). Compared with single-unit spikes, MUA recordings are much easier to obtain and more stable over time. Compared with LFPs, MUAs recorded from neighboring channels are more informative and less redundant. An off-line analysis of single unit, LFP and MUA in premotor cortex has shown that MUA yielded better or equal predictions of reach direction, grasp type and movement velocity and was informative even when spikes were artificially removed (Stark and Abeles 2007). Since our pool of single units was very limited (<3) we had to rely on MUA or LFP (which we initially tried but yielded lower performance than MUA; data not shown).
Stimulation artifact removal
One main challenge for a successful implementation of closed-loop neuroprosthesis is the real-time coupling of ICMS and neural decoding. Neural decoding was performed on neural features from the recording FMAs. However, large stimulation artifacts contaminated data recordings on most channels in the recording arrays. The simplest solution for dealing with stimulation artifacts is to disregard neural recordings during times of stimulation. This method would eliminate the problem altogether but also greatly reduce performance due to lack of control signals during the period of stimulation—notably, at a critical point when the prosthesis is interacting with another object. Another way to reduce the influence of stimulation artifacts is to lower the stimulation frequency or to interleave stimulation and neural recordings with a low clock rate (O'Doherty et al 2011). Others have tried to fill the gap containing stimulation artifacts by estimating the control signals based on the uncontaminated signals outside the gap (Walter et al 2012). Here we tried to separate stimulation artifacts from neural signals based on statistical features of both stimulation artifacts and uncontaminated neural signals. The approach utilized information contained in the contaminated signals and performed stimulation artifact removal on the feature level which is well suited for online use due to its simplicity. A method for retaining spike features was also developed off-line and may show promise for on-line applications.
Future prospects
The closed-loop system developed in this study could be readily transferred to future human clinical trials for the benefit of quadriplegic patients. Since the vMPL is a virtual replica of a real robotic limb, transition from the virtual reality to a physical implementation is straightforward. The real prosthetic limb has the same appearance and properties as the virtual one and also shares the same control interface. For human subjects, time could be saved due to easier task training: the human can report the qualitative sensation from stimulation and rate the magnitude of effects on percepts when stimulation parameters are changed. By this approach various decoding algorithms could be tested within a relatively short period of time and the control performance could be improved rather quickly. The main challenge for both human and nonhuman primate BMI studies still lies in the ability to obtain a large amount of informative and stable neural signals, which are closely tied to electrode array manufacturing quality and surgical implantation techniques. The demonstration of brain control in the face of stimulation artifacts that relies on MUA is a promising practical step for extending and improving the viability of cortical recordings.
Acknowledgements
We thank K Pejsa, J D Beaty, F V Tenore for project management; N Sammons and C Gonzales for animal care; V Shcherbatyuk, Applied Physics Laboratory at John Hopkins University for technical support, T Yao for administrative assistance; S J Bensmaia for valuable advice on stimulation study. This work was supported by the Defense Advanced Research Projects Agency and the National Institutes of Health.