However, existing BCI limb control remains crude and inflexible because we do not understand how the recorded neural activity relates to skilled movement. Finally, we studied the temporal dynamics of neural control to understand how existing models of neural activity extend to BCI control.
Neural prostheses to restore movement
Components of neural prosthetic systems
Action potentials provide informative low-latency input functions and have controlled the most impressive typing and hand mass indices to date (Wodlinger et al., 2015; Willett et al., 2021). These areas represent significant challenges and are the work of many promising interdisciplinary collaborations, as reviewed elsewhere (Collinger et al., 2013b, 2018; Bockbrader, 2019; Kleinfeld et al., 2019; Luan et al., 2020).
Barriers to clinical usage
However, control of BMI hands remains elusive (Collinger et al., 2013c; Hotson et al., 2016), because we do not yet understand how the recorded brain signals relate to intended hand movements. Hand movements involve different neural processes from arm extension (Suresh et al., 2020; Sobinov and Bensmaia, 2021), so we are only beginning to understand how the brain integrates the multitude of factors for hand movements.
Dissertation outline
Collinger et al., 2013c), no study has developed prosthetic controllers that can interact with physical objects robustly enough to assist patients in their daily activities. To compare with the second movement category, available studies described how several factors can modulate neural activity (Omrani et al., 2017): movement direction (Georgopoulos et al., 1982), posture (Aflalo and Graziano, 2006), sensory feedback ( Scott, 2016 ) and effector dynamics ( Shadmehr et al., 2010 ; McNamee and Wolpert, 2019 ; Kalidindi et al., 2021 ).
Neural coding of hand movements
Manual dexterity
Neural control of hand movements
Goodman et al., 2019) and speed signals (Nason et al., 2021), although their modulation could also be influenced by other factors. Consequently, when AIP is inactivated, subjects can still reach normally but fail to preform grasps (Gallese et al., 1994).
Somatosensory feedback for object manipulation
To overcome this barrier, bidirectional BMI systems aim to restore both motor control and somatosensory input (Collinger et al., 2018). Sometimes the lack of sensory feedback can actually benefit basic neuroscience studies; we can decouple motor control and sensory feedback, allowing us to separate these processes in our experiments (Golub et al., 2016).
Summary
Introduction
More generally, temporal structure may depend on the movement type (Suresh et al., 2020) and the brain region recorded (Schaffelhofer and Scherberger, 2016). In able-bodied individuals, the cortical representation structure of finger movements follows natural statistics of movements ( Lillicrap and Scott, 2013 ; Ejaz et al., 2015 ).
Results
- Intracortical recordings during finger flexion
- Accurately decoding fingers from PPC single-neuron activity
- Finger representational structure matches the structure of able-bodied individuals
- Representational structure did not trend towards task optimum
- Finger representational structure is motor-like and then somatotopic
These analyzes show that the representation structure does not tend toward the optimal task (Figure 2.2c) with experience, ruling out the task statistics hypothesis (3). The muscle activation model (Figure 2.4c) predicts that the representational structure should follow the coactivation patterns of muscle activity during individual finger movements.
Discussion
- Neural prosthetic control of individual fingers using recordings from PC-IP
- Connecting brain-computer interface studies to basic neuroscience
- Matching finger organization between tetraplegic and able-bodied participants
- Able-bodied-like finger representation is not explained by learning
- Representational dynamics are consistent with PPC as a forward model
- Preserved motor representations in PC-IP after paralysis
Furthermore, fMRI BOLD contrast is thought to reflect cortical input and intracortical processing ( Logothetis et al., 2001 ). The somatotopic model predicts that neighboring fingers will have similar cortical activity patterns (Ejaz et al., 2015).
Methods
- Data collection
- Experimental setup
- Statistical analysis
- Data availability
- Code availability
Cross-validation ensures that the (squared) distance estimate is unbiased; 𝐸[𝑑𝑗𝑘2 ] = 0 when the underlying distributions are identical (Walther et al., 2016). If a model reaches the noise ceiling, the model fits the data well (Nili et al., 2014).
Acknowledgments
To estimate the uncertainty about the model fit, we initiated RDMs (sessions) via the cross-validation procedure (Schütt et al., 2019). At each time point, we linearly decomposed the RDM data into the component models using nonnegative least squares.
Supplementary Material
Gardner-Altman estimate plot (Ho et al., 2019) of the WUC similarity between same-ROI pairs of RDMs (N = 630 pairs between 36 subjects). Noise ceiling: gray region estimates the best possible model fit (Methods). b) Paired Gardner-Altman estimate plot (Ho et al., 2019) of the agreement (WUC) between participant NS (average RDM across sessions) and individual MC and PC-IP RDMs of able-bodied fMRI participants. -isolated single neurons from the tetraplegic participant match the finger representation structure of able-bodied individuals. a) Histogram of L-ratio, a point-sort cluster metric.
The gray semicircle down indicates that the noise floor is significantly higher (P < 0.001) than the fit of the SPLa fMRI RDM and the unstructured model. d) Representative dynamics analysis repeated using only well-isolated units shows early fit to the muscle model and late fit to the somatotopic model.
Summary
Introduction
Using this cursor control, (Jarosiewicz et al., 2015; Pandarinath et al., 2017; Nuyujukian et al., 2018) then developed on-screen keyboard typing interfaces for tetraplegic participants. PPC uses partially mixed selectivity to encode multiple motor variables simultaneously ( Zhang et al., 2017 ), which may be useful for versatile neural decoding. Despite the clearly demonstrated role of PPC in grasping (Gallese et al., 1994; Schaffelhofer and Scherberger, 2016; Sobinov and Bensmaia, 2021), less is known about PPC responses during individual finger movements.
We connected this neural decoder to control a neural prosthetic hand, with accuracy greater than recent intracortical BMI studies (Jorge et al., 2020; Guan et al., 2022b).
Methods
- Study participants
- Tasks
- Implant location
- Neural signal recording and preprocessing
- Feature Extraction
- Single-neuron selectivity for finger movements
- Offline classification with cross-validation
- Online brain-machine interface (BMI) discrete control
- Neural distance between fingers
- Shared representations across hands
- Factorized finger representations
The task was similar to the Text-Cue version of the alternating-delay finger-pressing task, except without the Pre-Cue phase. The first array (labeled JJ-PPC) was implanted in the superior parietal lobule (SPL) of the left PPC. The intertrial interval (ITI) analysis window was defined as the last 500 ms of the ITI phase.
Labels consisted of finger movement cues, and features consisted of the firing rates during the period of each trial [𝑡𝑠, 1 + 𝑡𝑠].
Results
- Single-neuron modulation to individual finger presses
- Classifying finger presses from neural activity
- Brain-machine interface control of finger movements
- Classifying individual finger presses from both hands
- Factorized representation of finger type and laterality
Finger movements could also be decoded by PPC during the planning period (Figure 3.5e)), although classification accuracy was lower (NS-PPC: 66%; JJ-PPC: 61%; chance: 17%) than during execution of the movement. The confusion matrix for participant NS (Figure 3.6a) shows that she achieved high online control accuracies (86%; chance: 17%). Geometrically, these generic vectors would form parallelograms between corresponding sets of conditions (Figure 3.8c) (Fu et al., 2022).
Even when accounting for differences in neural population size, the finger-type CCGP of JJ-PPC was lower than the finger-type CCGP of NS-PPC and JJ-MC (Supplementary Figure 3.11).
Discussion
Here, the NS-PPC and JJ-MC finger codes can be decomposed into finger type and laterality subspaces (Figure 3.8d-e), which resembles the partial compositionality described by (Willett et al., 2020) for arm and leg movements. The difference between JJ-PPC and NS-PPC results may arise from neuroanatomical variability (Scheperjans et al., 2008; Gallivan and Culham, 2015) or differences in implant location. However, it is difficult to accurately compare implant locations because the anatomical location of individual functional areas can vary widely between participants (Scheperjans et al., 2008; Gallivan and Culham, 2015).
Future comparisons may benefit from multimodal preoperative neuroimaging to map implant locations on standard plots ( Glasser et al., 2016 ).
Acknowledgments
Supplementary Material
The colors of the lines indicate the epoch of the analysis. e) Overlay of JJ-PPC neurons that modulated significantly (q < 0.05) with large effect sizes (η2 > 0.14) during movement preparation (plan) and movement execution (movement). The colors of the lines indicate the epoch of the analysis. e) Overlay of JJ-MC neurons that modulated significantly (q < 0.05) with large effect sizes (η2 > 0.14) during movement preparation (plan) and movement execution (movement). Markers show mean across sessions.. e) Generalized linear decoders (Supplementary Figure 3.8) across finger type to classify hand (left) and across hand to classify finger type (right) (p < 0.001, permutation test) , showing that movement representations were factored into finger and hand type sizes.
Linear decoders generalized (Supplementary Figure 3.8) across finger type to classify the hand (left; p < 0.001, permutation test). right) Finger-type cross-condition generalization performance (CCGP; blue circle) was lower than standard decoding accuracy (orange triangle).
Summary
Introduction
Often (Shenoy et al., 2013; Sauerbrei et al., 2020), the system state is defined as the MC firing rate vector 𝒓, and 𝑓 is bounded by an additive linear function ℎ that models the local recurrence. Second, this dynamic causes movement; that is, neural perturbations disrupt movement if and only if the perturbations change the dynamics subspace of the task (O'Shea et al., 2022). Noise-robust autonomous dynamical systems exhibit low entanglement (described in MC (Russo et al., 2018)) (i.e., smooth flow fields) and low divergence (described in the supplementary motor area, but not in MC, (Russo) et al., 2020)).
During capable movement, the motor cortex reflects proprioceptive signals that are missing during BCI control (Stavisky et al., 2018).
Results
- Intracortical brain-computer interface (BCI) cursor control
- Ballistic and sustained BCI movements
- Sustained single-neuron and population activity in motor cortex (MC) during
- Comparing ballistic and sustained BCI movements reveals input-driven dynamics
During sustained movements, many neurons showed the same increase in firing rate (Figure 4.2a), but instead maintained their firing rate throughout the movement (Figure 4.2b-c). Average firing rate of a sample neuron, which was activated during movements to target 5 and sustained firing for sustained movements. During movement execution, the initial aDSH study described (quasi-)oscillatory dynamics, similar to a pendulum (Pandarinath et al., 2018a) or a spring-mass system (Figure 4.3a).
Next, we applied neural velocity and jPCA analyzes to MC activity during sustained BCI movements (Figure 4.3c).
Discussion
- Rigid dynamics versus flexible control of motor cortex activity
- Switching decoders for brain-computer interfaces
- Sensory and non-sensory inputs to motor cortex
- Unifying neural dynamics and flexible feedback
Most previous presentations of neural prosthetics have used stationary algorithms such as linear regression to decode movement speed (Collinger et al., 2013c). In this regime, real-time sensitivity (Shanechi et al. and fast error correction (Even-Chen et al., 2017) are more important than perfect first decoding. Furthermore, visual feedback of cursor position only weakly affects MC activity in the absence of movement (Stavisky et al., 2018).
First, the thalamus provides time-varying input to the motor cortex in mice to enable reaching (Sauerbrei et al., 2020).
Methods
- Data collection
- Experimental setup
- Statistical analysis
- Closed-loop decoding pipelines
On some sessions, the participant repeated the task with a small number of repetitions, so that he could familiarize himself with the decoder behavior (Willett et al., 2017b). The reach duration was set to approximately 500 ms (ballistic) or 2 seconds (sustained) using a velocity gain parameter of the neural decoder ( Willett et al., 2017b ). The Target Grid task (Nuyujukian et al., 2015) consisted of 5 × 5 targets arranged in a square grid, resembling an on-screen keyboard interface.
We applied jPCA to spike-sorted neural recordings to detect oscillatory dynamics as previously described in (Churchland et al., 2012).
Acknowledgments
During the sustained and ballistic BCI task, we preprocessed neural activity by binning spike counts in non-overlapping 30-ms bins, z-scoring firing rates for each channel, and reducing dimensionality to the first 15 principal components. We decoded the movement intention from the population activity with reduced dimensionality using a neural dynamic filter (NDF) (Kao et al., 2015) with a 10-dimensional latent state. We used a temporal convolutional neural network (denoted “FENet”) to extract features from raw voltage time series sampled at 30 kHz (Haghi et al., 2021).
Because FENet generates multiple (K=8) features per electrode, we used partial least squares regression to reduce this number to (K=2) informative features for each electrode.
Supplementary Material
The dashed arrow visualizes the intended movement direction for reference, but was not displayed during the current task. The cursor was fixated at the center during the ITI, Cue, and Delay phases, and the participant was instructed not to move during these periods. During the baseline Go variant, the participant moved the cursor to the target and calmed down after the target was acquired.
During the masked Go variant, the participant moved the cursor to the remembered target and relaxed based on self-pacing.
Next steps: a neuroprosthetic assistant through hybrid control
By combining signals from both the motor cortex and the posterior parietal cortex, we were able to achieve state-of-the-art finger classification accuracy. Representation of visual and motor aspects of reaching movements in the human motor cortex. Dissociation of sensorimotor deficits after rostral versus caudal lesions in the hand representation of primary motor cortex.
Composite coding of individual finger movements in the human posterior parietal cortex and motor cortex enables ten-finger decoding. Differential impairment of individual finger movements in humans after damage to the motor cortex or corticospinal tract. Reduced muscle selectivity during individual finger movements in humans after damage to the motor cortex or corticospinal tract.
