CHAPTER 4 ESTIMATION OF KINEMATIC-DRIVEN NEURAL REPRESENTATIONS FOR
4.6 E XPERIMENTAL R ESULTS
4.6.2 Neural Dynamics
I analyzed the dynamics of neural representations generated by each method (KLDF, SSC, FA, GPFA and LFADS). To this end, I projected all neural representations onto a 2D space using jPCA. Then, I visualized the neural trajectories of each method in the 2D spaces where each trajectory was specified by target directions (Fig. 4-3). I also marked motion onset on every trajectory (white circles in Fig. 4- 3). Note that each motion onset was defined as a time point corresponding to 15 % of the maximum speed [151]. Mean motion onset was 269 ± 64 ms after target cues in monkey C, 106 ± 73 ms in monkey M, and 162 ± 63 ms in monkey F, respectively.
Figure 4-3 shows that the motion onset points of KDLF were better confined in the output-null dimension than those of other neural representations. To quantify this, I collected all the points of the neural trajectories from the beginning (160 ms before motion onset) until motion onset (0 ms) corresponding to no movement and measured how well these points were aligned on an 1D axis. I calculated the perpendicular distance between these collected points and the output-null dimension where I determined the output-null dimension by a line defined as jPC2 = -jPC1. This distance was supposed to decrease as the neural trajectories were more confined to the output-null dimension. The results in Figure 4-4 and Table 4-1 showed that the average distance with KDLF was lower than those
Table 4-1 Perpendicular distance between jPCs and output-null dimensions
Monkey C Monkey M Monkey F
SSC 0.0107±0.008 0.0193±0.011 0.0252±0.019
FA 0.0082±0.005 0.0166±0.010 0.0198±0.015
GPFA 0.0087±0.006 0.0161±0.010 0.0216±0.017
LFADS 0.0166±0.009 0.0172±0.009 0.0189±0.013
KDLF 0.0042±0.002 0.0179±0.010 0.0051±0.006
66
with other neural representations, indicating that KDLF made better alignments to the output-null
dimension, except for monkey M (two-way ANOVA, Tukey-Kramer correction, p < 0.05).
During movements, the neural trajectories of all neural representations swept through both dimensions of jPC1 and jPC2, consistent with patterns reported in [35] (Fig. 4-3). In particular, the neural trajectories of KDLF appeared to show more consistent and clearer rotational patterns than other neural representations. In addition, the neural trajectories of KDLF tended to be more spatially Figure 4-3 Neural trajectory for the target directions during task. First rows denote averaged trajectories of arm movement for each monkey. Each color-code indicates target direction. A solid dark circle denotes motion onset for each reach trial. Arrows indicate the direction of attraction over time.
Note that the target directions in monkey M were binned as eight-angle widths uniformly distributed in an interval of 45º for each sub-trial. From second to bottom rows denote neural trajectory for reaching direction calculated by jPCA. Each axis was limited by -0.1－0.1.
67
organized such that adjacent neural trajectories corresponded to adjacent target directions.
I assessed the property of the neural trajectories on the space of jPCs by evaluating the sum of the 1st and 2nd jPCs, TCjPCs = 1st jPC + 2nd jPC, where I denote the sum as TCjPCs (see Section 4.4).
Then, I examined its temporal variation in single trials for each neural representation, as shown in Figure 4-4. Overall, TCjPCs remained relatively constant before motion onset for every neural representation.
LFADS showed changes of TCjPCs before motion onset possibly because of the effect of its intrinsic dynamical system. Moreover, TCjPCs before motion onset by KDLF appeared to be least variable across
trials with different target directions. This observation was assessed by calculating differences of TCjPCs
from 0 before motion onset and comparing the average difference among neural representations. The result of the average TCjPCs is summarized in Table 4-2. It showed that KDLF yielded the most consistent TCjPCs across all the trials with different target directions, except for monkey M, indicating that it could represent the null space of neural activation of muscles regardless of target directions (two- way ANOVA, Tukey-Kramer correction, p < 0.05).
Figure 4-4 Temporal changes in the average of the perpendicular distance between jPCs and output-null dimension. Each color corresponds to each different neural representation. Shaded line denotes the standard error across all trials. Vertical dotted line denotes motion onset.
Table 4-3 Deviation Ratio of The Sum of jPCs
Monkey C Monkey M Monkey F
SSC 7.1 ± 6.2 2.0 ± 3.0 4.3 ± 4.7
FA 8.2 ± 9.4 2.0 ± 1.6 4.4 ± 3.5
GPFA 8.4 ± 8.4 2.5 ± 3.5 4.8 ± 7.0
LFADS 2.7 ± 2.0 1.8 ± 1.9 4.5 ± 4.5
KDLF 22.1 ± 16.2 3.1 ± 2.2 30.3 ± 29.6
Table 4-2 Average of TCjPCs from target cue to motion onset
Monkey C Monkey M Monkey F
SSC 0.0480±0.031 0.0903±0.045 0.1100±0.081 FA 0.0367±0.023 0.0766±0.040 0.0864±0.062 GPFA 0.0389±0.026 0.0740±0.041 0.0936±0.070 LFADS 0.0976±0.055 0.0788±0.036 0.0826±0.053 KDLF 0.0207±0.013 0.0847±0.045 0.0247±0.026
68
After motion onset, TCjPCs should change dynamically to generate diverse movements. Figure 4-5 demonstrated that TCjPCs by KDLF showed systematic and dynamic changes according to each target direction. Some other neural representations such as LFADS also described such dynamics well but KDLF was most consistent over different monkey datasets (e.g., see TCjPCs patterns of KDLF for monkey F). To assess how distinctively and dynamically TCjPCs changed for producing movement- related muscle output from null output, I calculated a deviation ratio of the variance of TCjPCs after
motion onset to that before onset for all neural representations (Table 4-3). TCjPCs by KDLF clearly showed the largest deviation compared to others.
Figure 4-5 Temporal changes in the sum of jPCs (1st + 2nd jPCs) during arm movement. First rows denote changes in Euclidean distance D (dark line) between endpoint of arm and target and in averaged speed profile (magenta line) for each monkey. Shaded line denotes the standard error across all trials.
The dashed line indicates is motion onset (M). From the second to the bottom rows denote temporal changes in the sum of jPCs for each neural representation: SSC, FA, GPFA, LFADS, and KDLF, respectively. Each color corresponds to the target directions.
69 4.6.3 Motor Decoding
I evaluated the performance of decoding five different neural representations into various kinematic parameters with a simple linear decoder built and tested in each monkey. Here, I predicted four kinematic parameters (velocity, acceleration, jerk, and speed). As an example, Figure 4-6 illustrates the trajectories of actual velocity along with those of predicted velocity for all neural representations (SSC, FA, GPFA, LFADS, and KDLF). In this example, predicted velocities decoded from KDLF were relatively closer to actual ones than those from other neural representations over different monkeys. I then evaluated decoding error (i.e., MSE) of using five neural representations with two-way ANOVA, where independent variables consisted of neural representations and subjects (i.e., monkey). For velocity decoding, two-way ANOVA showed the main effect of neural representations (χ2(1) = 92.2, p
< 0.01) as well as that of subjects (χ2(2)= 309.1, p < 0.01) on MSE. The post-hoc multiple comparison test with Tukey-Kramer correction revealed that KDLF yielded a lower decoding error than LFADS (p
< 0.01), GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys (see Fig. 4-7 and Table 4-
Figure 4-6 Examples of actual and predicted velocity trajectories. Gray line denotes actual velocity in ten example trials of the testing set, and each colored line indicates the trajectory of the predicted velocity of segmented by a vertical dashed line according to five neural representations: Smoothing (dark), FA (cyan), GPFA (green), LFADS (magenta), and KDLF (red).
70
4, Table 4-5). KDLF yielded 9.3 % increase in decoding performance on average compared to the second-best neural representations by LFADS (monkey C: 11.4 %, monkey M: 4.1 %, and monkey F:
Table 4-5 Correlation coefficients between actual and predicted kinematic variables for all monkeys
Subjects Kinematic
Types SSC FA GPFA LFADS KDLF
Monkey C
VX 0.72±0.37 0.72±0.33 0.76±0.37 0.82±0.32 0.81±0.30 VY 0.81±0.15 0.77±0.18 0.81±0.16 0.88±0.18 0.93±0.08 AX 0.66±0.22 0.66±0.23 0.68±0.20 0.67±0.38 0.73±0.31 AY 0.57±0.18 0.56±0.20 0.61±0.18 0.76±0.19 0.80±0.13 JX 0.23±0.17 0.18±0.15 0.26±0.16 0.38±0.24 0.33±0.21 JY 0.28±0.13 0.22±0.12 0.29±0.12 0.39±0.21 0.41±0.16
||V|| 0.89±0.07 0.87±0.08 0.91±0.07 0.93±0.23 0.96±0.03
Monkey M
VX 0.67±0.32 0.66±0.32 0.71±0.37 0.67±0.34 0.73±0.28 VY 0.66±0.21 0.66±0.22 0.73±0.18 0.70±0.26 0.72±0.21 AX 0.62±0.22 0.62±0.21 0.65±0.21 0.62±0.25 0.70±0.18 AY 0.44±0.21 0.43±0.23 0.48±0.25 0.49±0.23 0.55±0.24 JX 0.17±0.12 0.17±0.12 0.17±0.11 0.17±0.11 0.15±0.11 JY 0.15±0.12 0.15±0.12 0.15±0.11 0.16±0.12 0.11±0.12
||V|| 0.68±0.23 0.69±0.23 0.72±0.26 0.82±0.12 0.78±0.21
Monkey F
VX 0.47±0.31 0.44±0.30 0.46±0.35 0.28±0.54 0.67±0.36 VY 0.69±0.30 0.69±0.29 0.71±0.30 0.75±0.39 0.81±0.30 VZ 0.52±0.29 0.51±0.31 0.56±0.34 0.67±0.38 0.76±0.26 AX 0.29±0.27 0.22±0.29 0.27±0.32 0.24±0.46 0.47±0.35 AY 0.44±0.27 0.41±0.29 0.43±0.32 0.62±0.40 0.67±0.26 AZ 0.28±0.27 0.23±0.26 0.27±0.33 0.56±0.35 0.52±0.31 JX 0.21±0.16 0.17±0.14 0.17±0.16 0.18±0.30 0.29±0.21 JY 0.26±0.18 0.26±0.17 0.26±0.18 0.43±0.26 0.36±0.25 JZ 0.17±0.14 0.15±0.14 0.14±0.16 0.31±0.24 0.26±0.20
||V|| 0.78±0.12 0.78±0.12 0.81±0.12 0.93±0.07 0.92±0.06 Values: mean ± standard deviation,
Kinematic Types: V = velocity; A = acceleration, J = jerk, ||V|| = speed.
Table 4-4 Decoding errors for all monkeys
Subject Kinematic types SSC FA GPFA LFADS KDLF
Monkey C
V 0.10±0.02 0.11±0.02 0.10±0.02 0.07±0.05 0.06±0.01 A 0.03±0.01 0.03±0.00 0.03±0.00 0.02±0.03 0.02±0.00
J 0.01±0.00 0.01±0.00 0.01±0.00 0.02±0.01 0.01±0.00 Monkey M
V 0.10±0.03 0.10±0.03 0.10±0.03 0.09±0.03 0.08±0.02 A 0.02±0.00 0.02±0.00 0.02±0.00 0.02±0.00 0.02±0.00 J 0.01±0.00 0.01±0.00 0.01±0.00 0.01±0.00 0.01±0.00 Monkey F
V 0.15±0.02 0.15±0.02 0.15±0.02 0.12±0.03 0.10±0.02 A 0.04±0.01 0.04±0.01 0.03±0.01 0.03±0.01 0.03±0.01 J 0.02±0.00 0.02±0.00 0.02±0.00 0.02±0.00 0.02±0.00 Values: mean ± standard deviation, Measurement: MSE = mean squared error,
Kinematic Types: V = velocity; A = acceleration, J = jerk.
71
12.4 %). Two-way ANOVA on CCs also showed the main effect of the neural representations (χ2(1) = 21.1, p < 0.01) as well as that of the subjects (χ2(2)= 98.2, p < 0.01) on the mean of the CCs. The post- hoc multiple comparison test revealed that KDLF yielded a higher CC than LFADS (p < 0.01), GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys.
For acceleration decoding, two-way ANOVA showed the main effect of neural representations (χ2(1) = 16.7, p < 0.01) as well as that of subjects (χ2(2)= 465.4, p < 0.01) on MSE. The post-hoc multiple comparison test revealed that KDLF yielded lower decoding error than LFADS (p < 0.01), GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys. KDLF yielded 7.5 % increase in decoding performance on average compared to the second-best neural representations by LFADS (monkey C: 16.6 %, monkey M: 4.2 %, monkey F: 1.6 %). It also showed the main effect of the neural representations (χ2(1) = 54.6, p < 0.01) as well as that of the subjects (χ2(2)= 268.8, p < 0.01) on the mean of the correlation coefficients for dimensions of working space. The post-hoc multiple comparison test revealed that KDLF yielded a higher correlation coefficient than LFADS (p < 0.01), GPFA (p <
0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys.
For jerk decoding, two-way ANOVA showed the main effect of neither neural representation (χ2(1) = 19.2, p = 0.9) nor subjects (χ2(1) = 12.6, p = 0.85) on MSE. Meanwhile, it showed the main effect of the neural representations (χ2(1) = 30.5, p < 0.01) as well as that of the subjects (χ2(2)= 150.2,
Figure 4-7 Comparison of velocity decoding errors. Open circle denotes average of the mean squared errors between actual and predicted velocity for tested trials, and each error bar indicates the standard error. Each color corresponds to monkey C, M, and F, respectively. Asterisks denote significant difference between neural representations (** p < 0.01, a two-way ANOVA by a post hoc analysis with a Tukey-Kramer correction for multiple comparison).
72
p < 0.01) on the mean of CCs. The post-hoc multiple comparison test revealed that KDLF yielded a higher CC than GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys, but no difference from LFADS (p = 0.43) (see Table 4-4).
For speed decoding, two-way ANOVA showed the main effect of neural representations (χ2(1)
= 38, p < 0.01) as well as that of subjects (χ2(2)= 104.9, p < 0.01) on CCs. The post-hoc multiple comparison test revealed that KDLF yielded a higher CC than GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys, but no difference from LFADS (p = 0.93).
I reconstructed the position trajectory by integrating the predicted velocity and compared it among neural representations. Figure 4-8 illustrates examples of the reconstructed position trajectories as well as actual ones in each monkey. It apparently demonstrated that the reconstructed trajectories from KDLF traced actual ones more accurately than those from other neural representations. This was especially well pronounced in the 3D movements in monkey F. Two-way ANOVA showed the main effect of neural representations (χ2(1) = 113.6, p < 0.01) as well as that of subjects (χ2(2)= 777.2, p <
0.01) on MSE of position. The post-hoc multiple comparison test revealed that KDLF yielded lower reconstruction error than LFADS (p < 0.01), GPFA (p < 0.01), FA (p < 0.01) and SSC (p < 0.01) for all monkeys (see Fig. 4-9). KDLF yielded 11.8 % increase in decoding performance on average than LFADS (monkey C: 8.2 %, monkey M: 3 %, and monkey F: 24.3 %).
Figure 4-8 Position trajectories reconstructed from predicted velocity. Each colored line denotes one trial for a specific target direction. For monkey C, colors denote eight radial targets, and trajectories correspond to 43 trials. For monkey M, colors correspond to four sequential sub-trials within a single trial. Solid filled circle denotes a starting point for one sub-trial. For monkey F, colors indicate 26-radial targets, and one trial per a target are shown.
73 4.7 Discussion
This study proposed a new approach for finding kinematics dependent neural representations of M1 population activity. We estimated kinematics dependent neural representations by nonlinearly mapping M1 population firing rates to kinematics dependent latent factors (KLDF), where the nonlinear mapper was approximated by DCCA. We validated the proposed neural representations by examining the neural dynamics information presented in the neural trajectory of them and by decoding neural representations into various kinematic parameters. We compared KLDF with other neural representations obtained by SCC, FA, GPFA, and LFADS. We found that neural trajectories produced by KDLF revealed neural dynamics before and after movement onset more clearly than those by other methods (Figs.4-3 ― 4-5) and that decoding KDLF enabled a simple linear decoder to predict kinematic parameters more accurately than decoding other neural presentations (Figs. 4-6 ― 4-9). Notably, KDLF showed consistently high decoding performance for different monkeys performing dissimilar tasks.
To validate the decoding performance of the proposed approach, we predicted four kinematic parameters from KDLF, including velocity, acceleration, jerk, and speed, and compared decoding performance with those with four other neural representations. Decoding KDLF yielded higher performance for predicting velocity and acceleration compared to all other counterparts and slightly higher performance for predicting speed than LFADS that was the second-best model. Note that this study did not attempt to investigate the effect of using multiple stitched sessions, in which LFADS has Figure 4-9 Comparison of reconstruction errors for position. Open circle denotes average of the mean squared errors between actual and reconstructed position for tested trials, and each error bar indicates the standard error. Each color corresponds to monkey C, M, and F, respectively. Asterisks denote significant difference between neural representations (** p < 0.01, a two-way ANOVA by a post hoc analysis with a Tukey-Kramer correction for multiple comparison
74
shown advantages for maintaining decoding performance. Nevertheless, the decoding result in the present study suggests that the proposed neural representations of KDLF would improve motor decoding for intracortical BMIs. Note that LFADS does not support real-time generation of neural representations yet, whereas KDLF can be generated and decoded in real time once the training of DCCA is completed. A further study will investigate a feasibility to apply our method for stitched sessions to achieve stable online BMI performance.
According to [25], neural trajectories of a complex kinematic model present clear rotational patterns for behavioral events (e.g., a reach toward a specific target in a single trial). Based on the output-null model that describes how neural activity drives target muscles [35], we observed that the neural trajectory produced by KDLF remained constant before motion onset with a small variability across target directions, reflecting neural states in a null space, and that neural trajectories began to dynamically vary by sweeping the jPC space during arm movement after motion onset. These characteristics of neural trajectories associated with arm movements were most pronounced with KDLF, compared to other neural representations. Thus, we speculate that more reliable and clear representation of movements in KDLF might be linked to improved decoding.
Neural representations are related to generating muscle activation patterns involved in complex movements [23], [26], [35]. For instance, muscles that generate a specific movement would be driven by a linear combination of neural activities [35]. However, since it is difficult to predict complex kinematic information from noisy neural manifolds with a simple linear model, a nonlinear model such as deep neural networks (DNNs) has often been proposed [9], [71], [163]. In this study, we also utilized DCCA to approximate nonlinear mapping between neuronal population activity and KDLF that is supposed to represent various kinematic information. Although it is possible to use other models than DCCA for our approach, we adopted DCCA due to its several advantages. As an alternative approach to DCCA, we considered a multilayer perceptron (MLP) and compared its performance with DCCA on the prediction of KDLF. We observed that DCCA could yield significantly higher performance than MLP in terms of the reconstruction error (7.6 % higher on average). In addition, it has been demonstrated that DCCA can effectively improve velocity prediction accuracy by decoding canonical variates of DCCA and be used for extracting features from a large-scale dataset [37], [149].
It is noteworthy that the proposed neural representations appeared to provide movement-related information better when the arm movements became more complex: e.g., from 2D tasks (monkeys C and M) to 3D task (monkey F). This was observed in both neural trajectories and decoding performance.
We speculate that neural representations directly related to kinematic parameters (i.e., KDLF) might embed kinematic information more clearly and thus become more effective than other unsupervised neural representations for encoding more complex movements. Note that KDLF suggests a new way of dealing with the same latent factors of M1 population activity that are also processed by Gaussian process (GPFA) and dynamical systems (LFADS), with a more potential to improve motor decoding.
75
This study proposed a novel neural representation of neuronal population activity via supervised learning based on kinematic information. It thus seems natural to obtain results that neural representations found by supervised learning yielded better performance of decoding kinematics than those by unsupervised learning without any kinematic information. So, it is reasonable to consider that previous neural representations found in unsupervised ways are more useful for investigating intrinsic nature of neuronal population activity whereas our neural representations are more focused on extracting specific kinematics information. Yet, our approach may provide a better way to generate useful neural representations applicable to BMI.
76
CHAPTER 5 CONCLUSIONS AND FUTURE WORK
This dissertation has proposed a new approach to finding kinematic-driven neural representations from M1 population activity based on a set of indisputable results that could help better verifying the extent to which modeling and decoding with kinematics-dependent latent factors might be computationally efficient and analytically effective. The relationship between latent factors from neural population activity and complex kinematic information has been highlighted, with simulation studies under hypotheses [23], [25], [35] and the help of several dimensionality reduction methods [8], [24], [25].
Inferring intrinsic latent factors, representing neural states obtained by unsupervised-learning based dimensionality reduction, can be prominent a temporal networking variation of neural population activity. Nevertheless, it may be challenging to guarantee kinematic information in practical BMIs, since unsupervised learning based intrinsic latent factors are robust only if neurons involved in movements exist sufficiently. On the contrary, supervised learning gives a mean to estimate kinematics- dependent latent factors from M1 population activity, with kinematic information and without concern for irrelevant neurons to movements. The fundamental hypothesis adopted here is that "latent factors may be closely associated with generating activation patterns for the target muscles" [23], [25], [26], [35]. There is still controversy over the neural trajectory that describes the correlation between M1 and target muscles. However, I validated our approach by considering this theoretical phenomenon as a reasonable fact in some anatomical contexts, such as movement mechanisms of the brain or the relationship between muscles' movements and M1 output-null model already known. Interestingly, through our approach, it has become more evident that latent factors indeed are related to diverse kinematic variables by muscles involved in physical movements. The meaning is that plausible latent factors standing out kinematic information allow effective motor decoding that ensures high performance. The main findings in this dissertation that support these can be summarized as follows.
In the first study, described in Chapter 2, I investigated the effects of changes in the neuronal ensemble properties according to decoding models considering population neurons jointly or individually. I found that the decoding models based on covariances such as LKF and OLE yield effective decoding performance rather than PVA following independent tuning properties of individual neurons to kinematic variables. This suggests that it makes sense to consider population neurons jointly rather than consider them individually. Although this fact was already accepted as an empirical fact in many studies, I presented more apparent rationales by systematically controlling the various ensemble properties of population neurons through simulation. In the second study, described in Chapter 3, I examined the feasibility of decoding latent variables extracted by DCCA, which falls into supervised learning. I found that latent variables jointly estimated by DCCA with kinematic velocity provide beneficial influence on both linear and nonlinear decoding model. In particular, I revealed that these latent factors yield better decoding performance than neural representations obtained by the