interface based on the non-linear features extraction and stacking ensemble learning

(1)

interface based on the non-linear features extraction and stacking ensemble learning

Authors Asmaa Maher; Mian Qaisar, Saeed; N. Salankar; Feng Jiang;

Ryszard Tadeusiewicz; Paweł Pławiak; Ahmed A. Abd El-Latif;

Mohamed Hammad

DOI https://doi.org/10.1016/j.bbe.2023.05.001

Publisher Elsevier

Download date 21/06/2023 04:17:13

Link to Item http://hdl.handle.net/20.500.14131/906

(2)

Original Research Article

Hybrid EEG-fNIRS brain-computer interface based on the non-linear features extraction and stacking ensemble learning

Asmaa Maher

^a

, Saeed Mian Qaisar

^b,c

, N. Salankar

^d

, Feng Jiang

^a

, Ryszard Tadeusiewicz

^e

, Pawe ł P ł awiak

^f,g,^*

, Ahmed A. Abd El-Latif

^h,i

, Mohamed Hammad

^h,j,^*

aSchool of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

bLINEACT CESI, Lyon 69100, France

cDept. of Electrical and Computer Engineering, Effat University, 22332 Jeddah, Saudi Arabia

dPersistent Systems Limited, Nagpur, India

eAGH University of Science and Technology, Department of Biocybernetics and Biomedical Engineering, Krakow, Poland

fDepartment of Computer Science, Faculty of Computer Science and Telecommunications, Cracow University of Technology, Warszawska 24, 31-155 Krakow, Poland

gInstitute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland

hEIAS Data Science Lab, College of Computer and Information Sciences, Prince Sultan University, Riyadh 11586, Saudi Arabia

iDepartment of Mathematics and Computer Science, Faculty of Science, Menoufia University, 32511, Egypt

jDepartment of Information Technology, Faculty of Computers and Information, Menoufia University, Egypt

A R T I C L E I N F O Article history:

Received 14 February 2023 Received in revised form 25 April 2023

Accepted 16 May 2023 Available online 24 May 2023

Keywords:

Hybrid BCI

Electroencephalogram Ensemble learning Genetic Algorithm

A B S T R A C T

The Brain-computer interface (BCI) is used to enhance the human capabilities. The hybrid- BCI (hBCI) is a novel concept for subtly hybridizing multiple monitoring schemes to maxi- mize the advantages of each while minimizing the drawbacks of individual methods.

Recently, researchers have started focusing on the Electroencephalogram (EEG) and ‘‘Func- tional Near-Infrared Spectroscopy” (fNIRS) based hBCI. The main reason is due to the development of artificial intelligence (AI) algorithms such as machine learning approaches to better process the brain signals. An original EEG-fNIRS based hBCI system is devised by using the non-linear features mining and ensemble learning (EL) approach. We first dimin- ish the noise and artifacts from the input EEG-fNIRS signals using digital filtering. After that, we use the signals for non-linear features mining. These features are ‘‘Fractal Dimen- sion” (FD), ‘‘Higher Order Spectra” (HOS), ‘‘Recurrence Quantification Analysis” (RQA) features, and Entropy features. Onward, the Genetic Algorithm (GA) is employed for

https://doi.org/10.1016/j.bbe.2023.05.001

0168-8227/Ó2023 The Author(s). Published by Elsevier B.V. on behalf of Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

* Corresponding authors.

E-mail addresses: asmaa_maher2219@yahoo.com(A. Maher),smianqaisar@cesi.fr (S. Mian Qaisar),rtad@agh.edu.pl(R. Tadeusie- wicz),pawel.plawiak@pk.edu.pl (P. Pławiak), aabdellatif@psu.edu.sa(A.A. Abd El-Latif), mohammed.adel@ci.menofia.edu.eg, mham- mad@psu.edu.sa(M. Hammad).

A v a i l a b l e a t w w w . s c i e n c e d i r e c t . c o m

ScienceDirect

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / b b e

(3)

Motor Imagery Tasks

Non-Linear Features Extraction

Features Selection (FS). Lastly, we employ a novel Machine Learning (ML) technique using several algorithms namely, the ‘‘Naı¨ve Bayes” (NB), ‘‘Support Vector Machine” (SVM), ‘‘Ran- dom Forest” (RF), and ‘‘K-Nearest Neighbor” (KNN). These classifiers are combined as an ensemble for recognizing the intended brain activities. The applicability is tested by using a publicly available multi-subject and multiclass EEG-fNIRS dataset. Our method has reached the highest accuracy, F1-score, and sensitivity of 95.48%, 97.67% and 97.83%

respectively.

Ó2023 The Author(s). Published by Elsevier B.V. on behalf of Nalecz Institute of Biocyber- netics and Biomedical Engineering of the Polish Academy of Sciences. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction

The brain-computer interface (BCI) design and deployment have evolved in recent decades. The BCI develops a link between the human brain and the real world. Through a BCI, the subject is able to communicate with or operate devices outside of their physiological motor system by using their brain activity[1]. The BCIs are currently employed in various domains namely, medicine, education, entertainment, gaming, marketing, and self-control[1,2]. The invention of BCI has particularly revolutionized the mental health and rehabilitation related solutions [3]. People suffering from cerebral palsy, and motor neuron like brain diseases may benefit from BCI. This transformation takes place through ‘‘event- related synchronization” (ERS) and ‘‘event-related desynchro- nization” (ERD).

Combined with the Artificial Intelligence (AI), the BCI has started growing rapidly as a new technology. It is capable of giving a direct way of communication between an otherwise silent brain and computing devices, smartphones, wearables, e-textile, Body-Area Network (BAN) and so on[1].

The effective BCIs can be realized by acquiring and processing the electroencephalogram (EEG) signals of the intended subjects [1]. The EEG signals can be collected non-invasively or invasively. Many researchers employ non- invasive BCIs because of their easy mounting[4]. The processing of non-invasive EEG signals permit to detect the motor imagery (MI) tasks, mental arithmetic (MA) activities, tongue movements, and left/right hand/arm/foot gestures[5,6].

The automated categorization of motor imagery (MI) tasks is the base of several contemporary BCIs[7]. The MI refers to a human brain’s ability to replicate motor events without any visible motions. MIs can be used to investigate processes of human cognition and motor activity because they can appear con- sciously or be originated and controlled by the individuals[8–10].

The machine-based identification of MI tasks needs pre- processing of EEG signals, feature extraction from the processed signals, dimension reduction, and classification [11].

Computer-based automated MI signals detection is crucial for continuously assisting the patients[7]. The MI tasks generate event-related EEG signals which can be identified by analyzing the alpha and beta bands of the BCI systems[12].

Moreover, oscillatory and empirical mode decompositions and time–frequency analysis of the EEG signals can also lead towards an effective identification of the MI tasks[13].

Ideally, the signal collector should be able to acquire the real-time data about brain activity in an efficient and stable manner. It is mandatory to enhance the applicability and effectiveness of BCIs. The EEG[14], ‘‘Functional Magnetic Res- onance Imaging” (fMRI) [15], ‘‘Magnetoencephalography”

(MEG) [3], and ‘‘Functional Near-Infrared Spectroscopy”

(fNIRS)[16]are currently common BCI monitoring methods, each with its own benefits and drawbacks. In recent years, the use of fNIRS for BCIs has gotten a lot of attention[16].

The identification of various mental processes is feasible, in fNIRS, by carefully observing variations in the ‘‘oxygenated hemoglobin” (HbO) and ‘‘deoxygenated hemoglobin” (HbR) concentrations[3]. This relatively new approach has already demonstrated its use in BCI applications.

The contemporary BCIs have their limitations. The major issue in hBCI applications is to improve the categorization accuracy while lowering the system complexity and improving the response time[17]. It is demonstrated that combining EEG characteristics with those generated from fNIRS signals can improve the BCI performance in categorizing the MI task, MA activities, hand rotations and hands motions. The EEG- fNIRS based multimodal BCIs have showed improved classification accuracy. However, most of the published research works have simply used all channels for classification, which always results in a large features dimension[17]. This neces- sitates a higher computational effort during the training and testing phases. Feature dimensions can be reduced by applying spatial filtering methods to the EEG-based BCIs. The process is based on the usage of neurophysiological optimization criteria[17]. In this context, the authors in[17]

have sued the ‘‘common spatial pattern” (CSP) approach for the realization of viable hBCI [17]. Other approaches used the ‘‘principal component analysis” (PCA) to identify the pertinent attributes among the considered categories[18].

The processing cost and latency of classification stage is determined by the number of features per instance. The right amount of relevant features need to be extracted from EEG- fNIRS signals in order to obtain high classification accuracy and computational effectiveness [11]. This is the gap that we have tried to bridge through the proposed approach. In this work, we combine the non-linear feature extraction with stacking ensemble learning technique and genetic optimization for the EEG-fNIRS based hBCI systems, which decrease the number of extracted features and reduce the computational complexity. Finally, the proposed approach achieved a

(4)

very good accuracy compared with other state-of-the-art in this field.

Section 2discusses the related works in this field and their limitations. The Section 3 discusses the proposed methods and the dataset which is used to evaluate our method.Sec- tion 4displays results with an analysis. The findings are discussed and compared with previous works in Section 5.

Finally,Section 6makes the conclusion.

2. Related work

An intelligent combining of several monitoring methods can result in a novel strategy that synthesizes the benefits of each technology while also overcoming their drawbacks. One of the practical approaches for enhancing the performance is to use a hybrid EEG-fNIRS BCIs (hBCI) system[3]. It detects both electrical and hemodynamic activities in the cerebral cortex, at the same time, in order to blend their properties in ‘‘near-real-time” and deliver more precise brainwave information. This multimodality offers a unique processing technique for extending the capabilities of existing BCI applications[16]. Because both modalities can be easily made portable and the two signals do not interfere substantially.

The EEG-fNIRS based hBCIs has been extensively researched among the contemporary techniques. The electrical activity of the brain is recorded via the EEG, and the hemodynamic changes in the brain are measured via the NIRS. Because the two signals’ sources are different, the quantity of accessi- ble information for the BCIs is increased, resulting in an effective realization.

In literature many authors have attempted the multimodality combination of EEG + fNIRS for various BCIs realiza- tions. Significant results attained with an appropriate selection of electrodes in both EEG and fNIRS.

Recent research has shown that the hBCI can be used to communicate with patients in a totally locked-in state (CLIS) [19]. Purely EEG-based BCIs, on the other hand, have never proven effective with the CLIS patients[20]. It confirms that combining the EEG-fNIRS modalities, in the right way, can augment the BCIs performance, as prior researchers have demonstrated[9,16,19].

Injuries or diseases to the nervous system are likely to produce the loss of motor function [8]. Also, the shortfall of attention can limit the involvement of concerned persons in daily activities[21]. As a result of these BCI assistive technolo- gies, wheelchairs, computers, and artificial prostheses can be controlled by the electrical activity of the brain. However, pre- cision and processing effectiveness of BCI systems are the main challenges[10]. Additionally, it permits the identification of psychiatric illnesses[21].

The neuronal and hemodynamic components of brain activity can be analyzed with a superior temporal and spatial resolutions by simultaneously using the EEG and fNIRS sensors. Therefore, a better insight and performance is attainable by using an effective hybridization of the EEG-fNIRS based hBCI strategies but at the cost of an increased data dimen- sionality and computational cost. Obtaining a real-time response requires the use of advanced and resourceful infor-

mation selection techniques. In this situation, several investi- gations have been presented[22–24].

In some applications identification of left- and right-hand movement plays important role and it is also effective for motor disable persons. In [22], 4-channels of EEG and 20- channels of fNIRS are used for discrimination between left and right hand movement. Wavelet algorithms were used to extract an approximate coefficient from EEG signals and fNIRS signals were used to determine slope. The PCA is used for selecting the pertinent features. The selected features are used to identify the left-hand and right-hand grasping activities. The classification is carried out by using the ‘‘linear discriminant analysis” (LDA). It is shown that the hybrid approach permits to tackle the limitation associated with fNIRS slope of Oxy- genated Hemoglobin (RbO), measured after occurrence of stim- ulus. In[23]authors pursued to distinguish left- and right-hand actions using a multisensory EEG + fNIRS combination. The EEG and fNIRS channels were chosen using the source analysis.

The EEG channels are used to extract phase-space reconstruction-based features. The fNIRS signals are used to extract Hurst exponents as features. The SVM classifier combines and processes the retrieved features from EEG and fNIRS signals. In [18], the ‘‘Discrete Wavelet Transform”

(DWT) was utilized to derive the approximation coefficients of the EEG signals. The PCA is used to denoise fNIRS signals, and mean HbO and HbR fluctuations are derived from the denoised fNIRS signals. Left- and right-hand movements are categorized using the SVM. In[25], a novel approach of Mul- tiresolution Singular Value Decomposition (MSVD) has been used in order to achieve the EEG-fNIRS based hBCI. The MSVD permitted the features extraction from a considered range of frequency bands. The KNN is used to identify the movements of the left and right hands, as well as the left and right arms.

The integration of EEG+fNIRS is employed in[26]to efficiently operate the brain-controlled switch. In [27] authors mined features by using the typical spatial pattern method. For the categorization of MI tasks, the Shrinkage Linear Discriminant Analysis (sLDA) is used. The ‘‘Mel-cepstral” coefficients with statistical features are derived from the fNRIS signals and processed by the SVM in[28]for the MI task identification.

The hBCI is also used for the categorization and identification of arithmetic activities and mental workloads. In [24]

authors have processed the EEG-fNIRS signals for an effective identification of the mental workload tasks. They used the Gramian Angular Summation Field (GASF) for conversion of time series data in image format. Onward these images are processed with Convolutional Neural Networks (CNNs) for the identification of different mental workloads. In [29], authors have identified the mental stress by using the Deep Belief Network (DBN). The changes occur in EEG and fNIRS signals are detected to identify between rest and mental stress states. The timeseries data is directly processed by the proposed DBN for classification. In [30], authors used the Point-Biserial Correlation Coefficient (PBCC) for features extraction. The classification between the mental arithmetic and baseline tasks is performed by using the sLDA.

Awareness about fatigue level of operator in any industry saves human lives and serves safety aspect, in[31]pilot cog-

(5)

nitive activity has been monitored by using an effective fusion of the EEG and fNIRS signals. The engagement ratio and wavelet coherence-based features are mined from the EEG-fNIRS signals. The classification task is carried out by using the sLDA.

3. Materials and methods

The processing stages of the designed approach are shown in Fig. 1. This method is based on non-linear features mining and ensemble learning methodologies. The functioning steps are: 1) pre-processing to eliminate noise and artifacts from the incoming signals, 2) extracting non-linear features from each pre-processed signal, 3) feature selection and optimization using Genetic Algorithm (GA) approach to select the most valuable features that render the highest accuracy, and 4) the classification using the ensemble learning approach for the final decision.

3.1. The dataset

In this research, a dataset from 18 healthy participants (eight women and ten men) with an average age between (21.3, 26.3) years is considered for evaluating the performance of devised method [19]. The EEG and fNIRS signals are respectively acquired at the sampling rates of 2048 Hz and 13.3 Hz respectively. Details about the placement of EEG and fNIRS sensors on the brain are detailed in[19]. The data is categorized in three classes namely, the ‘‘Idle State” (IS), ‘‘Mental Arith- metic” (MA) and ‘‘Motor Imagery” (MI). We focused on four classification problems: i) MI vs. MA (P1), ii) MI vs. IS (P3), iii). MA vs. IS (P3) and iv). MI vs. MA vs. IS (P4). We hypothesized that these classifications findings would be suitable for implementation of the BCIs.

Data size: It contains EEG + fNIRS signals from 18 participants. A single trial is made up of ‘‘instruction” (2 to 0 sec), ‘‘task” (0–10 sec), and break between consecutive tri- als (10 to 26–28 s).

Fig. 1 – The proposed system block diagram.

Testing conditions: During the session, the participants were seated comfortably on chairs. Instructions are dis- played on a 26-inch ‘‘liquid crystal display” monitor.

3.2. Preprocessing stage

We use a third-order Butterworth linear phase filter to remove

‘‘baseline drift”, ‘‘power line interference”, and ‘‘high frequency noise” from the EEG signals. Onward, we down- sampled all EEG signals to 200 Hz, which reduce the computational complexity of our method. Then the EEG signals are further denoised using the wavelet-based method proposed by Sharma et al. [32]to remove the white Gaussian noises and muscle artifacts.

3.3. Non-Linear features extraction

In this stage, we employed new non-linear features for our BCI system, which are higher order spectra (HOS)[33], recurrence quantification analysis features (RQA)[34], entropy features [35] and fractal dimension (FD) [36]. According to previous studies in EEG classification[36], the non-linear features are the efficient way for EEG analysis and considered the commonly features used in this area because of the random nature of the EEG signals. The description of these features is given below:

HOS: It consists of two-spectra. One is defined for deter- ministic signals, which is called higher-order moment spectra and the other one is defined for random processes, which is called cumulant spectra. We employed HOS as a non-linear feature to detect and characterize nonlineari- ties in the EEG signals.

RQA: It is a method of numerical analysis that is well sui- ted for nonlinear signals. It quantifies the frequency and duration of recurrences in phase space. We employed these features to evaluate the complexity of the EEG sig-

(6)

nals. The main features of the RQA are listed in Table 1.

Where, Accuracy (Acc) and F1-score are respectively defined by Eq.(2)and Eq.(3).

FD: is one of the measures of signal complexity. Conse- quently, EEG signals and details may be detected visually that are not obvious.

The mathematical descriptions of the features are listed below (Eq.(1)to Eq.(9)):

RR¼ 1 N²

X^N

i;j¼1

R ið Þ;j ð1Þ

WhereR(i,j) =ɵ(e-k!xð Þ i !ðjx ÞkÞ, whereɵ:R?(0,1) and e is a predefined distance. Also, !xð Þi is the phase space trajectory.

DET¼ PN

l¼lminlPðlÞ PN

l¼1lPðlÞ ð2Þ

WhereP(l)is the frequency distribution of the lengths of the diagonal lines.

ENT¼ X^N

l¼lmin

P lð ÞlnPðlÞ ð3Þ

MDL¼ PN

l¼lminlPðlÞ PN

l¼lminPðlÞ ð4Þ

Bispectrum f ₁;f₂

¼F f₁ :F f2 :Fðf1þf₂Þ ð5Þ WhereFis the ‘‘Fourier transform” of EEG signal and F* is the conjugate of F.

ShEn¼ Xⁿ

i¼1

xi:log_x

i ð6Þ

wherexiis the mass fraction for each compound that consti- tute the mixture.

ReEn¼ 1

1alogðXⁿ

i¼1

P^a_iÞ ð7Þ

wherePiis the corresponding probabilities fori= 1,. . .,n SamEn¼ logZ

W ð8Þ

where Z presents the count of template vector pairs having d X½ _mþ1ð Þ;i X_mþ1ð Þj<rand W is the number of template vector pairs havingd X½ mð Þ;i Xmð Þj<r, For a given embedding dimension m, r tolerance and N number of data points.

FD¼1þ lnðLÞ

lnð2ðN1ÞÞ ð9Þ

Where, L is the total length of the data, andNis the total number of data points.

3.4. Genetic Algorithm (GA) based features selection In the third stage, the unnecessary features are removed and the best features are selected. We employed the GA for this purpose. It lowers the processing cost and boosts the system’s overall performance. In GA an efficient iterative use of genetic operator plays a very crucial role to produce the selected population from initial one. The key ingredients are: i) current chromosome representation, ii) Selection, iii) crossover, iv) mutation and v) fitness. Mathematically the process is given by Eq.(10). (D. Liu, 2019).

R¼ðGþ2p Þffiffiffig

3G ð10Þ

Table 1 – Summary of the Extracted Features.

No. Features Acc. ± F1-score

1 RQA Recurrent Rate 1 (RR1) 0.751 ± 0.793

2 Recurrent Rate 2 (RR2) 0.80 ± 0.842

3 Determinant 1 (DET 1) 0.895 ± 0.939

4 Determinant 2 (DET 2) 0.875 ± 0.917

5 Entropy 1 (ENT 1) 0.718 ± 0.759

6 Entropy 2 (ENT 2) 0.715 ± 0.754

7 Mean diagonal length 1 (MDL 1) 0.814 ± 0.858

8 Mean diagonal length 2 (MDL 2) 0.812 ± 0.854

9 Recurrence time entropy (RTE) 0.775 ± 0.807

10 Longest diagonal line (DD) 0.821 ± 0.857

11 HOS HOS 1 0.714 ± 0.750

12 HOS 2 0.712 ± 0.747

13 HOS 3 0.711 ± 0.746

14 HOS 4 0.698 ± 0.740

15 Bispectrum 1 0.724 ± 0.760

16 Bispectrum 2 0.741 ± 0.787

17 Entropy Shannon Entropy (ShEn) 0.761 ± 0.795

18 Renyi Entropy (ReEn) 0.762 ± 0.798

19 Sample Entropy (SamEn) 0.850 ± 0.886

20 Permutation Entropy (PeEn) 0.697 ± 0.738

21 Approximate Entropy (ApEn) 0.825 ± 0.862

22 FD FD 0.740 ± 0.779

(7)

here G is the total number of evolutionary generations defined by the population.g denotes the number of generations andRrepresents the range. In terms of the initial and final iterations, the value ofRshould vary from low to high.

The purpose behind is if at initial levelRvalue is high then it might destroy an excellent genetic schema.

The GA is used to optimize the system performance in terms of computational effectiveness and accuracy by selecting the best non-linear features. The process is depicted as follows:

Initial Population: Based on a random number function that returns either 1 or 0, the genes of the N chromosomes (N features) of the initial population will be filled in. Each individual will contain M genes (M EEG signals) with 22 chromosomes in the initial population.

Selection: To create offspring, a selection policy selects the characteristics that should be mated to produce them. For selecting which individuals to use, we apply a selection policy that uses the fitness function. Three genetic operators are employed to select the features and find the best solution.

Selecting parent solutions is done using the roulette wheel.

A single point crossover is used to generate descendants, and these descendants are mutated via a single point mutation to offer variation.Fig. 2shows the steps of GA for opti-

mization and selection of the EEG features. The parameters of GA were presented inTable 2.

The fitness curve of GA is shown inFig. 3. The GA optimization was used to filter a set of input arguments based on the average and best fitness, after a series of iterations.

Finally, the selected features after applying GA will be RR2, DET1, DET2, MDL1, MDL2, DD, ShEn, ReEn, SamEn and ApEn.

3.5. The ensemble learning

The ensemble learning model is one of the most successful methods for improving a system’s performance. It is the process of integrating a variety of separate models to improve the model’s stability and predictive capacity. The following are some of the most often utilized ensemble learning techniques:

1)Bagging: Similar learners are used on small sample pop- ulations in this technique, and the mean of all predictions is obtained.

2)Boosting: In this iterative process, the weight of an obser- vation is adjusted depending on the categorization that occur before it.

3)Stacking: It uses a learner to combine the outcome from multiple learners. Depending on the used combiner, this might result in a diminishing in either the bias or variance error.

Fig. 2 – The GA process for selecting the features.

Table 2 – Hyperparameters of GA.

Parameter Value

Selection algorithm Roulette wheel Crossover method single point crossover Mutation method single point mutation Probability of crossover 0.8

Probability of mutation 0.8

Population size 18

Fitness function accuracy or F1-score

Maximum generation 100

Fig. 3 – The Fitness curve of GA.

(8)

In this study, in order to get a higher accuracy of classification, we employed stacking technique as an ensemble learning model for our system. In addition, we employed several classifiers to train on the same dataset, which gives high performance of the system when using stacking with this case.

This technique improves classification accuracy over individual classifiers, due to the usage of multiple classification models that have different abilities in solving the classification issues. In this case, we employed several classifiers such as SVM, KNN, ‘‘Naı¨ve Bayes” and ‘‘random forest” (seeTable 8 to find the parameters of each classifier) to obtain the probability of each class which is calledmeta-features. After that, we fed thesemeta-features to ameta-classifier[37]to perform the final classification.

3.6. Cross validation

Cross-validation is a technique that combines partially visible and unseen data to generate alternative validation combina- tions. Cross-validation strategies include the k-fold cross- validation procedure[38]. The data set is split inkequal sec- tions in order to preserve the percentage of samples from each class in each fold. In our study, 10-fold cross-validation scheme (the popular technique) is employed.

3.7. Evaluation measures

We used four typical measures to evaluate the suggested method: ‘‘accuracy” (ACC), ‘‘F1-score” (F1), ‘‘Kappa index”

(Kappa) and the ‘‘area under the ROC curve” (AUC).

The accuracy and F1-score are considered as the fitness function in our method. These are defined by following equations.

Accuracy¼ TNþTP

TNþTPþFNþFP ð11Þ

F1score¼2^TPþFN^TP _TPþFP^TP

TPþFNTP þ_TPþFP^TP ð12Þ

Kappa¼NPI

i¼1xiiPI i¼1xixj

N²PI i¼1xixj

ð13Þ We also employed common metrics such as sensitivity (Se) and specificity (Sp) to make a fair comparison between the proposed system and other previous systems.

Se¼ TP

TPþFN ð14Þ

Sp¼ TN

TNþFP ð15Þ

WhereTN,TP,FNandFPare clear fromTable 3,x_iiis the total count ofTPandTN,Nis the number of participants,xi

andxjare the total count of columns and rows, respectively.

4. Results

Our model is developed using the ‘‘MATLAB” 2019a. The

‘‘Microsoft Windows 10” Pro 64-bit is the ‘‘operating system”.

The proposed ensemble learning model is trained on ‘‘Intel Core i5-7300HQ 2.5 GHz” with 16 GB of memory.

In total 22 non-linear attributes were retrieved from the incoming signals (Table 1). The graphical depiction of the 10 selected attributes, contributed in attaining the best accuracy, is shown in Fig. 4. The data is categorized inthree classes namely, the ‘‘Idle State” (IS), ‘‘Mental Arithmetic” (MA) and

‘‘Motor Imagery” (MI). We focused onfourclassification problems: i) MI vs. MA (P1), ii) MI vs. IS (P3), iii). MA vs. IS (P3) and iv). MI vs. MA vs. IS (P4). We hypothesized that these classifications findings would be suitable for implementation of the BCIs.Fig. 5shows plots of the typical recurrence of MI and MA signals, respectively.

Each classifier undergoes feature selection and parameter tuning. ACC and F1 were selected as fitness functions of GA.

The best model achieved 95.48% accuracy rate on testing set.Tables 4–7show confusion matrix for this model in case of intended four problems.

FromTable 4, it can be noted that 95.49% of the MA signals are classified correctly as MA signals and 4.5% of the MI signals are wrongly classified as MA signals using the proposed method on the first classification problem. Also, fromTable 5, the proposed method classified 90.09% of MA correctly when dealing with problem ii and 13.83% of the IS signals are wrongly classified as MA signals. In addition, when working on problem iii, 68.18% of the MI signals is corrected classified as MI signals and 9.82% of IS signals are wrongly classified as MI signals as shown inTable 6.Table 7demonstrates that the suggested technique can accurately solve multi-class classification problems. According to the Table, 88.28% of MA classified correctly as MA when dealing with problem iv and 9% and 2.70% of MA classified wrongly as MI and IS Table 3 – Standard Confusion Matrix.

‘‘Actual” ‘‘Predicted”

‘‘Positive” ‘‘Negative”

‘‘Positive” ‘‘True Positive”

(TP)

‘‘False Negative”

(FN)

‘‘Negative” ‘‘False Positive”

(FP)

‘‘True Negative”

(TN)

Fig. 4 – Graphical representation of the 10 features with highest accuracies (the symbols of features used on the figure are explained inTable 1).

(9)

Fig. 5 – Plot of typical recurrence for MI (a) and MA (b).

(10)

signals, respectively. In addition, from the Table, 81.81% of MI signals are classified correctly as MI and 93.30% of IS signals are correctly classified as IS signals.

Table 8shows the selected features together with the classifier information for the best model, where 1 and 0 corre- spond to the picked and rejected features, respectively.

Figs. 6–8show examples of plots of the ROC curves. The ROC curve is a plot between the TP-rate (y-axis) and the FP- rate (x-axis). The AUC value can be computed from the ROC curve.

Tables 9–11show the performance of the proposed method in terms of Acc, Sp, F1, Kappa, Se, and AUC using intended classification algorithms and for different classification problems (P1, P2, P3, and P4). From the Tables, we can find that the Table 4 – Confusion Matrix for the Best Model for problem i

(P1).

Actual Predicted

MA MI

MA 106 5

MI 2 42

Table 5 – Confusion Matrix for the Best Model for problem ii

(P2).

Actual Predicted

MA IS

MA 100 11

IS 31 193

Table 6 – Confusion Matrix for the Best Model for problem iii

(P3).

Actual Predicted

MI IS

MI 30 14

IS 22 202

Table 7 – Confusion Matrix for the Best Model for problem iv

(P4).

Actual Predicted

MA MI IS

MA 98 10 3

MI 6 36 2

IS 9 6 209

Table 8 – Parameters of Different Classifiers.

Classifier Parameters

SVM ’linear’, (kernel, degree, gamma), [0,1,1,1,0,0,1,1,0,1,0,0,0,0,0,1,1,1,1,0,1,0]

Random forest [1,1,1,1,0,0,1,1,1,1,0,0,0,0,0,0,1,1,1,0,1,0]

KNN 3, ’uniform’ (k, metric, weight) [0,1,1,1,0,0,1,1,0,1,0,0,0,0,0,0,0,0,1,0,1,0]

Naı¨ve Bayes [1,1,1,1,0,0,1,1,1,1,0,0,1,0,0,1,1,1,1,1,1,0]

Meta NuSVM ’rbf’, (kernel, Nu, degree, gamma) [0,1,1,1,0,0,1,1,0,1,0,0,0,0,0,0,1,1,1,0,1,0]

Fig. 6 – ROC curve of our method for P1.

Fig. 7 – ROC curve of our method for P2.

(11)

proposed method using the meta-classifier NuSVM achieved the highest performance for all metrics among all classifiers for the four problems. The SVM classifier also obtained a good performance in most cases comparing to other classifiers but still lower than themeta-classifier.

5. Discussion

The results outlined previously show that for most cases the proposed method achieved a good performance using the hBCI system. Comparing determinant features to other features, they gained the highest accuracy (see Table 1). EEG determinants are evaluated based on spike occurrences, patterns, and subtle changes in the signals. Consequently, there is a higher RQA parameter for EEG signals with more rhyth- mic foci, which may be useful in identifying hidden signals.

Most of the entropy features attained higher accuracies in the case of MI EEG signals, which are less randomly and more periodically. These features are namely, the SamEn, ApEn and ReEn.

FromTables 9 to 11, we can find that working on one-to- one classification problems (i.e., P1, P2 and P3) obtained high Fig. 8 – ROC curve of our method for P3.

Table 9 – The Acc (%) and Sp (%) of different classifiers.

(% age) Acc (% age) Sp (% age)

Classifier/Dataset P₁ P₂ P₃ P₄ P₁ P₂ P₃ P₄

SVM 90.27 81.60 83.42 84.14 95.05 79.35 81.45 85.34

Random forest 89.50 79.20 82.44 82.61 92.91 77.59 82.08 85.50

KNN 87.84 75.21 78.40 79.30 88.03 74.36 79.59 79.09

Naı¨ve Bayes 87.42 74.40 73.60 76.61 87.86 70.27 71.90 76.43

Meta NUSVM 95.48 87.46 86.56 90.51 77.77 87.44 85.27 91.38

Table 10 – The F1 and Kappa values of different classifiers.

F1 (% age) Kappa (% age)

Classifier/Dataset P1 P2 P3 P4 P1 P2 P3 P4

SVM 94.51 78.68 82.11 83.96 88.56 55.20 63.65 65.50

Random forest 93.93 76.30 81.19 81.40 85.50 52.64 62.73 63.37

KNN 90.84 69.80 77.05 78.57 80.61 45.40 53.54 54.91

Naı¨ve Bayes 90.50 66.95 67.81 73.39 79.92 43.29 44.30 49.73

Meta NUSVM 97.67 86.19 85.35 90.11 94.64 70.46 69.06 80.38

Table 11 – The Se and AUC results of different classifiers.

Se (% age) AUC (% age)

Classifier/Dataset P1 P2 P3 P4 P1 P2 P3 P4

SVM 94.52 86.79 86.46 83.18 94.16 77.36 82.22 84.07

Random forest 86.61 82.98 84.47 80.44 93.26 74.98 81.30 81.94

KNN 87.18 81.95 78.36 80.37 90.33 67.74 75.70 77.25

Naı¨ve Bayes 85.24 84.57 80.23 78.11 90.04 65.63 66.46 72.07

Meta NUSVM 97.83 88.95 85.72 90.72 97.54 85.26 84.54 90.32

(12)

performance especially in problem P1 (MI Vs. MA) because of the nature of MI and MA signals, which presents in an infor- mative manner the by using the proposed features set. For multiple classification problem (P4), we also obtained a good performance comparing with P2 and P3 as we mentioned before the nature of extracted features makes it easy to detect the MI and MA signals correctly. In addition, the number of signals used in multiclass is more than the signals used in the binary class problems. It also effects on the final accuracy.

However, multiclassification is more complex and also needs more processing time. In addition, sometimes the normal cases may affect with several noise as discussed in[39], which may affect the classification accuracy in some cases. There- fore, we recommend using binary classification (MI Vs. MA) for the implementation of our BCI system, which is the most widely used classification problem in previous works.

Fig. 5shows that the MI EEG signal is more homogeneously and periodic compared to the MA EEG signal, which is more random and inconsistent. It proves that the extracted nonlinear features can be efficaciously used for the categorization of MI and MA tasks.

Comparing the BCI to state-of-the-art performance is a tedious task since the BCI domain has been well explored.

The combination of datasets, pre-processing, feature extraction, dimension reduction, and classification approaches used in earlier research is the fundamental reason for their success. To make it more comprehensive,Table 12compares the proposed framework to state-of-the-art methods utilizing hBCI datasets. In[22]the authors have employed the wavelet generated coefficients with slope information for the features mining. Onward, the PCA is used for dimension reduction and the selected feature set is classified with the LDA classifier.

The highest reported accuracy is of 86.00%. In [23], the phase-space reconstruction-based Hurst exponents are mined and used to prepare the feature set. The highest categorization accuracy of 81.20% is reported for the case of SVM classifier. The GASF is used for converting the time series data in images[24]. Onward, these images are processed by the CNN classifier for an automated categorization of mental

tasks. The highest categorization accuracy of 89.00% is reported. In [18], the wavelet generated coefficients with mean values of the HbO and HbR fluctuations are mined as features. Onward, the PCA is used for the dimension reduction. The categorization is carried out using the SVM classifier. The highest classification accuracy of 91.02% is reported. The MSVD is used for mining the sub-bands features in[25]. Onward, the KNN based categorization is carried out while securing the highest classification accuracy of 90.61%. In[29], the authors have identified the mental stress by processing the timeseries data via the DBN. The highest accuracy of 85.62% is reported while identifying between the rest and stress conditions. The PBCC based features extraction is performed in [30]. The feature set is classified with the sLDA. The highest categorization accuracy of 88.2% is reported while identifying between the mental arithmetic and baseline tasks. The Engagement ratio plus wavelet coherence-based features are mined in [31]. Onward, the sLDA is sued for the categorization purpose while securing a classification accuracy of 87.60%.

The above presented facts confirmed that compared with previously presented methods, the proposed method outper- forms. The findings suggest that the pre-processing, nonlinear feature selection, dimension reduction, and stacking ensemble learning technique proposed has a significant impact on the system’s accuracy and performance. Ensemble classifiers have the benefit of being able to adapt to a specific training dataset. By incorporating the event-driven tools, it may be feasible to improve the computational performance of the described technique [40]. This aspect can be investigated in future.

6. Conclusion

An effective method for the EEG-fNIRS based hBCI system is presented. It is based on a novel mix of the non-linear features mining, stacking ensemble learning methods, and genetic optimization. The proposed method achieved a superior performance comparing with previous works. Four classi-

Table 12 – Comparison with state-of-the-art solutions.

Study Pre-processing/Features Extraction Classifier Subjects Acc.(%age)

(Zhu et al., 2017)[22] Wavelet generated coefficients + Slope information + PCA

LDA 3 86.00

(Ge et al., 2017)[23] Phase-space reconstruction + Hurst exponents

SVM 12 81.20

(Saadati et al., 2019)[24] GASF CNN 26 89.00

(R. Li et al., 2017)[18] Wavelet generated coefficients + ‘‘Mean values of the” HbO and HbR fluctuations + PCA

SVM 11 91.02

(M. U. Khan & Hasan, 2020)[25] MSVD KNN 15 90.61

(Ho et al., 2019)[29] Not Applicable DBN 16 85.62

(Shin et al., 2017)[30] PBCC sLDA 11 88.2

(Dehais et al., 2018)[31] Engagement ratio + Wavelet coherence sLDA 4 87.60

This Study Non-linear Features Stacking ensemble

learning

18 P1:95.83 P2:87.46 P3:86.56 P4:90.51

(13)

fication problems are studied. The system respectively secures the highest accuracy of 0.9548, specificity of 91.38,

‘‘F1-score” of 0.9767, sensitivity of 0.9783, Kappa of 0.9464 and AUC of 0.9754. It confirms the potential of integrating the devised method in the future generation of hBCI systems.

Our suggested approach performed well for the considered dataset. In future its applicability will be investigated for other potential hBCI datasets. Another area to investigate is the use of event-driven processing and deep learning techniques.

CRediT authorship contribution statement

Asmaa Maher: Conceptualization, Methodology, Investiga- tion, Writing – original draft, Writing – review & editing.

Saeed Mian Qaisar: Validation, Formal analysis, Resources, Writing – original draft, Writing – review & editing.N. Salan- kar:Software, Validation, Resources.Feng Jiang:Supervision, Visualization.Ryszard Tadeusiewicz:Supervision, Visualiza- tion, Writing – review & editing. Paweł Pławiak: Writing – review & editing, Visualization, Supervision, Project adminis- tration. Ahmed A. Abd El-Latif: Supervision, Visualization.

Mohamed Hammad: Methodology, Investigation, Writing – original draft, Writing – review & editing, Visualization, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work is technically supported by the Harbin Institute of Technology, LINEACT CESI, Effat University, AGH University of Science and Technology, Cracow University of Technology, Polish Academy of Sciences, EIAS Data Science Lab at Prince Sultan University, and Menoufia University.

R E F E R E N C E S

[1]Mudgal SK, Sharma SK, Chaturvedi J, Sharma A. Brain computer interface advancement in neurosciences:

Applications and issues. Interdiscipl Neurosurg 2020;20 100694.

[2]Rashid M, Sulaiman N, Abdul Majeed PPA, Musa RM, Bari BS, Khatun S. Current status, challenges, and possible solutions of EEG-based brain-computer interface: A comprehensive review. Front Neurorob 2020;14:25.

[3]Liu Z, Shore J, Wang M, Yuan F, Buss A, Zhao X. A systematic review on hybrid EEG/fNIRS in brain-computer interface.

Biomed Signal Process Control 2021;68 102595.

[4]Gao W, Guan J, Gao J, Zhou D. Multi-ganglion ANN based feature learning with application to P300-BCI signal classification. Biomed Signal Process Control 2015;18:127–37.

[5]Guger C, Daban S, Sellers E, Holzner C, Krausz G, Carabalona R, et al. How many people are able to control a P300-based

brain–computer interface (BCI)? Neurosci Lett 2009;462 (1):94–8.

[6]Judith AM, Priya SB. Multiset task related component analysis (M-TRCA) for SSVEP frequency recognition in BCI. J Ambient Intell Hum Comput 2021;12(5):5117–26.

[7]Pillette L, Roc A, N’kaoua B, Lotte F. Experimenters’ Influence on Mental-Imagery based Brain-Computer Interface User Training. Int J Hum Comput Stud 2021;149 102603.

[8]Vasilyev A, Liburkina S, Yakovlev L, Perepelkina O, Kaplan A.

Assessing motor imagery in brain-computer interface training: Psychological and neurophysiological correlates.

Neuropsychologia 2017;97:56–65.

[9] Mora N, De Munari I, Ciampolini P. A Multi-modal BCI System for Active and Assisted Living; 2016. pp. 345–355.

[10]Bhattacharyya S, Mukul MK. Reactive frequency band based movement imagery classification. J Ambient Intell Hum Comput 2018:1–14.

[11]Zarei R, He J, Siuly S, Zhang Y. A PCA aided cross-covariance scheme for discriminative feature extraction from EEG signals. Comput Methods Programs Biomed 2017;146:47–57.

[12]Lee C-S, Wang M-H, Kuan W-K, Huang S-H, Tsai Y-L, Ciou Z- H, et al. BCI-based hit-loop agent for human and AI robot co- learning with AIoT application. J Ambient Intell Hum Comput 2021:1–25.

[13]Neuper CU, Pfurtscheller G. Event-related dynamics of cortical rhythms: Frequency-specific features and functional correlates. Int J Psychophysiol 2001;43(1):41–58.

[14]Foong R, Ang KK, Quek C, Guan C, Phua KS, Kuah CWK, et al.

Assessment of the efficacy of EEG-based MI-BCI with visual feedback and EEG correlates of mental fatigue for upper-limb stroke rehabilitation. IEEE Trans Biomed Eng 2019;67 (3):786–95.

[15]Valente G, Kaas AL, Formisano E, Goebel R. Optimizing fMRI experimental design for MVPA-based BCI control: Combining the strengths of block and event-related designs.

Neuroimage 2019;186:369–81.

[16]Li C, Su M, Xu J, Jin H, Sun L. A between-subject fNIRS-BCI study on detecting self-regulated intention during walking.

IEEE Trans Neural Syst Rehabil Eng 2020;28(2):531–40.

[17]Zhang S, Zheng Y, Wang D, Wang L, Ma J, Zhang J, et al.

Application of a common spatial pattern-based algorithm for an fNIRS-based motor imagery brain-computer interface.

Neurosci Lett 2017;655:35–40.

[18]Li R, Potter T, Huang W, Zhang Y. Enhancing performance of a hybrid EEG-fNIRS system using channel selection and early temporal features. Front Hum Neurosci 2017;11:462.

[19]Shin J, Kwon J, Im C-H. A ternary hybrid EEG-NIRS brain- computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state. Front Neuroinf 2018;12:5.

[20]Chaudhary U, Xia B, Silvoni S, Cohen LG, Birbaumer N. Brain–

computer interface–based communication in the completely locked-in state. PLoS Biol 2017;15(1) e1002593.

[21]Caban˜ero L, Herva´s R, Bravo J, Rodrı´guez-Benitez L, Nugent C.

eeglib: Computational analysis of cognitive performance during the use of video games. J Ambient Intell Hum Comput 2019:1–12.

[22] Zhu G, Li R, Zhang T, Lou D, Wang R, Zhang Y. A simplified hybrid EEG-fNIRS Brain-Computer Interface for motor task classification. 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER); 2017. pp. 134–137.

[23]Ge S, Yang Q, Wang R, Lin P, Gao J, Leng Y, et al. A brain- computer interface based on a few-channel EEG-fNIRS bimodal system. IEEE Access 2017;5:208–18.

[24]Saadati M, Nelson J, Ayaz H. Convolutional neural network for hybrid fNIRS-EEG mental workload classification.

International Conference on Applied Human Factors and Ergonomics 2019:221–32.

(14)

[25]Khan MU, Hasan MA. Hybrid EEG-fNIRS BCI Fusion Using Multi-Resolution Singular Value Decomposition (MSVD).

Front Hum Neurosci 2020;14, 599802.

[26]Han C-H, Mu¨ller K-R, Hwang H-J. Enhanced Performance of a Brain Switch by Simultaneous Use of EEG and NIRS Data for Asynchronous Brain-Computer Interface. IEEE Trans Neural Syst Rehabil Eng 2020;28(10):2102–12.

[27]Shin J, Kim D-W, Mu¨ller K-R, Hwang H-J. Improvement of information transfer rates using a hybrid EEG-NIRS brain- computer interface with a short trial length: Offline and pseudo-online analyses. Sensors 2018;18(6):1827.

[28]Ghaffar MSBA, Khan US, Iqbal J, Rashid N, Hamza A, Qureshi WS, et al. Improving classification performance of four class FNIRS-BCI using Mel Frequency Cepstral Coefficients (MFCC).

Infrared Phys Technol 2021;112 103589.

[29]Ho TKK, Gwak J, Park CM, Khare A, Song J-I. Deep leaning- based approach for mental workload discrimination from multi-channel fNIRS. In: Recent trends in communication, computing, and electronics. Springer; 2019. p. 431–40.

[30]Shin J, Mu¨ller K-R, Schmitz CH, Kim D-W, Hwang H-J.

Evaluation of a compact hybrid brain-computer interface system. BioMed Res Int 2017.

[31] Dehais F, Dupres A, Di Flumeri G, Verdiere K, Borghini G, Babiloni F, Roy R. Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI. 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2018. pp. 544–549.

[32]Sharma M, Acharya UR. Automated detection of

schizophrenia using optimal wavelet-based $$ l_1 $$ l 1 norm features extracted from single-channel EEG. Cogn Neurodyn 2021:1–14.

[33] Bairy GM, Lih OS, Hagiwara Y, Puthankattil SD, Faust O, Niranjan U, et al. Automated diagnosis of depression electroencephalograph signals using linear prediction coding and higher order spectra features. J Med Imag Health Informat 2017;7(8):1857–62.

[34] Zbilut JP, Webber Jr CL. Embeddings and delays as derived from quantification of recurrence plots. Phys Lett A 1992;171 (3–4):199–203.

[35] Hammad M, Rajesh KN, Abdelatey A, Abdar M, Zomorodi- Moghadam M, San Tan R, et al. Automated detection of Shockable ECG signals: A review. Information Sci 2021;571:580–604.

[36] Acharya UR, Hagiwara Y, Deshpande SN, Suren S, Koh JEW, Oh SL, et al. Characterization of focal EEG signals: A review.

Futur Gener Comput Syst 2019;91:290–9.

[37] Ksiaoˆ_zek W, Hammad M, Pławiak P, Acharya UR, Tadeusiewicz R. Development of novel ensemble model using stacking learning and evolutionary computation techniques for automated hepatocellular carcinoma detection. Biocybern Biomed Eng 2020;40(4):1512–24.

[38] Zeng X, Martinez TR. Distribution-balanced stratified cross- validation for accuracy estimation. J Exp Theor Artif Intell 2000;12(1):1–12.

[39] Khan B, Wildey C, Francis R, Tian F, Delgado MR, Liu H, et al.

Improving optical contact for functional near-infrared brain spectroscopy and imaging with brush optodes. Biomed Opt Express 2012;3(5):878–98.

[40] Mian Qaisar S. Signal-piloted processing and machine learning based efficient power quality disturbances recognition. PLoS One 2021;16(5):e0252104.