Table S1: Names and domains of all 24 neuropsychological tests in the ROS-MAP data.
Test name Cognitive test description Cognitive Domain
cts_wli word list episodic memory
cts_wlii word list recall episodic memory
cts_wliii word list recognition episodic memory
cts_ebmt East Boston immediate recall episodic memory
cts_ebdr East Boston immediate recall episodic memory
cts_story Logical memory I (immediate) episodic memory
cts_delay Logical memory II (delayed) episodic memory
cts_bname Boston naming (15 items) semantic memory
cts_catflu category fluency semantic memory
cts_animals category fluency - Animals semantic memory
cts_fruits category fluency - Fruits semantic memory
cts_read_nart reading test (10 items) semantic memory
cts_df digits forward working memory
cts_db digits backward working memory
cts_doperf digit ordering working memory
cts_lopair line orientation perceptual orientation
cts_pmat progressive matrices (16 items) perceptual orientation cts_pmsub progressive matrices (subset - 9 items) perceptual orientation
cts_sdmt symbol digits modality oral perceptual speed
cts_nccrtd number comparison perceptual speed
cts_stroop_cname stroop color naming perceptual speed
cts_stroop_wread stroop word reading perceptual speed
cts_idea Complex Ideational Matrix cts_mmse30 Mini Mental State Exam, 30 items
Test Name Description
UDSBENTC Total score for copy of Benson figure UDSBENTD Total score for 10- to 15-minute delayed
ANIMALS Animals — Total number of animals named in 60 seconds VEG Vegetable— Total number of vegetables named in 60 seconds TRAILA Trail Making Test Part A — Total number of seconds to complete TRAILARR Part A — Number of commission errors
TRAILALI Part A — Number of correct lines
TRAILB Trail Making Test Part B — Total number of seconds to complete TRAILBRR Part B — Number of commission errors
TRAILBLI Part B — Number of correct lines
UDSVERFC Number of correct F-words generated in 1 minute UDSVERFN Number of F-words repeated in 1 minute
UDSVERNF Number of non-F-words and rule violation errors in 1 minute UDSVERLC Number of correct L-words generated in 1 minute
UDSVERLR Number of L-words repeated in 1 minute
UDSVERLN Number of non-L-words and rule violation errors in 1 minute UDSVERTN Total number of correct F-words and L-words
UDSVERTE Total number of F-word and L-word repetition errors UDSVERTI Total number of non-F/L-words and rule violation errors MOCATOTS MoCA Total Raw Score — uncorrected
CRAFTVRS Craft Story 21 Recall (Immediate) — Total story units recalled, verbatim scoring CRAFTURS Craft Story 21 Recall (Immediate) — Total story units recalled, paraphrase scoring DIGFORCT Number Span Test: Forward — Number of correct trials
DIGFORSL Number Span Test: Forward — Longest span forward DIGBACCT Number Span Test: Backward — Number of correct trials DIGBACLS Number Span Test: Forward — Longest span backward
CRAFTDVR Craft Story 21 Recall (Delayed) — Total story units recalled, verbatim scoring CRAFTDRE Craft Story 21 Recall (Delayed) — Total story units recalled, paraphrase scoring CRAFTDTI Craft Story 21 Recall (Delayed) — Delay time
MINTTOTS Multilingual Naming Test (MINT) — Total score
Table S2: Names and description of all 30 tests used in the NACC study data.
Table S3: Number and percentage distribution of HV, MCI and AD across different subsets for NACC data set.
Distribution Values Percentage
Data set HV MCI AD HV MCI AD
1 54 18 11 65.06024 21.68675 13.25301
2 54 18 11 65.06024 21.68675 13.25301
3 60 12 11 72.28916 14.45783 13.25301
4 52 15 16 62.6506 18.07229 19.27711
5 60 12 11 72.28916 14.45783 13.25301
6 53 20 10 63.85542 24.09639 12.04819
7 58 16 9 69.87952 19.27711 10.84337
8 58 17 8 69.87952 20.48193 9.638554
Total 449 128 87 67.62048 19.27711 13.10241
Table S4: It presents the accuracy, sensitivity and specificity of prediction models for different data sets compared over 3 ML approaches namely, RF with ntree =800 and mtry =18, linear and radial SVM for NACC data.
Random Forests
Dataset Accuracy HV MCI AD
Sensitivity Specificity Sensitivity Specificity Sensitivity Specificity
1 0.7229 0.8889 0.5862 0.16667 0.89231 0.8182 0.9444
2 0.7349 0.9444 0.5172 0.11111 0.92308 0.72727 0.95833
3 0.8072 0.9667 0.5652 0.08333 0.92958 0.72727 0.98611
4 0.7711 0.9615 0.6774 0.26667 0.89706 0.625 0.9701
5 0.8434 0.9167 0.7391 0.5 0.92958 0.8182 0.9722
6 0.7108 0.9057 0.4667 0.25 0.88889 0.6 0.9863
7 0.7349 0.8621 0.72 0.3125 0.83582 0.66667 0.94595
8 0.7711 0.931 0.72 0.17647 0.92424 0.875 0.90667
Linear Support Vector Machine
1 0.7349 0.9444 0.5517 0.05556 0.92308 0.8182 0.9444
2 0.7831 0.9815 0.6207 0.22222 0.93846 0.72727 0.95833
3 0.7711 0.9333 0.5217 0.08333 0.90141 0.63636 0.98611
4 0.759 0.9808 0.5806 0.2 0.92647 0.5625 0.9701
5 0.8193 0.9 0.7826 0.41667 0.90141 0.8182 0.9583
6 0.6988 0.9057 0.4667 0.2 0.90476 0.6 0.9589
7 0.7711 0.8966 0.64 0.375 0.8806 0.66667 0.97297
8 0.747 0.9138 0.68 0.23529 0.87879 0.625 0.93333
Radial Support Vector Machines
1 0.7711 0.9444 0.5862 0.11111 0.95385 1 0.9444
2 0.747 0.963 0.5172 0.11111 0.96923 0.72727 0.93056
3 0.8193 0.9833 0.5652 0.08333 0.94366 0.72727 0.98611
4 0.759 0.9423 0.5484 0.06667 0.95588 0.8125 0.9552
5 0.8554 0.95 0.6087 0.41667 0.98592 0.8182 0.9722
6 0.6988 0.8679 0.4667 0.2 0.92063 0.8 0.94521
7 0.7711 0.931 0.6 0.1875 0.92537 0.77778 0.94595
8 0.7952 0.9655 0.68 0.17647 0.95455 0.875 0.92
Table S5: Contingency table for 4th data-set of NACC studies comparing the results from 3 different ML approaches, namely, Random Forests (800=ntree and 18=mtry), Linear and radial Support Vector Machine.
NACC
RF (800,18) SVM (Linear) SVM (Radial)
Acual HV
Actual MCI
Actual AD
Actual HV
Actual MCI
Actual AD
Actual HV
Actual
MCI Actual AD Pred
HV 50 9 1 51 10 3 49 12 2
Pred
MCI 2 4 5 1 3 4 2 1 1
Pred
AD 0 2 10 0 2 9 1 2 13
Total 52 15 16 52 15 16 52 15 16
Table S6: Accuracy, sensitivity and specificity for prediction of HV, MCI, AD classes in different data sets for ROSMAP study, comparing few different combinations of tests (6,8 and 9 tests) with the results of whole battery of tests (24) using Random Forests [ntree = 800 and mtry = (no. of tests/2) meaning 12,3,4 and 4 respectively].
All 24 tests
Dataset Accuracy HV MCI AD
Sensitivity Specificity Sensitivity Specificity Sensitivity Specificity
1 0.9407 0.9786 0.9394 0.9259 0.9447 0.41667 1
2 0.9289 0.9779 0.8611 0.8095 0.9684 0.77778 0.9918
3 0.9368 0.9784 0.8971 0.8413 0.9684 0.6 0.9879
4 0.9249 0.9944 0.8493 0.8361 0.9583 0.33333 1
5 0.9644 1 0.918 0.8868 0.985 0.625 0.99592
6 0.9368 0.9944 0.8243 0.8116 0.9837 0.6 1
7 0.9526 0.9891 0.9143 0.8923 0.9734 0.4 0.99597
6 tests (wlii, mmse30, delay, wliii, pmsub, lopair)
1 0.9486 0.9786 0.9545 0.9444 0.9497 0.5 1
2 0.9368 0.9779 0.9028 0.8571 0.9632 0.66667 0.9918
3 0.9447 0.9892 0.9118 0.8413 0.9789 0.6 0.98387
4 0.9368 0.9944 0.8767 0.8689 0.9635 0.41667 1
5 0.9644 0.9948 0.9344 0.9057 0.98 0.625 0.99592
6 0.9526 0.9944 0.9054 0.8986 0.9728 0.2 1
7 0.9723 0.9945 0.9571 0.9538 0.9787 0.4 1
8 tests (wlii, mmse30, delay, wliii, pmsub, lopair, catflu, story)
1 0.9565 0.9786 0.9697 0.963 0.9548 0.58333 1
2 0.9447 0.9834 0.9167 0.8571 0.9737 0.77778 0.9877
3 0.9368 0.9784 0.9118 0.8413 0.9684 0.6 0.98387
4 0.9368 0.9944 0.8767 0.8689 0.9635 0.41667 1
5 0.9684 1 0.9344 0.9057 0.985 0.625 0.99592
6 0.9407 0.9944 0.8649 0.8551 0.9728 0.2 1
7 0.9684 0.9891 0.9571 0.9538 0.9734 0.4 1
9 tests (wlii, mmse30, delay, wliii, pmsub, lopair, catflu, story, sdmt)
1 0.9526 0.9786 0.9697 0.963 0.9497 0.5 1
2 0.9328 0.9834 0.875 0.8095 0.9737 0.77778 0.9877
3 0.9368 0.9838 0.8971 0.8254 0.9737 0.6 0.98387
4 0.9289 0.9944 0.863 0.8361 0.9583 0.41667 1
5 0.9644 0.9948 0.9344 0.9057 0.98 0.625 0.99592
6 0.9368 0.9944 0.8514 0.8406 0.9728 0.2 1
7 0.9605 0.9891 0.9429 0.9231 0.9734 0.4 0.995968
Table S7: Accuracy, sensitivity and specificity for prediction of HV, MCI, AD classes in different data sets, comparing combinations of 7 tests with the results of whole battery of tests for NACC data using Random Forests [ntree = 800 and mtry = (no. of tests/2) meaning 15 and 4 respectively].
All 30 tests
Dataset Accuracy HV MCI AD
Sensitivity Specificity Sensitivity Specificity Sensitivity Specificity
1 0.7349 0.8889 0.6207 0.16667 0.89231 0.9091 0.9444
2 0.7349 0.9444 0.5172 0.11111 0.92308 0.72727 0.95833
3 0.8072 0.9667 0.5652 0.08333 0.92958 0.72727 0.98611
4 0.759 0.9615 0.5806 0.13333 0.92647 0.6875 0.9701
5 0.8434 0.9333 0.6957 0.41667 0.94366 0.8182 0.9722
6 0.7108 0.9057 0.4667 0.25 0.88889 0.6 0.9863
7 0.747 0.8793 0.7200 0.3125 0.85075 0.66667 0.94595
8 0.7952 0.931 0.7600 0.29412 0.92424 0.875 0.92
7 tests (CRAFTDRE, CRAFTDVR, MOCA, UDSBENTD, VEG, TRAILB, CRAFTVRS)
1 0.7349 0.8889 0.6207 0.22222 0.87692 0.8182 0.9583
2 0.759 0.9815 0.5172 0.11111 0.95385 0.72727 0.95833
3 0.8313 0.9667 0.5652 0.08333 0.95775 0.9091 0.9861
4 0.8072 0.9615 0.7419 0.3333 0.91176 0.75 0.9701
5 0.8193 0.8833 0.7391 0.5 0.90141 0.8182 0.9722
6 0.7108 0.9434 0.4667 0.2 0.90476 0.5 0.9726
7 0.759 0.8966 0.72 0.3125 0.86567 0.66667 0.94595
8 0.8072 0.931 0.8 0.35294 0.92424 0.875 0.92
Supplementary Figure S1: Tuning of Random Forests: The figure describes how multiclass ROC values were used to find an optimum ntree, mtry value for Random Forests for one of the representative data subsets for ROSMAP study. The blue line represents mtry as 1, orange as 3, silver as 6, yellow as 9, light blue as 12 and green as 18.
100 200 300 400 500 600 700 800 900 1000 2000 5000 0.84
0.86 0.88 0.9 0.92 0.94 0.96
1 3
ntree
6 9 12 18R O C
Supplementary Figure S2: Finding important tests by the prediction accuracy pattern. The number of tests were arranged according to their discriminative power in Fischer’s score and then added one by one for one representative testing data subset of ROSMAP study; subsequently accuracy was predicted using Random Forests with different ntree (100,200-1200,2000,3000-5000) and mtry. The pattern shows the first 3 tests (wlii, mmse30, delay), 11
th(wliii), 20
th(pmsub) and 21
st(lopair) tests to be important in all the 3 subfigures.
S2 (first subfigure) has mtry as (no. of tests/2), S2 (second subfigure) has mtry as (no. of tests/3) and S2 (third subfigure) has mtry as (no. of tests/1.414) for each iteration.
0 2 4 6 8 10 12 14 16 18 20 22 24 26
0.72 0.75 0.78 0.81 0.84 0.87 0.9 0.93
No. of tests
A cc u ra cy
0 2 4 6 8 10 12 14 16 18 20 22 24 26
0.73 0.75 0.77 0.79 0.81 0.83 0.85 0.87 0.89 0.91 0.93 0.95
No. of tests
A cc u ra cy
0 2 4 6 8 10 12 14 16 18 20 22 24 26
0.72 0.74 0.76 0.78 0.8 0.82 0.84 0.86 0.88 0.9 0.92 0.94 0.96
No. of tests
A cc u ra cy
Supplementary Figure S3: Increment in accuracy for 6 important tests in one of the representative data subsets of ROSMAP data. The 6 tests were arranged in the following order: wlii, mmse30, delay, wliii, pmsub and lopair and we observed for the change in prediction accuracy if any of the following test was missing from the subset. It was done using Random Forests with different ntree (100,200-1200,2000,3000-5000) and mtry as (no. of tests/2). S3 (first subfigure) represents when pmsub was missing, S3 (second subfigure) represents when lopair was missing and S3 (third subfigure) represents when all 6 tests were present. It shows the dependence of all 6 tests to get the maximum prediction ability.
1 2 3 4 5
0.65 0.7 0.75 0.8 0.85 0.9 0.95
No. of Tests
A cc ur ac y
1 2 3 4 5
0.7 0.75 0.8 0.85 0.9 0.95
No. of Tests
A cc u ra cy
1 2 3 4 5 6
0.7 0.75 0.8 0.85 0.9 0.95 1
No. of Tests
A cc ur ac y
Supplementary Figure S4: Ranking of 30 neuropsychological tests from NACC study.
Supplementary Figure S4A: Fischer’s score for 3 classes classification (HV, MCI, AD).
CRAFTDRE MOCATOTS
CRAFTDVR UDSBEN
TD
CRAFTURS CRAFTVRS
VEG TRAILB
AN IMALS
DIGBACCT TRAILBLI
DIGBACLS TRAILA
MINTTOTS DIGFORCT
UDSVER TN
UDSVER FC
UDSVER LC DIGFORSL
TRAILBR R
UDSBEN TC CRAFTDTI
TRAILALI UDSVER
TE
UDSVER FN
UDSVER LR UDSVER
TI
UDSVER LN
UDSVER NF TRAILAR
R 0
0.2 0.4 0.6 0.8 1 1.2 1.4 1.61.5
1.31.3
1.21.21.1
0.80.8 0.6
0.30.30.30.30.3
0.20.20.20.20.20.2
0.10.10.00.00.00.00.00.00.00.0
Test Name
F is ch er 's S co re
Supplementary Figure S4B: Average decrease in accuracy of prediction when a single test was removed iteratively.
UDSVER FN
TRAILB UDSBEN
TD
CRAFTDRE UDSVER
TE TRAILALI
DIGFO RCT CRAFTVR
S
UDSVER LCVEG
DIGBACCT TRAILBLI
TRAILA UDSVER
TI
TR AILAR
R
TRAILBR R
CRAFTURS DIGBACLS
UDSVER LR
UDSVER FC AN
IMALS DIGFO
RSL UDSBEN
TC
UDSVER NF CRAFTDTI
MINTTOTS UDSVER
LN
UDSVER TN
CRAFTDVR MOCATOTS -0.02
-0.01 -0.01 -0.01 -0.01 -0.01 0 0 0 0 0
