An Analysis of the World Health Organization Disability Assessment

(1)

https://doi.org/10.1177/1073191119834653 Assessment

2020, Vol. 27(8) 1731 –1747

sagepub.com/journals-permissions DOI: 10.1177/1073191119834653 journals.sagepub.com/home/asm

Article

The International Classification of Functioning, Disability, and Health (ICF; World Health Organization [WHO], 2001) describes the terms functioning and disability as the out- come of a complex, multidimensional interaction between a person’s health condition(s) and context (environmental and personal factors). Disability is determined by the interaction between bodily functions, structural impairments, and unhelpful contexts that limit an individual’s activities and hinder participation in previous activities of daily living (Federici, Bracalenti, Meloni, & Luciano, 2017). Hence, through the lens of the ICF, positive and neutral interactions between a person’s health and context is functioning, whereas negative interaction is a disability.

The ICF’s classification (WHO, 2001) was innovative when first introduced in 2001, offering a biopsychosocial approach to conceptualizing disability that encompassed both medical and health components while also integrating social aspects (Stucki, 2016). The ICF’s classification system differed from the previous classification system, the International Classification of Impairments, Disabilities, and Handicaps, which saw disability strictly through a medical point of view (WHO, 1980). With the paradigm shift to the ICF’s model, the goal became less focused on diagnosis and disease classification; but rather, on a better understanding of the impact of the disease (e.g., medical

conditions, health) and the influence of disease on a person’s functioning, ability, and quality of life.

Despite the ICF’s (WHO, 2001) more expansive approach to conceptualizing individuals’ functionality and potential disability, the framework lacked operational definitions of specific concepts associated with disability (Altman, 2001), limiting empirical measurements of disability. Therefore, in response, the WHO published its final version of the World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0; Üstün, Kostanjsek, Chatterji, & Rehm, 2010b) to capture the multifaceted nature of disablement under six domains as described below.

The WHODAS 2.0, previously known as the WHODAS-II (Rehm et al., 1999), was initially developed as a 96-item assessment with broad applicability across cul- tures and diverse populations (Üstün et al., 2010a; Üstün et al., 2010b). However, after extensive cross-cultural field 834653ASMXXX10.1177/1073191119834653AssessmentTabet et al.

research-article2019

1University of Central Florida, Orlando, FL, USA

2Taylor’s University, Subang Jaya, Malaysia Corresponding Author:

Saundra M. Tabet, Department of Counselor Education and School Psychology, College of Community Innovation and Education, University of Central Florida, P.O. Box 161250, Orlando, FL 32816-1250, USA.

Email: [email protected]

An Analysis of the World Health Organization Disability Assessment

Schedule 2.0 Measurement Model Using Partial Least Squares–Structural

Equation Modeling

Saundra M. Tabet

¹

, Glenn W. Lambie

¹

, Shiva Jahani

¹

, and S. Mostafa Rasoolimanesh

²

Abstract

The researchers examined the factor structure and model specifications of the World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) with confirmatory tetrad analysis (CTA) using partial least squares–structural equation modeling (PLS-SEM) with a sample of adult clients (N = 298) receiving individual therapy at a university-based counseling research center. The CTA and PLS-SEM results identified the formative nature of the WHODAS 2.0 subscale scores, supporting an alternative measurement model of the WHODAS 2.0 scores as a second-order formative–formative model.

Keywords

confirmatory tetrad analysis, disability assessment, factor structure, model specification, partial least squares, structural equation modeling, World Health Organization Disability Assessment Schedule 2.0

(2)

studies in 19 different countries, a substantial item reduc- tion yielded a revised 36-item version (Üstün et al., 2010b).

The final version of the WHODAS 2.0 is composed of 36 Likert-type scale items (5-point) with six unique domains:

(a) cognition, (b) mobility, (c) self-care, (d) getting along, (e) life activities, and (f) participation. Üstün et al. (2010a) designed the six subscales to calculate individuals’ level of functioning and provide indicators for potential disability, including (a) understanding and communicating (cognition, UC; six items); (b) moving and getting around (mobility, GA; five items); (c) hygiene, dressing, eating, and being alone (self-care, SC; four items); (d) interacting with other people (getting along with people, GAP; five items); (e) domestic responsibilities, leisure work, and school (life activities, LA; eight items); and (f) joining in community or social activities (participation, PAR; eight items). Since the WHODAS 2.0 development, it has become one of the most common measures in disability research, being translated into 47 languages and used in 27 diverse areas of study. The majority of research utilizing the WHODAS 2.0 is focused within the field of psychiatry (Federici et al., 2017).

Moreover, the pervasiveness of the WHODAS 2.0 has led to the measure’s inclusion in the Diagnostic Statistical Manual of Mental Disorders–Fifth edition (American Psychiatric Association, 2013) as the best measure of disability in routine clinical practice.

Research on the Measurement Model of WHODAS 2.0 Scores

The WHODAS 2.0 scores have demonstrated acceptable psychometric properties with the total internal consistency of α = .96 and overall good test–retest reliability of r = .98 (Üstün et al., 2010b). Likewise, initial validation studies of the WHODAS 2.0 by the WHO supported a robust second- order six-factor structure. The first WHODAS 2.0 level consists of the overall measure of disability and functioning, and the second level consists of the six WHODAS domains (i.e., UC, GA, SC, GAP, LA, and PAR; Üstün et al., 2010a; Üstün et al., 2010b). Despite findings of a statistically acceptable measurement model for the WHODAS 2.0 scores, the WHO failed to provide supporting measurement indices to offer evidence on all fit metrics (e.g., Tucker–Lewis index [TLI], comparative fit index [CFI], standardized root mean square residual [SRMR], root mean square error of approximation [RMSEA], Akaike information criterion [AIC], etc.). Nonetheless, several exploratory factor analyses (EFA) and confirmatory factor analyses (CFA) were completed with WHODAS 2.0 data (Buist- Bouwman et al., 2008; Sousa et al., 2010) among diverse samples (e.g., breast cancer, mental disorder, multiskelotal disorders) and languages: Chinese (Cheung et al., 2015;

Chiu et al., 2014; Zhao et al., 2013); Italian (Federici, Meloni, Mancini, Lauriola, & Olivetti Belardinelli, 2009;

Magistrale et al., 2015); Spanish (Galindo-Garre et al., 2015; Garin et al., 2010; Guilera et al., 2012; Guilera et al., 2015), German (Pösl, Cieza, & Stucki, 2007); and Turkish (Küçükdeveci et al., 2013). Table 1 presents the available fit metrics from studies using CFA to provide evidence of validity for the WHODAS factor structure.

Overall, factor analytic studies lend significant support to the WHODAS 2.0 data as a first-order six-factor model (Galindo-Garre et al., 2015; Guilera et al., 2012; Guilera et al., 2015; Küçükdeveci et al., 2013; Zhao et al., 2013).

One of the first to investigate the factor structure of the revised WHODAS 2.0, Zhao et al. (2013) reported, “as for the factor structure of the Chinese version of the WHODAS 2.0, the study outcomes are not consistent with the conclusions of the original developers” (p. 903). Their results identifed the theoretical second-order six-factor model had less than acceptable fit indices; χ²/df = 2.666; RMSEA = .091;

TLI = .790; and CFI = .810 (Zhao et al., 2013). Guilera et al. (2012) examined the utility of the WHODAS 2.0 on a sample (N = 352) of clinical patients with a schizophrenia spectrum disorder. The three-factor structures of the WHODAS 2.0 data tested included (a) six first-order factors; (b) a second-order single factor; and (c) two dimen- sional second-orders, were assessed with the six first-order factorial model representing the best fit: χ² = 708.12;

χ²/df = 1.577, p < .05; RMSEA = .041; nonnormed fit index (NNFI) = .99; CFI = .99; and AIC = 866.12.

Similarly, Guilera et al. (2015) examined the WHODAS 2.0 scores with a sample (N = 291) composed of individuals with bipolar disorder based on the DSM-IV-TR criteria.

Likewise, the six first-order WHODAS 2.0 factorial model indicated the best overall fit indices: χ² = 617.89;

χ²/df = 1.376, p < .05; RMSEA = .036; NNFI = .98;

CFI = .99; and AIC = 775.89. The same year, a study using partial credit and Mokken scale models evaluated the WHODAS 2.0 on a participant sample (N = 352) of patients diagnosed with a schizophrenia spectrum disorder (Galindo- Garre et al., 2015). Findings from the Galindo-Garre et al.

(2015) exploratory factor study supported the unidimen- sionality of the WHODAS 2.0 with six factors loading of

≤ 0.35 and the percentages of explained variance between 47% (GA), 51% (SC), 55% (UC), 56% (PAR), 61% (GAP), and 63% (LA). Although examined in a population not tar- geting mental disorders, Küçükdeveci et al. (2013) also found empirical evidence for the first-order six-factor measurement model of the WHODAS 2.0 on a sample (N = 188) of poststroke patients using a Rasch analysis; χ²(64, N = 188)

= 179.7, p < .001; person separation index = .95; confidence interval (CI) = 27.66 [24.5, 30.8].

The findings supporting the first-order six-factor structure of the WHODAS 2.0 data contrast with the developers’

second-order six-factor model. Üstün et al. (2010b) theo- rized the WHODAS 2.0 as a hierarchal measure of six life domains that contribute to the overall measure of general

(3)

Tabet et al. 1733

Table 1. WHODAS 2.0 Factor Analysis Studies.

Authors Country Language Sample Model Fit indices

Cheung et al.

(2015) China Chinese N = 1,020, Adults with disability or chronic illnesses

Second-order, six-

factor model χ²/df = 3.05, RMSEA = .053 (p = .051), CFI = .912, SRMR = .076

Chiu et al. (2014) China Chinese N = 307, Adults living in elderly or disability institutions

Second-order, six-

factor model GFI = .76, CFI = .81, RMSEA = .091 Guilera et al.

(2012) Spain Spanish N = 352, Adults with a schizophrenia spectrum disorder

Six first-order factors χ² = 708.12, χ²/df = 1.577, RMSEA

= .041, NNFI = .99, CFI = .99, AIC = 866.12

Six factors loading on a higher order factor

χ² = 763.56, χ²/df = 1.667, RMSEA

= .044, NNFI = .99, CFI = .99, AIC = 903.58

Two higher order factors with three domains each

χ² = 761.25, χ²/df = 1.666, RMSEA

= .044, NNFI = .99, CFI = .99, AIC = 903.25

Guilera et al.

(2015) Spain Spanish N = 291, Adults with

bipolar disorder Six first-order factors χ² = 617.89, χ²/df = 1.376, RMSEA

= .036, NNFI = .98, CFI = .99, AIC = 775.89

Six factors loading on a higher order factor

χ² = 740.73.12, χ²/df = 1.1.617, RMSEA = .046, NNFI = .98, CFI = .98, AIC = 880.73 Two higher order

factors with three domains each

χ² = 740.51, χ²/df = 1.620, RMSEA

= .046, NNFI = .98, CFI = .98, AIC = 882.51

Küçükdeveci et al.

(2013) Turkey Turkish N = 188, Adults

poststroke First-order six factor χ²(64, N = 188) = 179.7, p < .001; PSI = .95;

CI = 27.66 [24.5, 30.8].

Zhao et al. (2013) China Chinese N = 402, Adults with breast cancer receiving chemotherapy

Second-order, six- factor reflective model

χ²/df = 2.666, RMSEA = 0.091, CFI = 0.810, TLI = 0.790

Note. WHODAS = World Health Organization Disability Assessment Schedule 2.0; df = degrees of freedom; RMSEA = root mean square error of approximation; CFI = comparative fit index; SRMR = standardized root mean square residual; GFI = goodness-of-fit index; NNFI = nonnormed fit index; AIC = Akaike information criterion; PSI = person separation index; CI = confidence interval; TLI = Tucker–Lewis index.

disability. As noted, Üstün et al. (2010b) did not report the fit indices of their intended second-order measurement model. However, emerging research has offered empirical support to the hierarchical second-order model as well. A study that examined a sample of persons with disabilities in Hong Kong (N = 1,020) indicated support for the second-order model of the WHODAS 2.0 data: χ²/df = 3.05, p = .051; RMSEA = .053; CFI = .912; and SRMR = .076 (Cheung et al., 2015). Furthermore, a similar study that assessed the Chinese version of the WHODAS 2.0 data in Taiwan with a sample of individuals residing in a disability community (N = 307) yielded satisfactory results: RMSEA

= .091; CFI = .81; and GRI= .76 (Chiu et al., 2014).

As noted, Guilera et al.’s (2012; Guilera et al., 2015) findings supported the first-order six-factor WHODAS 2.0 model as the most robust model fit, CFAs also lent marginal evidence to the measure as a second-order six-factor model.

The CFA results from Guilera et al. (2012) offered slightly stronger fit indices: χ² = 763.58; χ²/df = 1.667, p < .05;

RMSEA = .044; NNFI = .99; and AIC = 903.58, as opposed to the CFA conducted on the same measurement model a few years later: χ² = 740.73; χ²/df = 1.617, p < .05;

RMSEA = .046; NNFI = .98; and AIC = 880.73 (Guilera et al., 2015). Despite the goodness-of-fit statistics obtained by both the first-order six-factor WHODAS 2.0 model and second-order six-factor model the, “ . . . [first-order six- factor] model represents the reality of the data” (Guilera et al., 2015, p. 356). The discrepancy observed between a first-order six-factor WHODAS 2.0 model and second- order six-factor model could be attributed to the moderate correlation between the six domain structure of the measure (Federici et al., 2017). Correlation among the six WHODAS 2.0 domains identifies similarities in their conceptual framework, and in essence, what the intended construct is

(4)

measuring. The correlation between WHODAS 2.0 items was further echoed by Küçükdeveci et al. (2013) who raised attention to the high residual correlations (r = 0.8+) between the items on the six domains, such as the LA construct warranting questions into replication and redundancy of the WHODAS 2.0. Moreover, Federici et al. (2017) rein- forced this earlier finding, “ . . . the number of cross loading items could be due to the partial conceptual overlap between some aspects of the different domains of WHODAS 2.0” (p.

2353). Thus, further examination of the WHODAS 2.0 conceptual framework and measurement model is warranted.

The limited empirical support for the factor structure of the WHODAS 2.0 provides evidence that the measurement model of the WHODAS 2.0 is potentially misspecified, and thus poses threats to the statistical validity of findings (MacKenzie, Podsakoff, & Jarvis, 2005). Despite empirical considerations (e.g., poor fit metrics), before altering the mode of measurement (i.e., reflective to formative), addi- tional ontological considerations should be considered and were examined for the WHODAS 2.0 scores (Bollen &

Ting, 2000; Gudergan, Ringle, Wende, & Will, 2008; Hair, Hult, Ringle, & Sarstedt, 2017; Jarvis, MacKenzie, &

Podsakoff, 2003).

Accordingly, theoretical and ontological evaluation is a vital component when determining or modifying the nature of a measurement model, and must be considered with empirical evidence (Hair et al., 2017). Historically in the social sciences, measurement theory has been underpinned by classical test theory and factor analyses, asserting that observable indicators are reflective effects of its construct (Howell, Breivik, & Wilcox, 2007). Bollen and Lennox (1991) argued that measurement theory typically neglects the assessment of relationships between its construct and indicators. Constructs are not directly observable; therefore, indicators that are observable are used to examine or assess a construct. Indicators are commonly assumed to be an effect of a construct (i.e., constructs cause the indicators); however, it has been suggested a casually reversed way of conceptualizing constructs may better correspond to the actual relation between the construct and its indicators (Bollen & Lennox, 1991; Howell et al., 2007; Jarvis et al., 2003).

To better understand the theoretical nature of the WHODAS 2.0, we used Jarvis et al.’s (2003) four conceptually grounded “decision rules” to scrutinize the nature of the WHODAS 2.0 subscales and their items as a reflective or formative (composite) constructs. The decision rules included (a) the presumed direction of causality from construct to measure, (b) the interchangeability of indicators, (c) the presumed covariation among indicators, and (d) the nomological net among the indicators (Jarvis et al., 2003).

Likewise, Hair et al. (2017) outlined the following criteria for choosing the measurement model mode: (a) casual priority between the indicator and the construct (construct

to the indicator [reflective] or indicator to the construct [formative]), (b) the construct is a trait explaining the indicators (reflective) or the construct is a combination of indicators (formative), (c) indicators represent consequences (reflective) or indicators represent causes (formative), and (d) items are mutually interchangeable (no [formative] or yes [reflective]). Therefore, using the noted criteria, the theoretical nature of a measurement model can systemati- cally be assessed. With regard to the WHODAS 2.0., its conception was derived from the conceptual framework of the ICF, where individuals’ six domains of functionality directly correspond and measure their overall health and disability (Üstün et al., 2010a). Hence, the conceptual ratio- nale may argue the WHODAS 2.0 theoretically is interchangeable as a formative or reflective model.

While theoretical and ontological conceptualizations are substantiated in model specification, there are also empirical means that can help researchers delineate whether a measurement model is reflective or formative (Garson, 2016; Hair et al., 2017; Hair, Sarstedt, Ringle, & Gudergan, 2018). Specifically, Bollen and Ting (2000) introduced the confirmatory tetrad analysis–structural equation modeling (CTA-SEM), which Gudergan et al. (2008) later adapted to partial least squares–structural equation modeling (PLS- SEM). CTA-PLS is a modern approach that builds off of more traditional CTA-SEM. PLS is a composite-based SEM approach contrary to the covariance-based SEM (CB-SEM) which is a factor-based approach (Rasoolimanesh

& Ali, 2018; Rigdon, Sarstedt, & Ringle, 2017). Traditional SEM using maximum likelihood focuses on minimizing the differences between empirical and model implied covariances which these covariances between indicators are calculated based on common variances of indicators (Rigdon, 2016). However, PLS path modeling does not aim to test a model by evaluating discrepancies between empirical and model implied covariances, eluding assumptions about data distribution and sample size (Hair et al., 2017; Rigdon, 2005). PLS does not refer to common and unique variance of indicators and as a composite-based approach calculates the score of construct by composing all variances of indicators (Rasoolimanesh & Ali, 2018). Therefore, statistically, traditional SEM does not focus on score of construct and only look at the common variance of indicators, and this common variance can lead to infinite values for score of construct, which this phenomenon is called factor indeter- minacy in traditional SEM (Rigdon, 2012, 2016). However, PLS-SEM can determine unique score for the constructs (Rigdon, 2016), so it is more appropriate when we work on scale development and assessment. Moreover, when the framework or scale includes various types of constructs such as reflective or formative (composite), using PLS is recommended (Hair et al., 2017).

Therefore, the purpose of this investigation was to examine the measurement model of WHODAS 2.0 scores with

(5)

Tabet et al. 1735 adult individuals receiving individual outpatient therapy

(N = 298) using a CTA-PLS (Gudergan et al., 2008).

Method Procedure

We obtained an exempt authorization from our university’s institutional review board as part of a larger study. Following institutional review board approval, we collected the data from archival records at a university-based community counseling and research center (UBCCRC). The UBCCRC is an on-site training clinic located in a large, metropolitan city in the Southeastern United States. The UBCCRC provides free individual, couple, and family counseling services to a diverse pool of clients that total over 1,400 per year. The data collected for this study was part of more extensive research exploring assessment measures used in the UBCCRC for information gathering and training pur- poses. As such, participants were not recruited nor con- scribed exclusively for this study, but instead, conveniently sampled from the archived data at the UBCCRC. Participants granted consent at the start of the therapy services at the UBCCRC as part of the overarching research endeavor conducted by the center. Specifically, the data collected for our investigation included adult clients receiving individual counseling services at the UBCCRC who completed a WHODAS 2.0 between January 2014 and December 2014.

Participants

An aggregate of 307 adult clients partook in the study; however, participants with missing demographic information or enrolled in more than one modality of counseling (i.e., individual counseling and family counseling) were removed from the data, resulting in 298 adult client participants.

Table 2 presents participants demographic data, representing a diverse participant pool (i.e., gender, age, race/ethnic- ity, employment status, and current relational status). The participants (N = 298; males, n = 117 [39.3%]; females, n = 181 [60.7%]) represented in the study had a minimum age requirement of 18 years with a mean age of 34.7 years (SD = 12.61; range 18-70). Caucasian/White (n = 123, 41.3%) was the most represented race/ethnic group fol- lowed by Hispanic/Latinx (n = 55, 18.5%), and African American (n = 37, 12.4%). Participants were also asked to identify their current employment and relationship status to account for variables relative to the overall measure (e.g., level of functioning and disability) of the WHODAS 2.0.

The majority of the participants (n = 174; 58.4%) reported current employment as either full-time (n = 99) or part- time (n = 75), whereas 33.2% identified as unemployed (n = 99) or as a student (n = 17, 5.7%). Participants reported their relational status as single (n = 94, 31.5%),

married (n = 63, 21.1%), partnered (n = 80, 26.8%), separated/divorced (n = 48, 16.1%), and prefer not to answer the item (n = 9, 3.0%; percentages do not total 100 due to rounding).

Data Instrumentation

The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0; Üstün et al., 2010a; Üstün et al., 2010b) is a self-report instrument that measures health and disability while also capturing different social participation roles unlike any other disability scale (Moen, Drageset, Eide, Klokkerud, & Gjesdal, 2017). The WHODAS 2.0 is composed of 36 Likert (5-point) questions (e.g., starting and maintaining a conversation) referencing the previous 30 days. The responses range from 0 (no diffi- culty), 1 (mild difficulty), 2 (moderate difficulty), 3 (severe difficulty), to 4 (extreme difficulty or cannot do) on six domains (Üstün et al., 2010a).

There are two options to score the WHODAS 2.0: (a) the simple scoring method by calculating the sum of each item rating or (b) the complex method of scoring called the item response theory. The item response theory option uses an algorithm to code each response by a different weighting of the items and their levels of severity. For this study, the simple scoring method was implemented to describe the degree of functional limitations with a total score range of 0 (no difficulty) to 100 (most difficult). Therefore, a score of 100, 36 (items) × 4 (extreme difficulty or cannot do) / 144 = 1 or 100%, indicates full disability, whereas a 0 ([36 (items) × 0 (no difficulty) / 144 = 0 or 0%, represents no disability.

Data Cleaning and Screening

The data were analyzed using the Statistical Package for Social Sciences (SPSS; Windows Version 25.0), WarpPLS 6 (Kock, 2017) and SmartPLS 3.2.7 (Ringle, Wende, &

Becker, 2015). Before conducting data analyses, we screened the data for missing values and tests of normality were employed. In SPSS, Little’s (1988) missing variable analysis, a multivariate extension of a simple t test was used to determine the mechanism of missingness through examining the mean differences of the 36 items (Enders, 2010).

The results, χ²(375) = 651.220, p = .725, indicated data were missing completely at random, therefore, were indis- criminate and ignorable (e.g., <5%, Kline, 2011; Osborne, 2013). Missing data were then addressed using a single imputation method in SPSS 25.0, a modern technique recommended in SEM to handle missing values when data are missing completely at random and only a very small portion of data are missing (Enders, 2010; Graham, 2009; Hair et al., 2017). A single imputation addresses missing values

(6)

by using the expectation maximization algorithm to provide an unbiased parameter estimates and improves statistical power of analyses (Enders, 2010; Graham, 2009).

Next, we determined the nonnormality of the data using the Kolmogorov–Smirnov and Shapiro–Wilk statistics pre- sented in Table 3. The data set failed to meet the assumptions of univariate normality with several items not normally distributed. Therefore, we did not examine multivariate normality because univariate normality is a condition needed to evaluate for multivariate normality (Mvududu &

Sink, 2013). Nonetheless, a moderately robust participant per item ratio of over 8:1 was more than satisfactory (Hair, Black, Babin, Anderson, & Tatham, 2010). Thus, our sample size was appropriate for all statistical analyses conducted (Tabachnick & Fidell, 2013).

Analysis Methods

The analytic procedures for this study were designed after a thorough review of the literature identified previous factor analytic and validation studies failed to support the WHODAS 2.0 as a reflective measurement model. Thus, a crucial next step in the validation and utility of the WHODAS 2.0 as a measure of disability is to clarify the nature (i.e., reflective or formative) of its measurement model. As such, we employed CTA using PLS-SEM to examine the reflective (i.e., also known as effect; Bollen &

Ting, 2000) and formative (i.e., also known as casual or composite; Bollen & Ting, 2000; Sarstedt, Hair, Ringle, Thiele, & Gudergan, 2016) nature of the WHODAS 2.0 scores and its factor structure.

CTA-PLS facilitates a statistical evaluation of cause- effect relationships for latent variables and their specification of indicators in measurement models (Hair et al., 2017). The use of PLS path modeling in SEM is a progres- sive alternative CB-SEM (Rigdon, 2005). PLS-SEM applies ordinal least squares regression-based estimation with the goal to minimize error terms (i.e., residual variance) of the dependent (i.e., endogenous) variables (Sarstedt et al., 2016). Through CTA-PLS, the statistical test yields empirical information used to determine latent variables’ mode of measurement model to provide support for avoiding incorrect measurement model specification.

CTA-PLS builds on the concept of tetrads. A tetrad is a dif- ference between one pair of covariances to another; and ideally, all tetrads should be zero to confirm a construct to be reflective (Gudergan et al., 2008; Hair et al., 2018). If at least one of the tetrads of each construct is significantly different from zero, the reflective construct must be rejected, leading to confirmation of the formative measurement specification and theoretical considerations (Hair et al., 2018).

Furthermore, as noted, CTA-PLS is an adaptation that follows Bollen and Ting’s (2000) original CTA-SEM approach. However, although both CTA-PLS and CTA- SEM use a similar evaluation process, CTA-PLS differs with regard to (a) methodological assumptions in both the single tetrad testing approach and the simultaneous tetrad testing procedure, (b) procedures for testing the significance of the model implied tetrads, and (c) evaluation of results by accounting for the relevant set of mul- titude tests (Gudergan et al., 2008). Methodological assumptions in both single and simultaneous testing represent a minor deviation that decreases the range of pos- sible model-implied vanishing tetrads in CTA-PLS.

Furthermore, since PLS does not adhere to distributional assumptions in traditional significance testing, all tetrads must be tested using bootstrap procedures (Garson, 2016).

Accordingly, CTA-PLS uses the bias-corrected bootstrap (i.e., two-tailed; 1 − α CI) to test the model-implied Table 2. Participants’ Demographic Characteristics.

Characteristic n Total %

Gender

Male 117 39.3

Female 181 60.7

Age, years

18-24 73 24.5

25-29 69 23.2

30-34 30 10.1

35-39 24 8.1

40-44 48 16.1

45-49 15 5.0

50-54 12 4.0

55-59 5 1.7

≥60 22 7.4

Racial background

African American 37 12.4

Native American 1 0.3

Asian American 3 1.0

Caucasian/White 123 41.3

Hispanic/Latinx 55 18.5

Biracial 1 0.3

Other 9 3.0

Unidentified 69 23.2

Relational status

Single 94 31.5

Partnered 80 26.8

Married 63 21.1

Separated/divorced 48 16.1

Widowed 4 1.3

Prefer not to answer 9 3.0

Current employment

Unemployed 99 33.2

Student 17 5.7

Part-time 75 25.2

Full-time 99 33.2

Unidentified 8 2.7

Note. N = 298.

(7)

Tabet et al. 1737

nonredundant vanishing tetrads. Next, to account for testing issues, CTA uses the Bonferroni method to address multiple testing issues. In doing so, the α level of CIs is adjusted for testing the model-implied vanishing tetrads (Gudergan et al., 2008). Thereby, PLS path modeling offers a tetrad test for evaluating the mode of the measurement model; both formative and reflective (Bollen & Ting, 2000;

Gudergan et al., 2008). As such, to confirmatory test the model specifications and examine the factor structure of the WHODAS 2.0 scores; we employed a two-stage analysis using PLS-SEM and CTA-PLS, an innovative extension to CTA-SEM.

Results Data Analysis

To examine the factor structure and model specifications of the WHODAS scores as a reflective and formative measure, we utilized PLS-SEM. In the first stage, we analyzed the WHODAS 2.0 and its six reflective subscales: UC, GA, SC., GAP, LA, and PAR using WarpPLS. Next, we assessed the reliability and validity of the six reflective subscales using reflective quality criteria (Hair et al., 2017).

Subsequently, a CTA-PLS was performed in SmartPLS to assess model specifications of the WHODAS 2.0 scores, Table 3. Descriptive Statistics and Tests of Univariate Normality.

Item Minimum

statistic Maximum

statistic M statistic SD statistic

Skewness Kurtosis

Statistic Standard error Statistic Standard error

D1.1 1 4 1.98 0.952 0.506 0.141 −0.863 0.281

D1.2 1 5 2.13 0.884 0.449 0.141 −0.205 0.281

D1.3 1 5 1.93 0.931 0.796 0.141 0.007 0.281

D1.4 0 5 1.70 0.858 1.107 0.141 0.927 0.281

D1.5 1 4 1.65 0.778 0.913 0.141 −0.053 0.281

D1.6 0 5 1.87 0.978 0.853 0.141 −0.102 0.281

D2.1 1 5 1.62 1.038 1.603 0.141 1.635 0.281

D2.2 1 5 1.47 0.796 1.640 0.141 2.039 0.281

D2.3 0 4 1.34 0.708 1.990 0.141 3.191 0.281

D2.4 1 4 1.51 0.858 1.516 0.141 1.129 0.281

D2.5 1 5 1.56 1.004 1.820 0.141 2.393 0.281

D3.1 1 4 1.17 0.513 3.240 0.141 10.447 0.281

D3.2 1 4 1.17 0.496 3.183 0.141 10.684 0.281

D3.3 1 5 1.30 0.668 2.715 0.141 8.462 0.281

D3.4 1 5 1.51 0.933 1.915 0.141 3.166 0.281

D4.1 0 4 1.73 0.900 1.002 0.141 0.133 0.281

D4.2 1 5 1.81 1.000 1.275 0.141 1.074 0.281

D4.3 1 5 1.82 .925 1.113 0.141 0.824 0.281

D4.4 1 5 2.17 1.251 0.767 0.141 −0.463 0.281

D4.5 0 5 1.82 1.268 1.396 0.141 0.936 0.281

D5.1 1 5 1.92 1.005 0.843 0.141 −0.164 0.281

D5.2 0 5 1.63 .883 1.298 0.141 1.305 0.281

D5.3 1 5 2.07 1.044 0.824 0.141 0.020 0.281

D5.4 1 5 2.04 1.027 0.800 0.141 −0.016 0.281

D5.5 0 5 1.86 0.981 0.975 0.141 0.358 0.281

D5.6 0 5 1.81 0.994 1.047 0.141 0.311 0.281

D5.7 1 5 2.09 1.039 0.706 0.141 −0.215 0.281

D5.8 1 5 2.11 1.080 0.693 0.141 −0.355 0.281

D6.1 1 5 2.04 1.173 0.962 0.141 −0.021 0.281

D6.2 0 5 1.79 1.011 1.162 0.141 0.738 0.281

D6.3 1 5 1.76 0.976 1.120 0.141 0.309 0.281

D6.4 1 5 1.99 1.048 0.756 0.141 −0.370 0.281

D6.5 1 5 2.12 1.185 0.603 0.141 −0.925 0.281

D6.6 1 5 1.81 1.194 1.276 0.141 0.392 0.281

D6.7 1 5 1.66 1.040 1.501 0.141 1.213 0.281

D6.8 0 5 1.96 1.077 0.706 0.141 −0.572 0.281

(8)

enabling the researchers to assess the formative or reflective nature of the six primary constructs (Gudergan et al., 2008; Hair et al., 2018; Rigdon, 2015). Similarly, to Bollen and Ting’s (2000) CTA-SEM application, CTA-PLS involves the following steps: (a) select and compute all vanishing tetrads for the proposed measurement model, (b) identify model implied vanishing tetrads, (c) eliminate redundant model-implied vanishing tetrads, (d) test the statistical significance for each vanishing tetrad, and (e) evaluate the results for all model-implied nonredundant vanishing tetrads per measurement model by accounting for multiple testing issues (Gudergan et al., 2008).

In the second stage, we assumed the WHODAS 2.0 scale as a second-order construct and performed a CTA-PLS to assess the formative and reflective nature of its dimensions (i.e., six constructs and overall measure). Furthermore, we employed the quality criteria to evaluate the WHODAS 2.0 as a second-order measurement model and then subsequently compared those results with the first stage results.

After results were compared, we selected the best alternative measurement model based on our analyses and calculated final scores of the chosen model, a formative-formative WHODAS 2.0 model.

Assessment of WHODAS 2.0 Scores as a Reflective Model

As noted, we analyzed the WHODAS 2.0 scores in two stages. Using WarpPLS, Table 4 shows the outer loadings, composite reliability (CR), and average variance extracted (AVE) of the items in the six reflective subscales of the WHODAS 2.0. To establish evidence of the reliability and convergent validity of the six reflective subscales, the outer loadings should be greater than 0.7, Cronbach’s alpha, CR, and ρ_A greater than 0.7, and AVE greater than 0.5; however, loadings between 0.4 and 0.7 are acceptable if the CR and AVE meet the threshold (Dijkstra & Henseler, 2015; Hair et al., 2017). In Table 4, the outer loadings of all items in the six reflective subscales of the WHODAS 2.0 are higher than 0.4; but, some are lower than 0.7. Therefore, the CR and AVE were checked to determine whether to retain or remove these items. As seen in Table 4, the Cronbach’s alpha, CR, and ρ_A of all six subscales of the WHODAS 2.0 are higher than 0.7, establishing internal consistency and reliability of the scores.

Nevertheless, the results identified the AVE of three reflective subscales (i.e., UC, SC, and GAP) were lower than 0.5, therefore, convergent validity was identified as a problem for three subscales of WHODAS 2.0 scores. To address and resolve the identified problem of convergent validity in the UC, SC, and GAP subscales, it is recommended to remove a few problematic items from the respected subscales (Hair et al., 2018). However, before removal of the problematic items, we performed a

CTA-PLS in SmartPLS to assess the nature (i.e., reflective or formative) of the six subscales. Specifically, two steps were taken to perform the CTA-PLS, including (a) all tetrads were calculated for the items of each construct and (b) the redundant tetrads were eliminated.

Subsequently, we conducted the significance tests using bootstrapping to calculate the p value (p = Pr [T ≥ t | H₀]) and the 95% bias-corrected CI to assess whether the nonredundant tetrads were significantly different from zero (Hair et al., 2018). The tetrad did not vanish and was significantly different from zero if its p value was lower than .05 and zero was not included within the bias-corrected CI_0.5 and CI_0.95 (Gudergan et al., 2008; Hair et al., 2018). The results (Appendix A) of CTA-PLS identified that several tetrads did not vanish for each of the six WHODAS 2.0 subscale scores. Therefore, the results demonstrated that the six subscales of WHODAS 2.0 scale were not reflective; instead, were formative in nature. Afterward, we then switched the items of all six subscales of WHODAS 2.0 scale to formative according to the results of CTA-PLS.

Assessment of WHODAS 2.0 as a Formative Model

We tested two criteria to assess the WHODAS 2.0 as a formative measurement model. The collinearity between an item, the variance inflation factor (VIF), and the outer weights of the items (Hair et al., 2017). Table 4 presents the results of the WHODAS 2.0 six subscales’ VIFs and outer weights. The VIFs of all the items were lower than 5; and, the p value of outer weights were lower than 0.05, except for Items D2.4 and D5.6. However, in the case of a nonsig- nificant outer weight within a formative construct, we can retain the problematic item if the associated outer loading is significant (Hair et al., 2017). Our results exhibited substantial outer loadings for both Items D2.4 and D5. Thus, the items were retained and the six subscales of WHODAS 2.0 met sufficient criteria to assess the formative measurement model. Consequently, the results of both the CTA-PLS and assessment of quality criteria for the formative measurement model confirmed the formative nature of six subscale scores of WHODAS 2.0.

In the second stage, we assumed the WHODAS 2.0 as a second-order measurement model. The associated WHODAS 2.0 items were represented to their respected subscale by latent variable scores extracted from the first stage. A CTA-PLS was then performed to assess the formative and reflective nature of the items of the WHODAS 2.0.

The results (Appendix B) of the CTA-PLS for the second- order WHODAS 2.0 revealed that several tetrads did not vanish. Therefore, a second-order WHODAS 2.0 measurement model cannot be reflective and was identified as formative. Additionally, Table 5 presents the criteria to assess the formative measurement model for the second-order

(9)

1739 Table 4. Assessment of Measurement Model for Six Dimensions (UC, GA, SC, GAP, LA, PAR) of WHODAS 2.0 Scale. ConstructItemsLoadingCronbach’s αCRAVEWeightspVIF Understanding and communicating (UC)

0.8150.8190.431 D1.1Concentrating on doing something for 10 minutes.0.7110.266<.011.8 D1.2Remembering to do important things.0.6880.217<.011.693 D1.3Analyzing and finding solutions to problems in day-to-day life.0.7020.246<.011.819 D1.4Learning a new task, for example, learning how to get to a new place.0.6410.174<.011.631 D1.5Generally understanding what people say.0.6560.214<.011.634 D1.6Starting and maintaining a conversation.0.5240.112<.051.297 Getting around (GA)0.8780.8820.60 D2.1Standing for long periods, such as 30 minutes.0.8290.325<.013.233 D2.2Standing up from sitting down.0.7370.127<.052.012 D2.3Moving around inside your home.0.8310.352<.012.866 D2.4Getting out of your home.0.6540.083.0751.617 D2.5Walking a long distance, such as a kilometer (or equivalent).0.8070.213<.012.935 Self-care (SC)0.6970.720.402 D3.1Washing your whole body.0.7160.216<.012.494 D3.2Getting dressed.0.7610.567<.012.77 D3.3Eating.0.5720.1670.021.345 D3.4Staying by yourself for a few days.0.4320.122<.051.159 Getting along with people (GAP)0.7870.7960.444 D4.1Dealing with people you do not know.0.6960.21<.011.793 D4.2Maintaining a friendship.0.7490.337<.011.972 D4.3Getting along with people who are close to you.0.620.178<.011.417 D4.4Making new friends.0.7420.328<.012.133 D4.5Sexual activities.0.4860.106.0311.193 Life activities (LA)0.9340.9350.642 D5.1Taking care of your household responsibilities.0.8450.189<.014.982 D5.2Doing most important household tasks well.0.8110.147<.0013.336 D5.3Getting all of the household work done that you needed to do.0.8050.13<.054.459 D5.4Getting your household work done as quickly as needed.0.810.125<.054.028 D5.5Your day-to-day work/school.0.7280.107<.052.823 D5.6Doing your most important work/school tasks well.0.720.087.0643.122 D5.7Getting all of the work done that you need to do.0.8490.21<.015.387 D5.8Getting your work done as quickly as needed.0.8340.158<.015.092 (continued)

(10)

1740

ConstructItemsLoadingCronbach’s αCRAVEWeightsp Participation in society (PAR)0.8880.8910.508 D6.1How much of a problem did you have in joining in community activities (e.g., festivities, religious, or other activities) in the same way as anyone else can?

0.60.095<.05 D6.2How much of a problem did you have because of barriers or hindrances around you?0.6370.123<.05 D6.3How much of a problem did you have living with dignity because of the attitudes and actions of others?0.6080.094<.05 D6.4How much time did you spend on your health condition or its consequences?0.7390.132<.01 D6.5How much have you been emotionally affected by your health condition?0.7930.237<.01 D6.6How much has your health been a drain on the financial resources of you or your family?0.8150.273<.01 D6.7How much of a problem did your family have because of your health problems?0.7530.119<.05 D6.8How much of a problem did you have in doing things by yourself for relaxation or pleasure?0.7250.143<.01 Note. WHODAS = World Health Organization Disability Assessment Schedule; CR = composite reliability; AVE = average variance extracted; VIF = variance inflation factor.

Table 4.(continued)

(11)

Tabet et al. 1741

WHODAS 2.0. The VIF of all the subscales of the second- order WHODAS 2.0 are lower than 5; and, the p value of outer weights are lower than .05 and significant. Therefore, the results of CTA-PLS for the second-order WHODAS 2.0 and quality criteria for formative measurement model identified the formative nature of the second-order.

Discussion

We employed a CTA-PLS to assess the reflective and formative nature of the WHODAS 2.0 as both a first-order six- factor model and a second-order six-factor model based on data from a sample of adult clients receiving individual outpatient counseling services at a UBCCRC. Through the use of a CTA-PLS and theoretical considerations in our study, we confirmed the inappropriateness of a reflective WHODAS 2.0 measurement model and identified support for a formative WHODAS 2.0 measurement model (Gudergan et al., 2008; Hair et al., 2018). Our results support the WHODAS 2.0 as a first-order six-factor measurement model, but most adequately as a second-order six-factor formative measurement model. Contrariwise to the developers intended model (Üstün et al., 2010b), our findings along with theoretical reasoning indicated the WHODAS 2.0 is best conceptualized as a formative model, and more precisely, as a second-order formative–formative measurement model as seen in Figure 1.

Overall, our findings contribute to the growing body of literature examining the factor structure of the WHODAS 2.0 and alternative measurement models. We are the first to evaluate the appropriateness of the WHODAS 2.0 scores as a reflective model and assess its factor structure as both reflective and formative. In doing so, we reiterate theoretical considerations first raised by Küçükdeveci et al. (2013) and Federici et al. (2017) over its intended conceptual overlap among the items and subscales as well

as its intended measurement (i.e., disability and functionality). The formative nature of our model further supports the six factors found in previous factor analytic studies (Cheung et al., 2015; Chiu et al., 2014; Federici et al., 2009; Galindo-Garre et al., 2015; Garin et al., 2010;

Guilera et al., 2012; Guilera et al., 2015; Küçükdeveci et al., 2013; Magistrale et al., 2015; Zhao et al., 2013). In Stage 1 of our two-phase analyses, we confirmed the six constructs to be formative through the comparison of pairs of covariances, also known as tetrads. We then assessed the criterion for the six subscales, demonstrating satiable outer loadings aiding evidence to the WHODAS 2.0 unidi- mensionality of its six subscales.

Based on conceptual, pragmatic, and statistical evidence, our innovative second-order formative–formative WHODAS 2.0 measurement model asserts that the items inform their intended subscale; and, subsequently, the six subscales inform the overall measure of disability.

Conceptually, disability refers to a vast array of medical and health components while also integrating social fac- ets. However, when disability and its subscales (i.e., UC, GA, SC, GAP, LA, and PAR) are modeled as reflective indicators, it is not explicit that the resulting latent variable accurately represents its intended conceptual framework. Rather than characterizing disability as the combination (summation) of a constituent interaction between a person’s health condition(s) and context (environmental and personal factors), reflective indicator models represent disability more narrowly as that variation that is shared across a series of impediments of daily living. It is the discrepancy between the conceptual definition of disability and the statistical representation of disability using reflective indicators that is the overarching interest of this study.

We conjecture that formative indicator models yield a statistical representation of disability that is more compati- ble with the proposed conceptual definition. Within this framework, disability encompasses a representation of a subset of manifestations. For example, Item D2.3, moving around inside your home, was previously purported to be a result of an individual’s measure of mobility in the WHODAS 2.0 reflective models. Whereas, in our formative model, Item D2.3, as well as the other four items on the mobility subscales, it was responsible for influencing or informing the total measure of mobility. As such, the six subscales cumulative measures are then weighted equally as indicators of the overall WHODAS 2.0 measure of disability and functioning, thus creating a second-order formative–formative model. Through reconceptualizing the theoretical framework of the WHODAS 2.0, the original second-order measurement model can be supported as seen on the second-order formative–formative model for this study.

Table 5. Assessment of Measurement Model for Formative Second-Order WHODAS.

Dimensions Weights p VIF

WHODAS (OM)

UC 0.239 <.01 2.787

GA 0.110 <.05 2.492

SC 0.141 <.01 1.906

GAP 0.105 <.05 2.236

LA 0.161 <.01 2.362

PAR 0.369 <.01 4.043

Note. WHODAS = World Health Organization Disability Assessment Schedule; VIF = variance inflation factor; OM = overall measure;

UC = understanding and communicating; GA = getting around;

SC = self-care; GAP = getting along with people; LA = life activities;

PAR = participation in society.

(12)

Limitations

Our sample was limited to a singular UBCCRC and one state within the United States. Thus, our sample is not rep- resentative of diverse locations and limits the generaliz- ability of results across subpopulations. Another limitation was the variability of presenting problem(s) across our participants. We did not implement screening measures and inclusionary criteria for specific mental disorders. As such, not accounting for specific mental disorders, may also limit the applicability of results using the WHODAS 2.0 in counseling centers. Nevertheless, our sample is representa- tive of various mental disorders as well as participants

from different ethnic and racial backgrounds. An addition limitation of our study was that we did not include multiple assessment points using the WHODAS 2.0 through the course of counseling services, hindering the measurement invariance of the WHODAS 2.0 across time and prohibited the evaluation of test–retest reliability on the WHODAS 2.0. Despite these limitations, this study was the first to evaluate the factor structure and model specifications of the WHODAS 2.0 using CTA-PLS. The innovative analysis offered favorable findings of the second-order formative–formative WHODAS 2.0 model, adding to the gap of empirical support for the WHODAS 2.0 as a second-order measurement model.

Figure1. Second-order formative–formative WHODAS 2.0 measurement model with outer weights.

Note. WHODAS = World Health Organization Disability Assessment Schedule; OM = overall measure; UC = understanding and communicating;

GA = getting around; SC = self-care; GAP = getting along with people; LA = life activities; PAR = participation in society.

(13)

Tabet et al. 1743

Implications for Health Care Practitioners

Several implications of our study have emerged for health care practitioners. Given that the WHODAS 2.0 is used worldwide as a measure of potential disability and functionality, interpretation should be made cautiously. As such, with no precise cutoff scores or validated clinical range, scoring of the WHODAS 2.0 aides in the understanding of disability is not a definitive indicator. Furthermore, the debatable factor structure of the WHODAS 2.0 scores war- rants further attention concerning the interpretation of measure. Health care practitioners using the WHODAS 2.0 are encouraged to apply our second-order formative–formative model as a framework for conceptualizing disability and functioning. Through the second-order formative–formative WHODAS 2.0 model, the 36 items influence their specified subscales, affecting the overall WHODAS 2.0 measure. Consequently, we suggest the above implications be taken into consideration when using the WHODAS 2.0 in clinical practice.

Recommendations for Future Research

Future research should aim to replicate the second-order formative–formative WHODAS 2.0 model found in our study with more diverse populations to include varying geographi- cal locations, samples from different clinical settings, and persons with various disorders/ailments associated with disability. As such, future research would benefit from assessing the formative nature of the 12-item, shortened WHODAS 2.0. Comparison studies would aid not only in determining the most appropriate measurement model of the WHODAS 2.0 but also contribute to known information about the invariance percentages between the original 36-item WHODAS 2.0 and the 12-item, shortened WHODAS 2.0. Moreover, assessing the WHODAS 2.0 concurrent validity with similar validated instruments measuring constructs of disability and functionality would provide support for its usage in determining a person’s disability level. Furthermore, an examination into the WHODAS 2.0 cutoff scores is needed to provide a baseline and clinical range for its providers.

Appendix A

Results of CTA-PLS for the Six Dimensions of WHODAS 2.0.

Tetrads p Bias

CI

LL UL

Understanding and communication

1: D1.1, D1.2, D1.3, D1.4 0.033 .225 −0.002 −0.013 0.076

2: D1.1, D1.2, D1.4, D1.3 0.031 .2 −0.003 −0.012 0.068

4: D1.1, D1.2, D1.3, D1.5 −0.009 .661 0 −0.041 0.023

6: D1.1, D1.3, D1.5, D1.2 −0.007 .753 0 −0.041 0.028

7: D1.1, D1.2, D1.3, D1.6 −0.024 .326 0 −0.064 0.017

10: D1.1, D1.2, D1.4, D1.5 0.066 .005 −0.001 0.027 0.104

16: D1.1, D1.2, D1.5, D1.6 0.038 .107 −0.001 −0.001 0.076

22: D1.1, D1.3, D1.4, D1.6 −0.003 .903 −0.002 −0.045 0.036

26: D1.1, D1.3, D1.6, D1.5 0.047 .067 0 0.005 0.089

Getting around

1: D2.4, D2.5, D2.1, D2.2 0.036 .155 0 −0.006 0.077

2: D2.4, D2.5, D2.2, D2.1 0.007 .82 0 −0.045 0.06

4: D2.4, D2.5, D2.1, D2.3 −0.006 .711 0 −0.035 0.021

6: D2.4, D2.1, D2.3, D2.5 −0.093 .003 0.004 −0.14 −0.038

10: D2.4, D2.1, D2.2, D2.3 0.023 .231 0 −0.009 0.056

Self-care

1: D3.1, D3.2, D3.3, D3.4 0.034 .015 0 0.012 0.058

2: D3.1, D3.2, D3.4, D3.3 0.03 .03 0 0.008 0.053

Getting along with people

1: D4.1, D4.2, D4.3, D4.4 −0.037 .305 −0.001 −0.098 0.022

2: D4.1, D4.2, D4.4, D4.3 −0.123 .004 0.005 −0.189 −0.047

4: D4.1, D4.2, D4.3, D4.5 0.051 .129 −0.003 −0.007 0.103

6: D4.1, D4.3, D4.5, D4.2 0.011 .658 −0.001 −0.031 0.052

10: D4.1, D4.3, D4.4, D4.5 −0.139 .015 0.009 −0.224 −0.036

(continued)