CHAPTER 5: CONCLUSIONS, LIMITATIONS AND
5.5 RECOMMENDATIONS FOR FUTURE RESEARCH ON ITEM FORMATTING
This section makes recommendations to assist future research endeavours specifically focusing on structural fit and item formatting of personality measurements. It also includes a discussion relating to the generalisability of this study’s design and sample.
The first recommendation is that this study, which focused on the best factorial fit between block- and random-item response forms, be duplicated utilising another sample to widen the representativeness of the data obtained from the analysis. This recommendation specifically addresses the limitation of non-representative sampling as discussed in section 5.3. Future researchers could also increase the sample size and the geographical scope of the sample to ensure inclusion of the multi-cultural context found in South Africa.
The researcher would also recommend that future researchers follow the second data collection strategy used in this study by scheduling specific administration sessions. This strategy facilitates the data collection process immensely and improves administrative matters.
Due to limited literature on item formatting within personality assessments it would be interesting to use this study’s data and investigate alternative components, such as the impact of block- and random-item formatting on the different demographic groups in South Africa. Previous research suggests that there are differences between South African individualistic and collectivistic cultures and therefore investigating the impact of block- and random-item sequencing on cultural groups and linking this to collectivism and individualism could be interesting.
To build on research concerning the SAPI’s item formatting, future researchers could use the data acquired for this study and perform a Confirmatory Factor Analysis (CFA) to confirm the results stated in Chapter 4 (Suhr, 2006). According to Byrne (2005), the use of a CFA allows the findings of a study to become more theoretically grounded, which will also legitimise the findings of this study.
In relation to the fourth limitation discussed in section 5.3 regarding the factors with less than three items, it is advised that, for practical reasons, these factors be removed in future research (Costello & Osborne, 2005; Labuschagne, 2010). This recommendation is based on the fact that factors with less than three items do not divulge sufficient information about a specific construct, nor do they significantly contribute to the overall reliability of the construct. One such example is the block Emotional Stability factor.
By adhering to the above recommendations, it would appear that the design and sample utilised in this study could also be used in additional studies relating to psychometric assessments in South Africa. These alternative studies do not specifically need to focus on block- and random-item order, but could investigate other item sequencing options as noted in section 2.7.1.
5.5.1 RECOMMENDATIONS PERTAINING TO THIS STUDY
This section discusses the recommendations pertaining to this specific study. It is incorporated to address the issues pertaining to reliability, and the empirical, statistical and psychometric properties of the SAPI. It aims to create awareness around these issues for potential future studies.
Firstly, it is recommended that the “I try to fool others” item be removed from the block Integrity analysis to improve the cluster reliability score from 0.65 to 0.71. Both these Cronbach alpha values are deemed as acceptable.
Secondly, attention should also be given to the random Extraversion and Intellect clusters as the factor analysis showed unsatisfactory statistical and empirical pattern matrixes. The random Intellect analysis showed that two of the factors presented with only one item
each. This is not an ideal situation as the results are deemed unacceptable statistically and empirically for analysis. It was therefore decided to make use of a three-factor solution, for comparability reasons with the block response set. Additional analysis was performed to improve the output of the analysis, whereby the researcher set the cut off at
< 0.35 (to allow for more items to load onto the problematic factors). However, similar results were obtained and therefore the < 0.40 results were utilized for consistency and comparability reasons pertaining not only to the block response set, but also to the rest of the SAPI clusters.
The random Extraversion analysis showed that the fourth factor did not load any visible items. Additional analysis was performed to improve the output of the analysis, whereby the researcher set the cut off at < 0.35 (to allow for more items to load). However, similar results were obtained and therefore the < 0.40 results were utilized for consistency and comparability reasons pertaining not only to the block response set, but also to the rest of the SAPI clusters.
Costello and Osborne (2005) noted that if the factor loadings appear disorganised and uninterpretable, as was the case for the random Intellect and Extraversion clusters, this is unlikely to be improved by manipulating number of factors or the cut off values of the factor loadings as the problem probably originates from the data. To improve the factor loadings, they advise the removal of problematic items (with low loadings) followed by repeating the analysis (only if the removal of the items does not compromise the reliability of the data).
However, considering that this study made use of descriptive research, allowing for comparison between the block- and random-results, item removal was not ideal as it would impact the comparability of the formats and thereby compromise the reliability of the data.
Therefore no additional items were removed; only items that were problematic in both response sets were removed as part of the data screening process and this was done specifically for comparability reasons. Costello and Osborn (2005) further suggested that if the structure factors remain problematic probable causes include the item construction, scale design or a limited sample size and future researchers should investigate these possible problems.
Thirdly, according to Costello and Osborne (2005), factors with less than three items should be removed. This recommendation was accepted and promoted for future research with the exception of the factors within the Openness cluster namely, Individualism, Traditionalism and Religiosity, which loaded onto both the block- and random-response sets. These three factors repeatedly loaded and appeared to be pertinent and clear cut factors. It is therefore recommended that further research be conducted on these factors, perhaps allowing for the inclusion of items specifically measuring these three factors in more depth within the South African context.