Consistency of scoring within and between groups was measured by my study using Inter- class Correlation Coefficient (ICC). ICC is scored on a scale of 0-1, with less than 0.50 indicating poor reliability, between 0.50 and 0.75 moderate reliability, between 0.75 and 0.90 good reliability, and above 0.90 excellent reliability103. Before training, all participant groups scored good to excellent inter-rater reliability for all FAUs, except for animal welfare
58 inspectors who only had moderate reliability (ICC = 0.73) when scoring prominent strained chewing muscles. Lower 95% confidence intervals often dropped into the moderate reliability range. The animal welfare inspector participant group often had more variation in their
scorings, especially for strained nostrils and flattening of the profile (ICC = 0.80), and prominent strained chewing muscles (ICC = 0.73), both of which had 95% confidence
intervals that fell into the poor reliability range. It is unclear why this is, but it is possible that animal welfare inspectors have more variable levels of experience and knowledge related to horses.
After training, ICC scores for most FAUs also indicated good to excellent inter-rater reliability. There was slightly more variation in scores by equine physical therapists when considering strained nostrils and flattening of the profile (pre-training ICC = 0.92, post- training ICC = 0.87). There was also a particular decrease in variation of scoring by the animal welfare inspector group when considering prominent strained chewing muscles, strained nostrils and flattening of the profile, and orbital tightening. Increased reliability after training was also indicated by a decrease in 95% error by all groups, especially in relation to orbital tightening, strained nostrils and flattening of the profile, mouth strained and
pronounced chin, and prominent strained chewing muscles. Stiffly backwards ears scores were in particularly strong agreement after training for all groups. This supports some findings by Dai et al. (2020), who found that training particularly increased accuracy for scoring orbital tightening (post-training Cohen’s Kappa coefficient = 0.91) and stiffly backwards ears (post-training Cohen’s Kappa coefficient = 0.90) on the HGS when
comparing control participants to one another87. Other research has also suggested that stiffly backwards ears (ICC = 0.97) is the most consistently scored FAU66.
Stiffly backwards ears had excellent reliability for all participant groups, even when
considering 95% error. This indicates that stiffly backwards ears may have been the easiest FAU for all participant groups to score even before training was received. However, this margin of error increased slightly after training for physical therapists and animal welfare inspectors. This was not a substantial increase and overall ICC remained within the excellent reliability range. Previous studies have also indicated that participants have stronger
agreement when scoring stiffly backwards ears when using the HGS33,66. Before training, when compared to scoring of the researchers, Cohen’s Kappa coefficient was 0.68 for stiffly backwards ears in comparison to only 0.20 for tension above the eye area87. Other research considering the reliability of stiffly backwards ears between participants also found it to be
59 the most consistently scored FAU (ICC = 0.97)33. This agrees with the findings of the present study.
Previous research by Dai et al. (2018) has found that tension above the eye area and strained nostrils may be more difficult to score than other FAUs on the HGS before participants received training106. However, this study only included five experienced observers with knowledge of equine welfare and/or FAU scoring. After training in my study, participants had more variability in their scores when considering the FAUs tension above the eye area, and mouth strained and pronounced chin (ICC for each = 0.91). This was followed by strained nostrils and flattening of the profile (ICC = 0.92), prominent strained chewing
muscles (ICC = 0.93), orbital tightening (ICC = 0.94), and then stiffly backwards ears (ICC = 0.97). However, for both my study and the study by Dai et al (2018), ICC results were all similar and within the excellent reliability range.
Another study by Dalla Costa et al. (2016) on the use of the HGS for horses experiencing laminitis found slightly more variable results in ICC for HGS scorings, with an ICC of 0.44 for prominent strained chewing muscles, 0.68 for tension above the eye area, 0.76 for strained nostrils and flattening of the profile, 0.80 for mouth strained and pronounced chin, 0.93 for orbital tightening, and 0.95 for stiffly backwards ears66. All FAUs in this study, apart from orbital tightening and stiffly backwards ears, had noticeably less consistency between scorers compared to my study. However, this is unsurprising since that study only included four veterinarians as their entire participant group, so individual differences in scoring would have had a large effect on ICC66.
The horse owner/rider participant group had the highest similarity for all FAUs in accordance with ICC within the group. It is possible that because they were such a large group, making up 47.5% of all respondents who answered the pre-training questions, individual variations in scoring did not have as much of a significant effect on overall inter-rater similarity. Another possibility is that horse owners/riders have similar understandings of horses and similar knowledge/beliefs about how they expect horses to display pain. However, this would contradict with the findings in the study by Stellon et al. (2022), that suggested that
differences in demographic factors within such as number of horses owned, sex, and previous professional experience, had a significant impact on how they scored pain105. Number of horses owned was not considered in the present study as a factor that could influence scoring
60 of the HGS, although number of years participants had been closely involved with horses did not significantly affect the present study’s results.
61