Chapter 3: Methodology
3.6. Validity and Reliability
3.6.2. Threats to Internal and External Validity
“attractiveness” was a cumulative term for both pleasantness and sexiness. In addition, the term “attractiveness” is defined as how attractive one individual or object seems to another, whereas, the term pleasantness and sexiness both may not necessarily imply attraction specifically.
It was also necessary to establish whether masculinity as a construct was judged similarly to what previous research suggests. According to Thornhill and Gangestad (1999) masculinity is precipitated by a pronounced jaw, facial hair and a heavier set brow, which would be evident in the photographs. Measures of reliability such as 𝑊̃ would indicate whether the features that are deemed masculine in the literature are validated by the consensus of the sample.
Furthermore, whether individuals consider visual masculinity (VM) similarly to scent masculinity (SM) and how these rankings of masculinity fared against rankings of attractiveness, especially for women. Therefore, it was necessary to include both measures of attractiveness and masculinity to provide criterion validity to the study.
Extraneous and confounding variables
Tredoux and Smith (2006) state that confounding variables are any extraneous elements that may inadvertantly affect the outcome of a study and thus subvert the internal validity of the research. Therefore it was vital that the researcher aimed to control for as many confounding variables as was possible, as the validity of the conclusions drawn depended upon it. In this study, a number of possible extraneous variables were considered as potential threats to the internal validity of the research. Firstly the demographics of the participants both for the stimuli and judges was considered to be potentially confounding.
For the stimuli participants, age, race and SO was controlled for. Firstly as there were only six stimuli participants any significant variation in the age of the participants may affect how they appear visually and affect the degree of T that they produced and consequently how attractive or masculine they presented. Therefore the age range of the stimuli participants was limited between 19 and 25 years. The race of the stimuli participants was also considered a threat to the internal validity as mentioned previously, as it is intrinsically linked to culture and subsequently diet which would affect BO (van Beek, 1992). Therefore only two race groups were used to create the stimulus, black and white and were matched accordingly.
Furthermore, according to the biological theories described in chapter 2 regarding SO, hormonal differences often occur in differing SOs, which would affect the pheromones secreted. Therefore, all of the stimuli items used in the study were produced by participants reported to be heterosexual. Additionally the stimuli participants were asked to follow a set of guidelines as used by Thornhill and Gangestad (1999) which, if followed would eliminate the potential for confounding scents to contaminate the t-shirts. These guidelines included an avoidance of smoking, and recreational drug and alcohol use whilst wearing the t-shirts, the avoidance of sexual encounters whilst wearing the t-shirts as well as the avoidance of particular foods. Participants were also asked to refrain from using any fragranced cosmetic products whilst wearing the t-shirts and were given a fragrance free soap to wash with, thus ensuring that only the natural BO was left on the t-shirt. Participants were reminded not to wash the t-shirts. All of the T-shirts given to the participants were identical in size and colour, thus eliminating any confounding effect that size or colour preference may have.
Furthermore, the use of only six male stimuli participants may have limited the range of represented attractive and masculine men in terms of the stimuli target population. A larger stimulus range may have been necessary to obtain adequate variation amongst the stimuli.
Any other extraneous variables that could potentially threaten the internal validity of the
study were either impractical or unethical to measure, such as disease or hormonal imbalance and therefore were not accounted for.
For the judging participants, confounding variables such as menstrual phase, race and age were also considered to potentially confound the results. Therefore an estimation of menstruation for female participants was included in the questionnaire and ovulation estimated from that. This however, is not the most accurate measure of detecting ovulation as women may not be accurate in their estimations and furthermore, the assumption that all the female participants have regular menstrual cycles was made. A more precise indication of ovulation would have required the implementation of ovualtion tests, which would have been infeasable and potentially unethical. Therefore the basic estimation of ovulation was used.
Additionally in the creation of the questionnaire, the potentially confounding affects that hormonal contraceptives used by women could have, was overlooked and therefore questions regarding hormonal contraceptive usage was not included. This may be a confounding factor in the study (Wedekind, et al., 1995; Singh & Bronstad, 2001). For the judging participants, all races were included, as race was only an important consideration in the production of BO.
However, additional analyses were still conducted to assess whether race did signicantly predict the way any of the individuals ranked the visual and scent stimuli, although this was not expected. Age was also included in additional analyses as a potential confounding variable. Previous literature suggests, that men regardless of age prefer younger partners and women generally prefer older partners (Geary, Vigil, & Byrd-Craven, 2004). This, however, is predominently shown in heterosexual populations, additional analyses were conducted assessing whether age and SO were associated with regard to preference rankings.
Campbell’s Schema
In 1957, Campbell proposed a schema for assessing the internal and external validity specifically in experimental research. Campbell (1957) identified six potential threats to the internal validity of a study design which include the threat of covarying events, maturation, testing effects, instrument decay, statistical regression to the mean and subject mortality. For this study Campbell’s schema for identifying threats was quite useful, although not all the threats were applicable.
The first threat identified using Campbell’s schema, was the threat of covarying events.
According to Campbell (1957), this is the potential for outside, extraneous events occuring that would affect the attitudes or behaviours of the repondents. Because the sampled
population of this study included students from UKZN, threats of this nature may have included disruptive conduct in and around the UKZN campus (i.e. protest action), or prior commitments such as lectures which may have rushed respondents in there completion of the ranking process. Therefore, all respondents were asked before being recruited if they were available and then shown to a private room, whereby no other individuals or distractions were present and participants were able to complete the entire procedure without the influence of covarying events.
Order effects were of significant consideration with regard to the threat of testing effects as the research made use of a repeated measures design, where each subject was asked to rank multiple stimuli (treatment conditions). Order effects may threaten the internal validity of a study as the dependant variable or in this case the rankings may be determined by the order in which stimuli are presented rather than the independent variable (Gravetter & Forzano, 2009). Although no randomisation steps were implemented to order the presentation of the stimuli to the participants, they were allowed to look at each VS and smell each t-shirt more than once and in any order. It was therefore unlikely that the order in which participants viewed and smelled each stimulus was the same for any participant.
This study was also vulnerable to the threat of instrument decay. According to Campbell (1957), the threat that instrument decay may have on the internal validity of a study occurs when a change in the instrument of measurement influences the outcome of the dependant variable. In this study instrument decay occurred in two ways. Firstly, Campbell (1957) suggests, particularly in the social sciences, that the instrument of measurement can often be human subjects, which is the case in this study as particpants are asked to be judges by providing rankings. Instrument decay in this case would occur due to olfactory fatigue as participants were asked to smell a number of t-shirts, and their ability to discriminate between smells, especially when they are subtle as in the t-shirt, would be affected. For this reason, participants were asked to smell a lemon between each t-shirt in order to refresh their olfactory senses. The second way in which instrumnet decay occurred, and which was most problematic to the recruitment of adequate sample sizes, was the decay of the t-shirts. That is that the t-shirts began to lose their scent and became contaminated by other scents. It was therefore necessary to wash the t-shirts without detergent, to reduce the effect that detergent may have on scent and ask the stimuli participants to rewear the t-shirts. Fortunately all six of the original participants agreed to further participate in the study and received a further incentive of R20.00.
Threats to External Validity
External validity refers to sample selection and whether conclusions drawn from a study would be relevant and applicable to other samples or populations (Tredoux & Smith, 2006).
In this study, purposive and convenience sampling was used to recruit participants. The judging participants were selected from the student population at the local UKZN campus as well as the local GLN. Furthermore, sample sizes were small which would convey an underrepresentation of the actual population. Therefore the external validity concerning the generalisability of the results to other populations is unfortunately biased, and conclusions drawn from that data may only be generalised to the specific population it was drawn from.