CROSS-CULTURAL DESIGN
2.3 Methodology
2.3.1 Cross-Cultural User Research
Culture is a complex and multidimensional concept.
People from different cultures are different in their perception, cognition and thinking styles, language, color coding and affect, and so on. Thus, a better understanding of different cultural traits in the design process is imperative in cross-cultural design. This is particularly true in the Asian Pacific area, especially in China, since in the future the Chinese will comprise one of the largest user populations.
Chinese users include people in Mainland China, Taiwan, Hong Kong, Singapore, and other areas with Chinese heritage. Chinese people speak Mandarin and other dialects, and even within the Chinese population,
Table 3 Human Body Movement Database
No. Author Nationality Sample Data
1 Barter (1957) U.S. Military 19 joint movements
2 Lehto and Buck (2008) U.S. Civilian 21 joint movements
3 NASA (2010) U.S. Military 25 joint movements
4 Hu et al., (2006) Chinese Civilian, elderly 18 joint movements
5 DINED Dutch Civilian 11 joint movements
Table 4 Guidelines for Physical Ergonomics and Anthropometry
Supporting Research/
Category Guidelines Best Practice
Body dimension and movement
When designing devices, equipments, and systems, target user’s body measurements and movement measurements should be checked and designed to accord with the critical measurements in the anthropometric database for different nations.
Refer to the Appendix.
CROSS-CULTURAL DESIGN 171 there are diverse Chinese subcultural groups. For exam-
ple, Chinese users in Mainland China use a simplified Chinese writing system and Chinese users in Taiwan use a traditional Chinese writing system. In general, Chinese users in Taiwan have had more opportunities to learn American culture than Chinese users in Mainland China within the past 50 years. Thus, it is expected that the differences between users in Taiwan and users in the United States would be smaller than the differences between users in Mainland China and users in the United States (Rau et al., 2004).
Bonds (1986) presented an extensive overview of the psychology of the Chinese people. In his book, he discussed Chinese patterns of socialization, perceptual processes, cognition style, personality traits, psychopathology, social psychology, and organizational behavior. The book provides insights of the culture differences between the Chinese and people from other cultures.
In addition, Yang (2001a) collected her previous published papers on how to study the Chinese, the indigenous approach. She systematically summarized the ways to conduct studies in China and how to local- ize studies in China. The indigenous approach Yang adopted is based on local materials and observations, a set of commonly shared meaning systems with which the people under investigation make sense of their lives and their experiences and give out and derive meanings while interacting with each other (Yang, 1991, 2000a, 2000b, 2001b). This also helps indigenous researchers understand and interpret the behaviors manifested by the people under study. The indigenization movement flourished from a general dissatisfaction among psychol- ogists and other social scientists over employment of the Western cross-cultural approach for understanding non-Western peoples (Li et al., 1985).
When conducting cross-cultural studies in China, special issues concerning cross-cultural comparison should be carefully considered to ensure the reliability and validity of the study.
First, cautions should be taken in the explanation of culture differences by different countries. Researchers normally use country as a proxy for culture, for example, they select participants from China and the United States to represent Eastern and Western cultures. According to the results from Schaffer and Riordan (2003), 79%
of the cross-cultural organization studies use country as a proxy for culture. That would be very dangerous, since, for example, even within the Asia Pacific Chinese population multilingual and cultural issues exist. Chu et al. (2005) compared the decision process of two closely related nations in East Asia. These results demonstrate the danger of generalizing decision theories across national boundaries, even when the nations are seemingly closely related. The results also indicate that the differences in decision processes among nations cannot be easily characterized as East versus West.
Therefore, a better approach for researchers is to start from the theoretical framework to select and collect data, rather than simply characterize the cultural differences by countries.
Second, cautions should be taken in the equivoca- tion of concepts in different cultures. Many studies have been extensively conducted in the United States, and many concepts in academia come from U.S. standards.
However, some of the concepts may not carry the same meanings in the Chinese culture. For example, the con- cepts of personality traits, self-construal, achievement, and so on, may be different. Farh et al. (2004) have compared the forms of organizational citizenship behav- ior (OCB) that have appeared in the Chinese and West- ern literature. They found 10 dimensions of OCB in China, with at least one dimension not evident at all in the Western literature. Thus, careful analysis should be performed when examining the equivalence of concepts in different cultures.
Third, caution should be taken on the equivocation of measurements in different cultures. This is especially important in the use of questionnaires. Back-translation is essential for researchers conducting cross-cultural studies with subjects whose native languages are different from the researchers’ mother tongue. Brislin’s (1970) classic paper investigated the factors that affect translation quality and how equivalence between the source and target versions can be evaluated.
2.3.2 International Usability Evaluation The Moderator and the Test Subject Usability testing involves human social interaction between a test moderator and a test subject. Social and cultural norms affect this interaction just as they affect other interpersonal interactions. There is a growing literature on how Easterners and Westerners react in usability tests. A number of best practices now can be described that help to mitigate cultural bias in usability tests resulting from social interaction effects.
First, it is a good practice to use moderators, evalu- ators, or interviewers from the same culture as the test subjects. Vatrupu and P´erez-Qui˜nones (2006) studied how test participants from different cultures behaved in a structured interview setting in which the participant’s task was to comment on a website. Indian participants found more usability problems and made more sugges- tions with an Indian interviewer than with an Anglo- American interviewer, but the comments they made to the Anglo-American interviewer tended to be more posi- tive than negative. With an Anglo-American conducting the interview, Indians also were reluctant to discuss culture-related problems with the website and kept their comments quite general. The participants were more detailed and candid with the Indian interviewer. Yam- miyavar et al. (2008) found that when subjects were paired with evaluators from the same culture, they used more head and hand gestures to communicate than if the evaluator was from a different culture, providing a richer source of nonverbal data to analyze. Sun and Shi (2007) studied how using one’s primary versus sec- ondary language (English versus Chinese in this case) in a think-aloud test affected the process of the test.
Chinese evaluators speaking Chinese to Chinese test users gave more help to users and a more complete introduction to the product being tested and encouraged
users more frequently than Chinese evaluators speaking English during the test.
A second good practice is to avoid pairing evaluators and test users who differ in their perceived status or authority. Particularly in cultures with high power distance, such as China and Malaysia, the behavior of both the test user and the test evaluator is affected by perceived differences in status or authority. Participants in high-power-distance cultures do not challenge or question the evaluator because of the perception of the evaluator as a person of authority (Burmeister, 2000).
Yeo (1998) illustrated this with an example of a usability test conducted in Singapore in which a participant broke down and cried from frustration. A posttest interview revealed that the participant’s behavior was in part due to the Eastern culture, in which it is not acceptable to criticize the designer openly, because it may cause the designer to lose face. Evers (2002) evaluated cultural differences in the understanding of a virtual campus website across four culturally different user groups (England, North America, the Netherlands and Japan) by using the same methods for each group. The results indicated that Japanese participants who were secondary school students felt uncomfortable speaking out loud about their thoughts and seemed to feel insecure because they could not confer with others to reach a common opinion.
But the effect of culture can go both ways. In a pilot study of think-aloud tests, Sun and Shi (2007) found that evaluator’s behavior is also affected by differences in level of perceived authority. When the evaluator’s academic title or rank was higher than that of the users, the evaluator tended to more frequently ask the user what he or she was thinking during the test. The evaluator also tended not to provide the user with more detailed instructions during the test.
The third guideline is to train evaluators to combat the “conversational indirectness” of Asian users. Sub- jects from Asian cultures will tend to seek a compromise and be indirect when evaluating user interfaces. For example, Herman (1996) studied cultural effects on the reliability of objective and subjective usability evalua- tion. The results of objective and subjective evaluation correlated poorly in Herman’s study. The Asian par- ticipants were less vocal, very polite, and not inclined to express negative comments in front of observers, so that the results of subjective evaluation tended toward the positive despite clear indication of poor user perfor- mance. Herman’s solution was to invite test participants to work in pairs to evaluate the interface and make the usability test more of a peer discussion session.
Shi (2008) conducted observations of usability tests in China, India, and Denmark and, like Herman, also noted that Chinese users often kept silent and did not speak out actively, particularly in formative evaluations. Two studies (Shi, 2008; Clemmensen et al., 2009) explained this observation in terms of Nisbett’s (2003) cultural theory of Eastern and Western cognition.
Accordingly, Chinese people tend to have a holistic process for thinking as opposed to the more analytic style of Westerners. Holistic thought is not as readily verbalized as analytic thought. So, they theorized that
Chinese users in a think-aloud test situation are thinking about the user interface in holistic terms and simply have a more difficult time putting those holistic thoughts into spoken words.
Shi recommended that evaluators receive special training to conduct testing with Chinese users that is based on think-aloud methods. Evaluators should be trained to use reminders and questions, “digging deeper probes,” to get the users to talk. For example, evaluators in Shi’s study reported that if they knew users were looking for some object or feature on the screen, they would ask, “what are you looking for?”
Then, immediately, the user would tell them about what they were looking for. This method of asking related questions to encourage speaking aloud was found to be more natural than just asking people to “keep talking.”
Evaluators also should be trained to be patient with Eastern participants as they pause and think in between verbalizations (Shi, 2009). All of the above suggestions require a significant amount of training and skill on the part of the evaluator. If such resources are not available to support the test, then perhaps the best approach is to consider an alternative to the think-aloud method for conducting the evaluation.
Shi (2009) found no significant differences in the set of usability problems reported by Chinese and Danish evaluators. However, their ratings of the severity of the usability problems did differ significantly. Chinese evaluators rated problems less severely than Danish evaluators and often rated problems in the middle of the five-point severity scale. Shi (2009) suggests that if problem severity rating is part of the test, then a four- point, rather than five-point, scale should be used to prevent middle-of-the-scale ratings.
Participant and Evaluator Recruiting Recruiting participants with similar background in different places at the same time has made international usability eval- uation very difficult. The experimenters have options to carry out international usability evaluation such as going to the foreign country, running the test remotely, hiring a local usability consultant, or asking help from staff in a local branch office (Nielsen, 1990, 2003). Many researchers (Choong and Salvendy, 1998, 1999; Dong and Salvendy, 1999; Fang and Rau 2003; Fukuoka et al., 1999; Prabhu and Harel, 1999; Evers, 2002) chose to recruit participants in two or more countries by going to the countries. The Web has made conducting inter- national usability evaluation a new option. Lee and Harada (1999) conducted an evaluation by recruiting participants on the Web. However, for countries with no experimenters present, they found it difficult to recruit participants on the Web.
Clemmensen et al. (2007) address the problem of
“hidden user groups,” groups of people who represent significantly different target user segments within the same culture. They suggest that test planners attempt to balance out potential hidden user groups within user segments. For example, users who are accustomed to foreigners and adapt quickly to international test conditions should be balanced by users who are not accustomed to foreigners. Traditional and culturally
CROSS-CULTURAL DESIGN 173 sensitive users such as those one might find in rural areas
need to be balanced in the pool of test participants by more modern, urbanized people who are less influenced by a country’s local culture. To avoid missing critical usability problems during the test, they recommend that evaluators be chosen carefully from evaluator groups suitable to the members of the identified hidden user groups.
Language
Verbal Language-Using Interpreters and Trans- lators Language is a significant factor in international usability evaluation. Without translated content, the tar- get audience may be limited to Web users with a certain education and social background. Nielsen (2003) indi- cated that the first two concerns for international usabil- ity testing are displaying in the user’s native language, character set, and notations and translating the user inter- face documentation into the user’s native language. Rau and Liang (2003b) conducted a card-sorting test with Chinese users in Taiwan for an international website.
Even though all the information items were translated into Chinese, a dictionary and instructions also were made available to participants. Also, anything offensive in the user’s cultural background should be avoided. The testing materials and procedure should accommodate the way business is conducted and the way people commu- nicate in various countries (Nielsen 2003). Interpreters are necessary if the evaluators are not able to speak the user’s native language well. The evaluators need to avoid any important information lost in translation so that videotaping and audio recording are useful for further analysis. Nielsen (2003) suggests meeting inter- preters beforehand and reminding them that they should not help users during the test.
Nonverbal Language Non verbal language dis- played in the form of hand and head gestures is com- monly used by test participants as an occasional substi- tute for verbal language or to elaborate on or supplement it. Hand and head gestures also often communicate the comfort level of the participant and his or her readiness to communicate with the evaluator. Observing and ana- lyzing gestures during a test can provide a rich source of data that adds context to the verbal data being recorded.
Yammiyavar et al. (2008) questioned whether users from different cultures exhibited similar or different patterns of nonverbal communication, including the type, frequency, and usage of gestures. They analyzed the occurrence of gestures in video recordings of think- aloud tests that used subjects from Denmark, India, and China. Gestures were grouped into four types (Ekman and Friesen 1969): emblems, illustrators, adapters, and regulators. Emblems replace words with gesture-based signs like nods of head for “yes” or a V sign for victory. They tend to be culture-specific. Illustrators include such actions as banging on the table or sketching shapes in the air. They help the subject to verbalize their thoughts. Yammiyavar et al. (2008) discovered that these can be quite important markers for usability problems because illustrator gestures frequently precede
the verbalization of a usability problem to the evaluator.
Adapter gestures are actions of the body that convey feeling of pressure or discomfort, for example, cracking one’s knuckles, tapping one’s feet, stroking hair or chin while in deep contemplation, and “squirming” in one’s seat. They indicate the subject’s comfort level with the test situation. Finally, subjects control the flow of conversation with the evaluator by using regulator behaviors such as nodding the head up and down to indicate agreement.
Yammiyavar et al. (2008) found that the frequency of using gestures during the test was not significantly dif- ferent across the three cultural groups. This contradicts the popular belief that Indians, for instance, use more gestures to communicate than other cultural groups.
Also, regulator gestures appeared to be used similarly across the cultures studied, but with some tendency for Chinese to use them the least. In contrast, there were significant differences across the cultural groups in the specific emblem gestures used to replace words and in adapter behaviors. Certain illustrators appeared to be culture specific as well. The researchers concluded that there is a need to benchmark gestures used in these different cultures and their meanings and then provide those to usability test evaluators to guide observations and understand what they observe.
The facial expression associated with surprise often is used as a marker by evaluators to indicate that a usability problem has been detected by the user.
Clemmensen et al. (2009) have questioned the validity of this practice in cross-cultural usability tests. From Nisbett’s (2003) theory of cultural cognition, they hypothesize that Easterners will experience less surprise than Westerners when presented with inconsistencies in user interfaces. With their logical, analytic orientation, Westerners tend to focus on fewer causes of observed events, while Easterners, with their holistic orientation, tend to consider more causal factors as well as the context of the event. As Clemmensen et al. (2009) point out, this makes it easier for them to identify a rationale for why the event occurred in the way it did, resulting in less surprise.
Instructions, Tasks, and Scenarios Instructions to the test participant can vary significantly in how much contextual information they provide. At the one extreme, instructions are strictly focused on the task to be performed with the application being tested.
No explanation is given about the purpose of the application, when you might use it, or why. At the other extreme, the explanation of the task is embedded in a rich context provided by a real-life scenario.
Clemmensen et al. (2009) suggest that Westerners, with their tendency to focus on the central elements presented, such as the details of the task, will be able to obtain that in either presentation of instructions. East- erners, however, will find that the stark instructions are insufficient. With their holistic style of thinking, they will prefer to have the task explanation embedded in the context of a real-life scenario. The recommendation from Clemmensen et al. (2009) is that if cross-cultural testing is to be conducted; then the test planner should