On completion of the pilot study and the creation of a robust generic skills coding scheme, some issues remained are required refinement. These are described below. A new generic skills coding scheme was created and this was then used to examine the remaining video data.
7.3.1 Experimental Participants
All participants were designers working in architectural practices. Ideally participants should have been recruited from a diverse range of disciplines (for example,
structural engineering, construction or other similar professions) but engaging practitioners from more that one profession proved challenging. To attract sufficient participants it was necessary to recruit from our CRC-CI industrial partners as well as through the University of Sydney.
Five design teams were assembled, each consisting of two designers. Participants were grouped in pairs based on their availability, with preference being given to those who had worked together before. Participants were predominantly male (with only one female taking part) and had varying degrees of experience and influence (ranging from top management to junior staff).
7.3.2 Participants Methodology
Designers interacting in the co-located condition were familiar and comfortable with their surroundings and the collaboration techniques they used. However, in the second and third stages participants needed to be trained in the use of the new collaborative software. Before each session the research team spent 1-2 hours familiarizing designers with the technology. Participants were then allowed to participate in the design sessions.
Each designer received a brief for the architectural aspects of a project and instructions about their role in the design process. All projects were fictional and related on an actual site at Sydney University. Three different design briefs were provided. They were for the structural design of an art gallery, a library and a dance school. The collaborators were given 30 minutes to prepare each design using the different collaboration technology. The outline for the presentation of these briefs and schedule is presented in Table 7.3.
Table 7.3 Table showing schedule for experimental video recording of design collaboration. Including time length, interaction and design tools, and the design brief to be followed.
Experimental Stages Length Tools Design Brief Face-to-Face
(Co-located) Experiment - Stage 1
30 min Paper/pencil Design of an Art Gallery Shared E-Whiteboard
(Group Board) Experiment - Stage 2
30 min Group Board on
Smart Board Design of a Library 3D Virtual World (Active
Worlds) Experiment - Stage 3
30 min Active Worlds on desktop
Design of a Dance School
7.3.3 Experimental Coding Scheme
Once the design sessions were recorded, the data were analysed using the generic skills coding scheme. A pilot generic skills coding scheme was based on human factors research, specifically the area of behavioural markers. Behavioural markers are observable non-technical (Klampfer et al. 2001) ‘…aspects of individual and team performance’ (Carthey et al, 2003: p, 411) which are related to the effectiveness of an individual and a team. Behavioural markers, or more specifically the methods for the creation of behavioural markers, were analysed because they offer a physical description of non-technical skills [generic skills] (Kjellberg et al. 2003). Klampfer et al (2001) enforce the need for simple and clear behavioural markers, which use appropriate jargon and terminology. They argue that emphasis should be placed on an observable behaviour rather than an ambiguous attitude or opinion. So if generic skills behavioural markers from other industries, such as aviation and medicine, are used to identify observable behaviours in design terminology, they will need to be adapted to be relevant for design team interactions. This adaptation means altering discipline specific terminology or other behaviours.
Further research into behavioural markers revealed that the pilot Generic Skills coding scheme could be simplified. The Anaesthetists’ Non-Technical Skills (ANTS) (Fletcher et al. 2004) system was adapted for this purpose (see Table 7.4) . The nature of generic or non-technical skills implies that they are transportable between teams, as they are not directly involved in the technical aspects of the process.
mentioned to be merged, and resulted in a simpler coding scheme. Furthermore decision-making and communication are accounted for or incorporated within the ANTS scheme.
While effective communication would appear to be an essential component of team interaction, it is impossible to separate it from the observable behaviours associated with skills. Behavioural marker research indicates that communication is such an essential part of these non-technical skills that the nature of the analysis dictates that communication cannot be separated from these skills (Fletcher et al. 2004).
Communication was thus excluded from the generic skills coding scheme as it is inherent in every other skill.
The skills included in the original and revised generic skills coding schemes were derived from behavioural marker research (Klampfer et al. 2001), so the transition and adaptation of these skills is well founded. Behavioural marker systems provide a rating system where the actual effectiveness of the skills is examined. However the requirements of this study are different and aim to detect whether the level of skill usage changes depending on the virtuality of the team. Searching for the presence and frequency of these skills is appropriate, and results in a set of behaviours
recording effective skills usage. Interestingly, investigations in the design profession (Lewis and Bonollo 2002) into generic (or professional) skills showed that the skills identified related closely to the new generic skills coding system.
Table 7.4 Non-technical skills as presented by the ANTS system (Fletcher et al. 2004) and their adapted definitions.
Non-Technical Skill
Definitions
Task
Management
“Managing resources and organising tasks to achieve goals, be they individual or team related”
Team Working “Working with others in a team context, in any role, to ensure effective joint task completion and team satisfaction; focus is particularly on the team rather than the task”
Shared Situational Awareness
“Developing and maintaining an overall dynamic awareness of the situation based on perceiving the elements of the…
environment”
Decision Making “Making decisions to reach a judgement or diagnosis about a situation, or to select a course of action, based on experience or new information under both normal conditions and in time- pressured crisis situations”
Table 7.5 illustrates how the new skills are composites of the original skills, and also how the professional skills identified by Lewis and Bonollo (2002) were taken into consideration.
Table 7.5 Table showing the relationship between the ANTS generic skills, Salas’s (2000) generic skills and Lewis and Bonollo’s professional design skills.
ANTS Non-Technical Skills (Fletcher et al.
2004)
Generic Skills found in the Literature (Salas, Burke and Cannon-Bowers 2000)
Professional Design Skills (Lewis and Bonollo 2002) Task Management Leadership/Team
Management
Project Management Performance Monitoring and
Feedback
Acceptance of Responsibility Team Working Interpersonal Relations Interpersonal Skills Co-ordination
Situational Awareness
Adaptability Negotiation Decision Making Decision Making Problem Solving
Full descriptions and observable behaviours for the adapted ANTS coding scheme (Final Generic Skills Coding Scheme) can be viewed in Table 13.3.
7.3.4 Reliability Testing
Intra-rater reliability was established for the generic skills coding scheme on a 30- minute face-to-face session using Noldus Observer Pro. Point-by-point agreement was 81% and 80% on the frequency of coding strings and frequency and sequence of the coding strings, respectively. These were both at or above the minimum level of acceptability of 80% (Kazdin 1982b). Detailed reliability analysis of this session can be found in Appendix 18.