Designing A Mixed Methods Approach in
Communication Research
2
Criteria for success
MBA
Creation of new knowledge
Understanding of existing knowledge
Academic Focus Managerial Focus
PhD
MSc MBA
DBA
MBA
▪
Academic quality, standards and originality of the thesis
▪
Evidence of adequate understanding of existing knowledge
▪
The extent of contribution to applied/practitioner research
▪
The quality of fieldwork
▪
Presentation, methodical exposition and demonstration, relevance and coherence of argument, effectiveness of style.
▪
for PhD/DBA/EngD candidates, the extent of original contribution to knowledge.
▪
The implications of the research.
4
Domains of
Contribution Extent of Contribution
What has been confirmed
What has been developed
What has been found which is brand
new Theoretical
knowledge Empirical evidence
Methodological approaches
Knowledge of practice
The insufficient argument – either quantitative or qualitative may be insufficient by itself
Multiple angles argument – quantitative and qualitative approaches provide different “pictures”
The more-evidence-the-better argument – combined quantitative and qualitative provides more evidence
Community of practice argument – mixed methods may be the preferred approach within a scholarly community
Eager-to-learn argument – it is the latest methodology
“Its intuitive” argument – it mirrors “real life”
6
• Mixed methods research is both a method and methodology for conducting research that involves collecting, analyzing, and integrating quantitative and qualitative research in a single study or a longitudinal program of inquiry.
• The purpose of this form of research is that both
qualitative and quantitative research, in combination,
provide a better understanding of a research problem
or issue than either research approach alone.
Qualitative Text Data
This is a sample of a text file of words that might be
collected on interview transcripts, observation fieldnotes, or optically- scanned documents.
Quantitative Numeric Data
2 3 4 2 5 2 3 1 1 2 3 4 2 3 3 2 1 1 1 5 3 4 1 2 3 1 4 4 5 5 4 1 2 1 4 3 3 5 1 4 2 3 1 5 5 2 2 1 5 3 5 1 3 1 5 3 2 2 5 1 3 2 4 4 3 1 2 4 2 2 4 1 5 5 4 2 1 5
Collects both quantitative and qualitative data
“Mixes” them
“Mixes” them at the same time (concurrently) or one after the other (sequentially)
Emphasizes both equally or unequally
Quantitative data
Instruments
Checklists
Records
Qualitative data
Interviews
Observations
Documents
Audio-visual materials
10
Qualitative analysis
Use text and images,
For coding
For theme development
For relating themes
Quantitative analysis
Use statistical analysis,
For description
For comparing groups
For relating variables
Results Connect data:
Results
Converge data:
Embed the data:
Quan data
Qual data
Qual Quan
Qual Quan
To compare results from quantitative and qualitative research
To use qualitative research to help explain quantitative findings
To explore using qualitative research and then to generalize findings to a large population using
quantitative research
To develop an instrument because none are available or useful
To augment an experiment with qualitative data
Procedures for handling your qualitative and quantitative data
Sequence – concurrent or sequential or both
Emphasis – emphasis on qualitative or quantitative
Sometimes both concurrent and sequential phases are used
Designs may include more than two phases
Think about using a simple, elegant design
14
QUAN Data &
Results Interpretation
QUAL Data &
Results
Pre-test QUAN Data &
Results
Post-test QUAN Data &
Results
Intervention
qual Process
Interpretation
Triangulation Design
Embedded Design
Concurrent Mixed Methods Designs
Sequential designs
Data & QUAN Results
Interpretation
Data & qual Results
Following up
Data & QUAL Results
Data & quan
Results Interpretation
Building to
Before- intervention
qual QUAN
Intervention
After- intervention
qual Interpretation
Exploratory Design Explanatory Design
Sequential Embedded Design
16
Design Name Equal priority QUAN emphasis QUAL emphasis Concurrent,
triangulation QUAL+QUAN QUAN+qual QUAL+quan
Concurrent,
embedded n/a QUAN(qual) QUAL(quan)
Explanatory,
sequential, quan first QUAN→QUAL QUAN→qual quan→QUAL Exploratory
sequential, qual first QUAL→QUAN qual→QUAN QUAL→quan
Sequential, embedded n/a (qual) → QUAN
QUAN →(qual) (quan) → QUAL QUAL → (quan)
Quantitative Data*
Number of cigarettes CES-D6
Qualitative Data*
Semi-structured interviews, audio recorded and transcribed
* Data collected 10 times over the course of a calendar year for 40 participants
Quantitative Analysis Graphic plot of CES- D6 scores over time for each participant Graphic plot of cigarettes/day values over time for each participant
Case Selection Qualitative Analysis Description of each case
Identification of life events occurring during critical months where smoking increased or decreased
Thematic analysis of life events for each case
Cross-case thematic analysis
Interpretation Why did changes in
smoking occur?
Quantitative Data Collection (quan)
Qualitative Data Collection
Interpretation – based on quan ad QUAL results Qualitative
Data Analysis (QUAL) Case Selection
Quantitative Data Analysis
Selected 5 cases maximally varying Identified critical months in which smoking varied
Source: Creswell, Plano Clark, Shope, McVea +
Phase I Qualitative Research - Year 1
Qualitative Data Collection Unstructured Interviews - 50 participants
8 observations at the site 16 documents
Qualitative Data Analysis Text Analysis: Using QSR N6
Qualitative Findings Development of codes and themes for each site
Phase II Quantitative Research - Year 2
Quantitative Instrument Development
Quantitative Test of the Instrument
Create approximately a 80-item instrument plus demographics Administer survey to 500 individuals Determine factor structure of items and conduct reliability analysis for scales Quantitative Results
Determine how groups differ using ANOVA test
20
Process – collection and analysis of qualitative
data
(before, during, after trial)
Experiment
Intervention
QUAN
Data collection Pre-test
QUAN
Data collection Post-test
Type of
Mixing Type of Design Why Mixing
Occurs Where Mixing Occurs in Research Process Connecting Sequential One phase builds
on the other Between data analysis (Phase 1) and data collection (Phase 2) Merging Concurrent Bring results
together After analysis of both quan and qual –
typically in discussion Embedding Sequential or
Concurrent Either building or bringing results together
Either between phases or in discussion after analysis
22
Literature
review Qualitative study (12 interviews)
First draft questionnaire
Content and face validity (10 expert opinions)
Second draft questionnaire
A survey of 35 Corporate Communication
practitioners
Reliability analysis
Final questionnaire
A main survey will distributed to 960
Corporate Communication
practitioners Delete low CVI
score (<.80) Delete low item-to-total correlation item (<.50) STAGE 1: Item
creation STAGE 2:
Interview STAGE 3: Content
validity STAGE 4: Instrument
testing
Instrument development process
24
Stage 1: Item creation
to create a pool of items by identifying the item from existing scales.
existing scales are reviewed and related domains and items are extracted from various related journals
all items identified in the existing instruments were
categorized according to the various constructs which they were originally intended to address.
this generated an initial pool of items for every constructs
item considered to be too narrow in focus and applicable only in particular situations or to particular industries was culled
Stage 2: Interview
Face-to-face interviews were conducted in Malaysia in March 2010
12 from 25 selected corporate communication managers participated and agreed to be interviewed
9 corporate communication managers of public listed companies and 3 corporate communication consultants.
interview protocol and procedures were used throughout the data collection phase – interview guide
Full interview transcriptions
qualitative data were classified through deductive content
analysis based on key words or themes (Hinkin, 1995; Krippendorff,
26
Stage 2: Interview
Stage 2: Interview
Bil Position Institution Duration
(minutes)
Interview Date 1 General Manager Corporate
Communications
Company 1 (Trading/Service) 60 11 March 2010
2 Head, Regional Communications Company 2 (Trading/Service) 60 15 March 2010
3 General Manager, Corporate Affair Division
Company 3 (Trading /Service) 60 15 March 2010
4 Public Relations Manager Company 4 (Trading /Service) 60 16 March 2010
5 Group Advisor Company 5 (Corporate
Communication Consultant)
60 17 March 2010
6 Managing Director Company 6 (Corporate
Communication Consultant)
60 22 March 2010
7 Head, Corporate Communications Company 7 (Trading /Service) 60 23 March 2010
8 Director Company 8 (Corporate
Communication Consultant)
60 25 March 2010
9 General Manager, Corporate Communications Division
Company 9 (Trading/Service) 60 26 March 2010
10 Head of Communications Company 10 (Infrastructure Project Companies)
60 29 March 2010
11 Senior Manager, Corporate Communications
Company 11 (Trading/Service) 60 30 March 2010
12 Manager, Corporate Company 12 (Industrial 60 30 March 2010
28
Stage 3: Content validity
The goals of this stage were twofold:
➢ to assess the content validity of various scales being developed
➢ to attempt to identify any items which still may have been unclear.
the content and the clarity of scale based on procedure suggested by Rubio et al. (2003).
the response forms were sent to 10 practitioners and experts in corporate communication, PR and marketing.
The content experts are professionals who have published or worked in the field (Rubio et al., 2003).
Stage 3: Content validity
Each item is rated on a scale from 1 to 4.
Representativeness is demonstrated by an item’s ability to represent the content domain
The clarity is evaluated on the basis of clearly an item is worded.
The Content Validity Index (CVI) is calculated based on the representativeness of measure.
All the scales have shown CVI scored between .70 to 1.00.
for the new measures, the CVI scored at least .80 needed (Davis (1992)
all the items have CVI score less than .80 was culled from the list.
30
Stage 3: Content validity
Stage 3: Content validity
Constructs Items dropped Change sentence structure
Original item from literature
& interview
Number of items dropped
New number of items for
pilot test Corporate
communication management (CCM)
A1, A2, A6, A9, A12, A15, A17, A20, A25, A33
A5, A23, A24 33 10 23
Corporate culture B15, B21, B22, B23, B24
B5 24 5 19
ICT Diffusion Innovation
C1, C6, C9, C14, C17, C19, C22, C24
C2, C3 24 8 16
Corporate leadership
D13 Nil 19 1 18
Mission achievement Nil Nil 20 0 20
32
Stage 4: Instrument testing
The online survey started on 25 September 2010 for one month.
35 out of 100 respondents from Corporate Communication Research (CCR) group in Facebook had replied to the survey.
English language questionnaires were used instead of other local languages on Malaysian subjects (Schumaker and Barraclough, 1989).
Malaysians, especially those involved in the business sector, are assumed to be proficient in the English language (Lim, 2001).
The data obtain from the pre-test will subjected to a further purification process - Reliability Analysis
Stage 4: Instrument testing
PILOT TEST
Scale Name ITEMS ALPHA GLB Number of
items dropped
New number of items for main
survey Corporate
Communication Management (CCM)
23 0.922 0.878 5 18
Corporate Culture 19 0.883 0.641 3 16
ICT Diffusion Innovation
16 0.952 0.915 1 15
Corporate Leadership 18 0.956 0.933 3 16
Mission Achievement 20 0.965 0.924 2 18
Notes ITEMS: Number of items in each scale. APHA: Crobach’s reliability coefficient.
GLB: Guttman’s Lower Bound for reliability
34 34
Field testing and data collection
1,020 corporate communication and public relations managers of public listed companies were included in this research
The online survey questionnaires used Survey Monkey, prominent software in the market.
the requirement about the sample size by chosen analysis techniques (EFA) requires a considerable sample size in order to obtain reliable estimates (Joreskog and Sorbom, 1996).
In total, 223 usable surveys were entered into the analysis, producing an effective response rate of 23 percent.
Exploratory Factor Analysis (EFA)
Exploratory factor analysis is conducted to investigate the
factorial structure of the scales to identify groups of variables
(Field, 2005)
the data analysis for exploratory factor analysis uses
orthogonal and oblique rotation (Field, 2009; Hair et al., 2010;
Tabachnick and Fidell, 2007).
Rotation is essential to enhance the interpretability and
scientific utility of the solution which is to simplify and clarify the data structure.
The purpose of this analysis is to maximise high correlations between factors and variables and minimise those which are low. Moreover, a rotation technique is very useful compared to others to develop factors from variables
36 36
Exploratory Factor Analysis (EFA)
Items wording Items codes
Corporate Advertising (Alpha = .898)
• We manage the advertisements, marketing and promotion (AMP) programme for the company’s brand positioning.
CCM03 .934
• We are responsible to design all corporate visual identity (e.g. logo, slogan or typography) for our company.
CCM04 .929
• We developed our own content for corporate advertising and other advertisement materials.
CCM05 .870
Public Relations (Alpha = .941)
• We are responsible to produce publications for the company (e.g. newsletter, company profile and corporate brochures).
CCM11 .769
• We organise the Corporate Social Responsibilities (e.g. sponsorship, charities and education) programme for stakeholders’ engagement and participation.
CCM13 .773
• We control the content of electronic publication (e.g. e-newsletter, electronic press releases and corporate website) for our company.
CCM14 .823
• We organise external events (e.g. awareness campaign, trade exhibition and product launch) to keep board members and employees engaged with external stakeholders.
CCM16 .865
• Our works includes speech writing, translations and a documentation process for various important documents.
CCM18 .819
Exploratory Factor Analysis (EFA)
Items wording Items codes
Media Relations (Alpha = .858)
• We organise media programmes (e.g. media night or golf with media) to build a relationship with the media.
CCM19 .917
• We are responsible to address media needs and concerns professionally at any time. CCM21 .863
• We are responsible for press release and media conferences if necessary. CCM23 .868 Investor Relations (Alpha = .944)
• We work with the company secretary to organise the annual general meeting where the company will announce the annual financial result to the shareholders.
CCM24 .921
• We work closely with the finance department to prepare the annual report and quarterly financial report.
CCM26 .919
• We provide information and manage relationship with investors through various programmes (e.g. investor relations programme and annual general meeting).
CCM27 .945
• We create a positive context and better perspective on financial data to show the positive performance of the company.
CCM28 .920
38 38
Exploratory Factor Analysis (EFA)
Items wording Items codes
Employee Relations (Alpha = .873)
• We provide training (e.g. communicate to the public and understand the public needs) at lower levels up to highest level to build internal branding for the company.
CCM29 .875
• We help the company create open communications within the organisation during a particular crisis (for example, sales drop, economic down turn or merger and
acquisition) .
CCM30 .899
• We assist HR to explain key initiative the company is taking and manage change on regular basis through dialogue in the organisation.
CCM31 .905
Literature review
Qualitative study (12 interviews)
First draft questionnaire
Content and face validity (10 expert opinions)
Second draft questionnaire
A survey of 35 Corporate Communication
practitioners
Reliability analysis
Final questionnaire
A survey of 223 Corporate Communication
practitioners
Delete low CVI score (<.80)
Delete low item-to-total correlation item (<.50)
Item generation Pilot Study Main Survey
Research Design Process
Exploratory Factor Analysis (EFA)
Reliability Analysis
Confirmatory Factor Analysis (CFA)
Delete low item-to-total correlation item (<.30)
Test measurement model
40
Conclusion
• creation of an overall instrument to measure corporate communication management practice in Malaysia.
• the creation process included surveying known existing instruments, choosing appropriate items, creating new items as necessary, and then
undertaking an extensive scale development process.
• the method of developing the scales provides a high degree of confidence in their content and constructs validity.
