How to Design a Mixed

Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark,
M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004

How would you combine two

types of data?
Qualitative
Text Data
This is a sample of a
text file of words that
might be collected on
transcripts through
interviews, fieldnotes
from observations, or
from opticallyscanned documents.

Quantitative
Numeric Data
2342543112232132
23322543
3122432432132433
32334441
2222111432143213
22111555
2331432432132433
32135432

Objectives of the
workshop:

Lets design a mixed methods study

Lets study how people learn mixed
methods research in this room?
(or you can work on your own project
and follow along at each step)
Lets start with a title. Write a title.
What data will we collect?

What are types of quantitative and

qualitative data?

Quantitative data
Close-ended
scales
Attitudinal/behavi
oral scales
Behavioral
checklists
Census,
attendance
records

Qualitative data
Open-ended
responses
Semi-structured
interviews
Semi-structured
observations
Records/document
s
Videotapes
5

Lets identify our quantitative and

qualitative data collection

Now lets consider some reasons

for why we are collecting (and
mixing) both forms of data

Together quantitative and qualitative data provide both

precise measurement and generalizability of
quantitative research and the in-depth, complex picture
of qualitative research
To validate quantitative results with qualitative data
We do not have an adequate instrument. Thus, we need
to explore views and develop an instrument
Our quantitative data provide a general explanation and
we need to follow-up with participants and have them
explain the quantitative results
In our experiment, outcomes to be measured are not
enough; they need to be complemented by
understanding the process of participants

Lets identify our reason for mixing

So

There are good reasons for

gathering both forms of data
Butthere are certain
requirements for this to work best

Requirement #1: Now lets

consider whether we have the
skills, time, and resources?

We need minimum skills in both

qualitative and quantitative data
collection. What do we need?
We need time and resources for
extensive data collection and
analysis. How much time and
resources do we need?
10

Write down the skills, time, and

resources we will need

11

Requirement #2: The

audience(s)

Does our audience appreciate both

numbers and stories?
Are they familiar with this design?
Do they need to be educated?
Are examples of published studies
available in our content area?

12

Lets identify the audiences

13

But audiences may not

recognize it yet because it is
so new

Increased use and acceptance of

qualitative research from 1990s to
present
The complexity of our research problems
today requires understanding trends,
differences, as well as individual stories,
setting
Individuals advocating for and writing
about mixed methods research as a
distinct, new procedure (e.g., books)
14

They may think that it is

analyzing data separately
Quantitative Data

Qualitative Data

Mixing: converging the data or connecting

the data
15

But how do we mix?

Converge data:
Qual

Results

Quan

Connect data:
Qual

Quan

Results

16

Why our audience may

recognize it

The evidence

Books
Methodological articles
Many published research studies
using it
Federal agencies
Private foundations
17

18

Other writings, initiatives

on mixed methods
research:
Research studies reported in journals

Methodological articles exploring issues

and procedures
Website for bringing mixed methods
writers together
Conference sessions
Handbook of Mixed Methods in Social
and Behavioral Research
Private foundation interest; federal
agency interest
19

NIH Guidelines
- Mentioned several
approaches for combining
qualitative and
quantitative research

- Considerations for deciding

what model to use (literatur
available, prior studies,
realistic design, expertise)
- Need to describe each
method thoroughly

20

Quotes:

Combining qualitative and quantitative

methods has gained broad appeal in
public health research. The key question
has become not whether it is acceptable
or legitimate to combine methods, but
rather how they will be combined to be
mutually supportive and how findings
achieved through different methods will
be integrated. (NIH, Office of Behavioral
and Social Science Research, 1999).
21

National Academy of
Sciences
Three major research questions in
quality educational research:

What is happening? (qualitative designs

Is there a systematic effect?
(a quantitative experiment)
Why or how it is happening?
(a qualitative followup)

22

But even if they recognize it, they

may not appreciate or understand
how to design a mixed methods
study

We are interested in a randomized

control trial with a non-experimental
approach embedded within it. (a
private foundation officer)
We accept multi-method studies, but
investigators mostly do not sort out the
complexity of these projects so that we
can understand them. (a federal
projects officer).
23

We need to define mixed

methods research for our
audiences

Mixed methods research is a design for

collecting, analyzing, and mixing both
quantitative and qualitative data in a
single study or series of studies to
understand a research problem.
The purpose of this form of research is
that both qualitative and quantitative
methods, in combination, provide a better
understanding of a research problem or
issue than either method alone.
24

Now we could mix within single

studies or multiple studies
Single Study:
Quan

Qual

Results

Multiple Studies (called multimethod researc

Quan

Qual

Qual

Quan

Study 1

Study 2

Study 3

Study 4
25

So how do we design a
mixed methods study?
The
model
Worldviews, theoretical frameworks, problem and research
question, skills, resources
Type of mixed methods design
Procedures for:
designing the title
writing the introduction to a study
writing the purpose statement and research
questions/hypotheses
data collection
data analysis
writing the mixed methods report
evaluating the mixed methods research
26

What is a worldview?

Philosophy about your preferences for how

you learn about something through research
You prefer the quantitative worldview:
you are the expert, you decide what needs
to be learned, you build in objectivity
You prefer the qualitative worldview:
participant is the expert, participant helps
you build knowledge, you bring personal
bias in
You prefer both the quantitative and
qualitative worldview
27

The next steps in planning

our study

Lets write the overall research

question for our study

28

Then lets choose a type of

mixed methods study to
conduct

What designs are possible?

29

Types of mixed methods

designs
I. Triangulation Mixed Methods Design
QUAN
Data and Results

QUAL
Data and Results

Interpretation

II. Nested Mixed Methods Design

QUAN
Pre-test
Data and Results

Qual Process

QUAN
Post-test
Data and Results
30

Types of mixed methods

designs
III. Explanatory Mixed Methods Design
QUAN
Data and Results

Follow-up

qual
Data and
Results

IV. Exploratory Mixed Methods Design

QUAL
Data and Results

Building

quan
Data and Results
31

Triangulation Design:
Characteristics

Collecting both quantitative and qualitative data

Collecting these data at the same time in the
research procedure
Analyzing the quantitative and qualitative data
separately
Comparing or combining the results of the
quantitative and qualitative analysis
Example: collect survey data (quantitative) and
collect individual interviews (qualitative) and
then compare the results

32

Triangulation Design:
When is it used?

When you want to combine the

advantages of quantitative (trends, large
numbers, generalization) with qualitative
(detail, small numbers, in-depth)
When you want to validate your
quantitative findings with qualitative data
When you want to expand your
quantitative findings with some openended qualitative data (e.g., survey with
closed- and open-ended data)
33

Nested Design:
Characteristics

Collecting both quantitative and qualitative data

Collecting both types of data at the same time
Having ONE form of data play a smaller role in the
study than the other form of data
Also,

Using one form of data to answer one question; the other

form another question
Collecting one form of data at one level of analysis and
another at another level of analysis

Example: You conduct an experiment and during

the experiment you gather qualitative interview
data. The outcomes of the experiment assessed
quantitatively address different questions than the
process of the experiment explored qualitatively. 35

Nested Design: When is it

used?

When you do not have time or

resources to commit to extensive
quantitative and qualitative data
collection
When you want to study the process
of an experiment as well as the
outcomes
When you want to examine different
levels in an organization
36

Nested Research Design

Experiment
Quan
Data collection
Pre-test

Intervention

Quan
Data collection
Post-test

Process collection
and analysis of qualitative
data

37

Explanatory Sequential
Design: Characteristics

Viewing the study as a two-phase project

Collecting quantitative data first followed
by collecting qualitative data second
Typically, a greater emphasis is placed on
the quantitative data in the study
Example: You first conduct a survey and
then follow up with a few individuals who
answered positively to the questions
through interviews
38

Explanatory Sequential
Design: When do you use
it?

When you want to explain the

quantitative results in more depth
with qualitative data (e.g., statistical
differences among groups, individuals
who scored at extreme levels)
When you want to identify
appropriate participants to study in
more depth qualitatively
39

Here is an example of an explanatory design:

Quantitative
Data Collection
(quan)

Quantitative
Data Analysis
(quan)

Case Selection

Qualitative
Data Analysis
(QUAL)

Interpretation
based on quan
and QUAL results

+
Qualitative
Data Collection

Quantitative Data* Quantitative Analysis

Case Selection
Number of cigarettes Graphic plot of-CES Selected 5 cases
D6 scores over time maximally varying
CES-D6
for each participant Identified critical
months in which
Qualitative Data*
Graphic plot of
cigarettes/day valuessmoking varied
Semi
-structured
over time for each
interviews, audio
participant
recorded and
transcribed
* Data collected 10 times
over the course of a
calendar year for 40
participants

Creswell et al. (in progress)

Qualitative Analysis

Interpretation

Descri
ption of each
Why did changes in
case
smoking occur?
Identification of life
events occurring
during critical
months where
smoking increased or
decreased
Thematic analysis of
life events for each
case
Cross
-case thematic
analysis

Exploratory Sequential
Design: Characteristics

Viewing the study as a two-phase project

Qualitative data collection precedes
quantitative data collection
Typically, greater emphasis is placed on
the qualitative data in the study
Example: You collect qualitative diary
entries, analyze the data for themes, and
then develop an instrument based on the
themes to measure attitudes on a
quantitative survey administered to a
large sample.
41

Exploratory Sequential
Design: When do you use
it?

To develop an instrument when

one is not available (first explore,
then develop instrument)
To develop a classification or
typology for testing
To identify the most important
variables to study quantitatively
when these variable are not known
42

Phase I Qualitative Research - Year 1

Qualitative Data Collection

Qualitative Data Analysis

Qualitative Findings

Phase II Quantitative Research - Year 2

Quantitative Instrument Development

Unstructured Interviews 50 participants

8 observations at the site
16 documents
Text Analysis: Using QSR N6
Development of codes and themes
for each site

Create approximately a 80-item

instrument plus demographics
Administer survey to 500 individuals

Quantitative Test of the Instrument

Quantitative Results

Determine factor structure of items and

conduct reliability analysis for scales
Determine how groups differ
using ANOVA test

Sequential Exploratory Mixed Methods Design

How will we analyze the quantitative

and qualitative data (within the
design types)?
Types of analysis:
Quantitative
analysis

Numeric data
Descriptive trend
analysis
Hypothesis testing,
effect size, interval
estimates

Qualitative
analysis

Text/image data
Coding
Themes
Description
Interrelated
themes
44

Triangulation data analysis

QUAN
data collection

Separate QUAN
and QUAL data
analysis

QUAL
data collection

QUAN
data analysis

Two options
Data transformation (change
QUAL to QUAN or QUAN to QUAL)
Comparison (keep separate and
compare/contrast)

QUAL
data analysis

Results

45

Table. Example of Data Transformation of Text Units into Numeric Data

Count
Adj.Count**
Row Pct
Column Pct

Patients
N=2

Physicians
N=4

Medical Assistants
N=4

Familiarity
With the
Form

13
6.5
41.9
5.8

17
4.25
27.4
3.0

19
4.75
30.7
5.3

49
15.5
100.0

Reactions to
the Form

23
11.5
22.2
10.2

100
25.0
48.3
17.4

61
15.3
29.5
16.9

184
51.75
100.0

Use for
Managing
Depression

67
33.5
38.6
29.8

177
44.25
51.0
30.7

36
9.0
10.4
10.0

280
86.75
100.0

Changes to
the Form

115
57.5
37.5
51.1

196
49.0
32.0
34.0

187
46.8
30.5
51.7

498
153.3
100.0

Situational
Use of the
Form

7
3.5
8.9
3.1

86
21.5
54.4
14.9

58
14.5
36.7
16.1

151
39.5
100.0

225
112.5
100.0

576
144.0
100.0

361
90.3
100.0

Themes

Nested data analysis

Quantitative Experiment
Quan
Data collection
Pre-test

Intervention

Quan
Data collection
Post-test

Qualitative Process
Data
Analysis

Pre-test
scores

Themes/Codes/
Interrelated Themes

Post-test scores
or gain scores

Compare/Describe Results
47

Explanatory sequential data

analysis
QUAN
data analysis

Qual
data collection
(purposeful sampling)

Qual
analysis

codes
Statistical results Select cases based on s.d. variables
themes
Select cases to represent outliers
Outlier cases
cases

Select
cases
to
represent
extreme
cases
Extreme cases
Select cases to make group comparisons

48

Exploratory sequential data

analysis
QUAL data
analysis

Quan data analysis

instrument development

Quotes

Items on a survey

Codes

Variables on a survey

Themes

Scales on a survey

49

Lets identify how we will analyze

the data

50

Drawing our Design

Lets draw a picture of our design

Identify the type of design
Add in data collection
Add in data analysis
Show the flow of activities
Add in products for our
audiences
51

Helpful tips for creating this visual:

1.
2.
3.

4.

5.
6.
7.
8.
9.

Give a title to the visual model.

Choose either horizontal or vertical layout for the model.
Draw boxes for quantitative and qualitative stages of data
collection, data analysis and interpretation of the study
results.
Use capitalized (QUAN) or small letters (quan) to designate
priority of quantitative and qualitative data collection and
analysis.
Use single-headed arrows to show the flow of procedures in
the design.
Specify procedures for each quantitative and qualitative data
collection and analysis stage.
Specify expected products or outcomes of each quantitative
and qualitative data collection and analysis procedure.
Make your model simple.
Size your model to one page.
52

Now lets rework our

purpose statement using
some scripts

53

Sample Script for a Concurrent

Design
(Triangulation or Nested)
The purpose of this concurrent mixed methods study
is to better understand a research problem by
converging both quantitative (numeric) and
qualitative (text or image) data. In this approach,
___________ (quantitative instruments) will be used to
measure the relationship between the ________
(independent variables) and __________ (dependent
variables). At the same time in the study, the
__________ (central phenomenon) will be explored
using _____________ (qualitative interviews,
documents, observations, visual materials) with
_________ (participants) at ____________ (the research
54
site).

Sample Script for a Sequential Exploratory

Design
The purpose of this two-phase, exploratory mixed
methods study will be to explore participant views
with the intent of using this information to develop
and test an instrument with a sample from a
population. The first phase will be a qualitative
exploration of a _______(central phenomenon) by
collecting ___________(data) from ____________
(participants) at _______ (research site). Themes from
this qualitative data will then be developed into an
instrument (or survey) so that the __________ (theory
and research questions/hypotheses) can be tested
that ________ (relate, compare) ____________
55
(independent variable) with __________ (dependent

Sample Script for a Sequential Explanatory D

The purpose of this two-phase, explanatory mixed
methods study will be to obtain statistical,
quantitative results from a sample and then follow-up
with a few individuals to probe or explore those results
in more depth. In the first phase, quantitative
research questions or hypotheses will address the
relationship or comparison of __________ (independent)
and ________ (dependent) variables with ___________
(participants) at ___________(the research site). In the
second phase, qualitative interviews or observations
will be used to problem significant _______(quantitative
results) by exploring aspects of the ________ (central
56
phenomenon) with

Criteria for evaluating our plan:

Use appropriate
terminology for title and
design
Provide a rationale for
mixing and include it
early in the study
(when you use)
Create a mixed methods
purpose statement
Identify types of qual
and quan data to be
collected and qual and
quan data analysis
steps

Include a
visual/procedural
diagram of methods
with timeline
Use rigorous
procedures for the
quantitative data
collection and
analysis

57

Lets share our drawings of

our mixed methods
procedures

58

How to Design a Mixed

Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark,
M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004