Exam-style questions

Topics
Intro to Research

Research problems (choosing qualitative versus quantitative)

Literature review and Purpose statement

Critiquing research

Collecting quantitative data

Descriptive statistics

Collecting qualitative data

Research Design

Mixed Methods

Case Study

Action Research

Comparing groups

Whats the difference between a single item score &

a summed score?
1 item measuring a thing
Overall score reported with multiple items addressing 1 idea

Intro to Research
1. What is research? Why is it important in your discipline?
2. What are the steps in the research process? Explain what
each step involves, in your own words.
3. Compare and contrast quantitative and qualitative research.
What criteria would you use to categorize a study as qualitative
or quantitative?

Topics
Research problems (choosing qualitative versus
quantitative)
4. What kinds of research problems are best investigated using
a quantitative approach? A qualitative approach?
5. Using Figure 2.2 on pg. 58: What characteristics would you
use to categorize a study as quantitative or qualitative?
6. Generate one example of a research problem that is suited
to a quantitative approach, and one that is suited to a
qualitative approach.
7. What are some factors that you should take into account
when selecting a research topic? List as many as you can.
8. Define and distinguish between the following terms: theory,
purpose statement, research problem, research question,
hypothesis, and central phenomenon? What is the purpose of
each?

9. In your own words, describe and contrast the types of

research questions described in Chapter 5. There are 4 main
types of quantitative questions, 2 main types of qualitative
questions.
10. Describe the ethical principles outlined in the Tri-Council
research ethics policy. Why is an ethics policy important? Who
does it protect?

Topics
Literature review and Purpose statement
11. Whats the purpose of the literature review? Explain how the
purpose of the literature review for a quantitative study differs
from that of a qualitative study.
- The purpose of the lit review is to place our research in the
already existing theoretical framework. To help us define key
terms, definitions and terminology. It also gives an overview of
the existing literature to support the rationale of the study,
demonstrate existing gaps, contradicting stances and
opportunities for further development.
- While a Quantitative lit review pertains to substantiate the research problem,
identify variables of interest and provide a rationale for the studys purpose, a Qualitative
lit review documents the importance of the research problem, describes the conceptual
framework for the study and documents models for the methods. Also, a Quant. Lit
review is generally done only at the beginning of the of the study while the Qual lit review
is done throughout the execution of the study.

12. What are the key elements of a quantitative purpose

statement?
- 1) If you plan to test a theory put it in the statement 2) Options
exist for using variables in a statement: a. To relate 2 or more
variables, b. To compare a variable composed of two or more
groups in terms of the dependent variable c. To describe one
variable
(Taken from http://apps.fischlerschool.nova.edu/toolbox/Dissertation/Handouts/The
%20Purpose%20Statement%20Handout%20Sep%2019%202012.pdf)

13. What are the key elements of a qualitative purpose

statement?
- 1) Use the term qualitative or tell the qualitative type of study
you are conducting 2) Use qualitative terms such as: explore,
discover, understand, describe 3) State the central
phenomenon you plan to investigate 4) State the participants
and research site in the study
(Taken from http://apps.fischlerschool.nova.edu/toolbox/Dissertation/Handouts/The
%20Purpose%20Statement%20Handout%20Sep%2019%202012.pdf)

Topics
Critiquing research
14. What major elements should you look for in critiquing a
quantitative study? Explain
15. What major elements should you look for in critiquing a
qualitative study? Explain.
16. How does the writing style used in reports of quantitative
studies differ from that used for qualitative studies?

Topics

Collecting quantitative data

17. What is probabilistic sampling and why is it important in
quantitative research? Give examples of probabilistic and nonprobabilistic sampling methods.
What is probabilistic sampling and why is it important in quantitative research?
In probabilistic sampling, the researchers uses a random process to select individuals (or units, or
groups) from the population so that each individual has a known chance (probability) of being
selected for the study. This is considered the gold standard in experimental research. The
researcher needs to know the precise size of the population (such as a list of all children in foster
care in the city). Done well, the researcher can make the strongest claim that the sample represents
the population. Any bias in the population will be equally distributed. Particularly important to use this
kind of sampling when describing trends in a population.
Examples:
An example of random sampling cited in the textbook (a kind of probabilistic sampling): A researcher
obtains a list of all caseworkers and randomly selects participants from the list.
Non-probabilistic sampling (such as convenience sampling, snowball sampling or purposive
sampling) is used when it is not feasible to do probability sampling. Individuals are selected because
they are available, or meet some criteria that the investigator wants to study. for example, a
purposive sample is used because participants are appropriate for the purpose of the study, such as
a researcher selecting 3 children from foster case who are in need of an intervention to improve
behaviour.
(See Chapter 7, pp. 234-5 in the textbook)

18. What is the difference between random sampling and

random assignment? Why is random assignment difficult in a
quasi-experimental design? And how do researchers attempt to
control for pre-existing group differences?
19. What is a variable? What are the different kinds of
variables?
A variable in a quantitative study refers to a characteristic, attribute, or key idea about individuals or
organizations that researchers collect, measure and analyze variations for their research purpose.
Variables can be measured as categories or along a continuum. There are two types of categorical
variables: nominal and ordinal. Nominal variables are assigned a number which acts a label for

counting but not for ordering. For example American = 1; Canadian = 2; Mexican = 3. Ordinal
variables can be counted and ordered (ranked), but not measured. For example, house numbers or
grade level categories: freshman = 1, sophomore = 2, junior = 3, senior = 4.
Continuous variables refer to scores along a continuum of possible scores such as test performance
from 0% to 100% and level of satisfaction between 1 and 10.

**You can also describe variables based on their role in the research design:
Independent
Dependent
Confounding
Intervening

20. Name and describe the four scales of measurement.

Explain why scales of measurement are important.
Nominal scale: deals with variables that can take on one level among a set of categorical labels (eg:
a variable such as "Nationality" can take on one value among a set where there is no relationship or
order among the different categories).
Ordinal scale: deals with variables that can take on one level among a set of categorical labels, but
where the categorical labels have some sort of order or rank. For example, if after a race, we label
the competitors as "Fastest", "Second Fastest", etc, we effectively give them labels that represent
their order, but these labels don't give us any precise information on how far apart each competitor
is.
Interval scale: deals with variables that take on a numeric value where the difference between any
two numbers that are the same "distance" apart has the same meaning. For example, a Likert scale
can be designed to have equidistant rankings, from which we can meaningfully calculate things like
"average scores".
Ratio scale: deals with variables that take on a numeric value where there is a clear starting point
(ie: a true zero). An example of ratio scale is the weight of objects, where we can meaningfully say
that one object weighs twice as much as another. On the other hand, a score of 4/5 on a Likert scale
vs a score of 2/5 on the same Likert scale doesn't necessarily imply that the first thing is twice as
good as the second.
These scales are important because they give us a set of tools to meaningfully interpret each type of
data. Not all data is amenable to the same kinds of analyses: knowing what type of scale of
measurement is applicable to a variable lets us know exactly what kinds of descriptions and
inferences we can make from the data.

21. Name and describe the various types of validity and

reliability in quantitative research.
Internal vs. external validity (applies to the whole study)

Internal validity refers to our ability to infer a causal relationship between independent and
dependent variables in an experimental study. Threats to internal validity therefore include outside
elements that might interfere in an experiment and have an impact on the dependent variable (ie: a
threat to internal validity is something that isn't successfully controlled for in an experiment).
External validity refers to our ability to generalize that causal relationship to a wider population in
different settings. Threats to external validity therefore include elements of the research design
which may impact our ability to generalize the study's findings. These include concerns about the
representativeness of a study's participants vis--vis the wider population, representativeness of the
study's setting, and generalizability of the study's time frame to different and/or broader periods.

Validity & reliability of measurement

Types of reliability estimates:

Test-retest reliability
Inter-rater reliability
Split-half reliability

Topics
Descriptive statistics
22. Describe the three different ways in which a researcher
might use descriptive statistics.
23. What kind of descriptive statistics indicate: general
tendencies in the data? spread of scores? relative standing?
24. What are the key features of a histogram that you can use
to summarize or describe a distribution?

Topics
Collecting qualitative data
25. Briefly describe each of the qualitative research designs that
Plano Clark and Creswell discuss.
A narrative research design is a chronological retelling of the participants story. It provides a
description of the context and themes that emerged using a story-telling format. For example, in
Performing my Recovery: A Play of Chaos, Restitution and Quest After Traumatic Brain Injury, Claire
Smith (2008) uses a narrative approach to share her experiences from an equestrian accident on
September 13, 1997, throughout her recovery journey, and ending with her graduate studies.
A case study is an in-depth description of a specific situation and themes that emerged, and where
multiple cases were studied, the themes and a comparative analysis across the cases. For
example, in 2008, Dworsky wrote, Educating Homeless Children in Chicago: A Case Study of
Children in the Family Regeneration Program, which explored the educational experiences of the
children in the program. http://www.chapinhall.org/sites/default/files/ChapinHallDocument(2).pdf
Grounded theory is a qualitative research design that uses the principle of collecting data, coding the
data to create themes, then gathering more data. As more data is coded, as concepts become more
apparent, they are grouped further to result in categories, which form the basis for a new theory.
This is different from other approaches which select an existing theoretical framework then collect
data to show if the theory applies or does not apply to the phenomenon under study. More can be
learnt by watching, A Discussion with Prof Kathy Charmaz on Grounded Theory at
https://www.youtube.com/watch?v=D5AHmHQS6WQ .
An ethnography is a research design that provides a detailed description of a groups culture how it
behaves, thinks, or talks. In ethnography, researchers generate understandings of culture through

representation from the insiders point of view by exploring several sources of data. These sources
of data may come from interviews, participant observations, and artifacts such as newspaper
articles, government reports, and magazines. Several examples of ethnographic research can be
found on https://sites.google.com/a/bhmschools.org/cis-writing/samples-1/samples

26. What is purposeful sampling and why is it important? Which

sampling methods are used before data collection begins?
Which sampling methods are used after data collection has
started?
27. What are the different data collection procedures in
qualitative research? List as many as you can. Why do we not
use measures to control for internal and external validity in
qualitative research?
28. What are the roles a researcher can assume during the data
collection process?
29. List and describe 3 ways of validating the accuracy of your
findings in a qualitative study.
M
A
T

List and describe various ways of increasing the trustworthiness

of a qualitative study.
Bracketing
Peer debriefing/review
Using participants own words
Time spent
Purposeful sampling
Using data collection protocols

Topics
Research Design
30. In your own words, whats the difference between internal
and external validity?
31. Distinguish between the instrumentation and testing threats
to validity in experimental design.
32. Name and describe the threats to validity according to
Campbell and Stanley. Explain how 2 of these threats might
alter the conditions of an experiment.

Topics
Mixed Methods
33. List and describe the different types of mixed methods
designs. Explain the rationale for each type.
34. Generate a unique scenario for a research study using a
mixed-method design. Describe why each method would be
used, using the scenario to provide concrete examples.

NOTE: If youre asked to GENERATE a mixed methods study, remember that we need
evidence that you understand the following:
1) Is there a clear need that justifies employing a mixed methods report? For instance, is it clear that
using a stand-alone quantitative or qualitative approach would have been insufficient to answer the
research question?
2) Is the mixed methods design appropriate? There are a variety of mixed methods designs which
vary the sequencing, priority and mixing of the qualitative and quantitative portions of the study. The
chosen design should be effective in answering the research question.

3) Are the qualitative and quantitative portions of the study mixed in a meaningful way? The results
of both portions should be discussed explicitly, and each portion should be explicitly and
meaningfully linked together.

Topics
Case Study
35. Cite three techniques to help increase rigour in a case
study. Give a simple example of each.
Site - good access to the site
Understanding the site - historical view/understanding the system
Triangulation across sources
From slides:

Triangulate the data (examine phenomena from multiple sources/perspectives)

Member check: provide a draft of the write up
Auditing
Multiple coding

These choices are fine - anything that increases rigor for qual in general would be suitable. Any
suggestions from Stark & Torrance would also be acceptable. This style of question would be openended, so just ensure that your responses make sense.

36. Provide a concrete example to describe how case study

methodology advances our knowledge about a given
phenomenon.
Practical case study
Understanding a system

37. What are three factors that influence the selection of sites in
case study research, and how can these influence the
trustworthiness of a study?
Stark & Torrance p. 36 has a discussion of site selection in case study, but you could
also discuss this in terms of purposeful sampling approaches. The connection to
trustworthiness is based on your own reasoning. If Richard didn't emphasize it, it's probably
not on the exam.

Topics
Action Research
38. In what situations would you choose to use an action
research approach? Give examples of research objectives that
you might have.
39. Bob teaches remedial literacy courses in a middle school.
Bob wonders how his students experience reading outside of
the classroom. He wants to know how he can use that
information to enhance his instruction, and to motivate his
students for class. Suggest an action research plan that would
be suitable for Bob. (Answer: what are the key characteristics of
your action research?)
Practical action research design - Bob is studying his own practice, the participants wont be
creating the action plan
Purposeful
-we want a Variety of students -> maximal variation sampling
-5
+

Parents --> confirming/disconfirming

Interviews with parents & students

-Individual interviews
-semi-structured

Thematic analysis
Develop the plan

Topics
Comparing groups & correlation
40. What is are the four main types of quantitative design? How
do they differ?
41. What key characteristics should you look for when
evaluating the quality of a survey research study?
42. What are some common errors in survey question
construction? How can you avoid making these errors when
creating your own survey instruments?
43. What information can the correlation coefficient tell you?
What does r2 mean?
Correlation coefficient is r
-1 to 1
Direction of the relation
Strength of the relationship ---- closer to 1 is stronger

Coefficient of determination r2
How much of the variation in y is explained by x

44. What does it mean to find significance?

WATCH THE VIDEOS
Statistical significance IS NOT the same as substantive significance/practical significance
If you have reached statistical significance, you can reject the null hypothesis.
The level of statistical significance is often expressed as the so-called p-value.
The researcher chose a criterion (the alpha level) and they compare the p value from their data
to that criterion level and make a decision.
Statistically significant results mean that a research result occurred rarely by chance.
The p value is calculated from a critical value of the test statistic.
This group difference is so extreme that it is unlikely to arise from sampling
error alone- the groups are probably actually different.

45. When would a researcher use an ANOVA rather than a ttest?

ANOVA - 2+ groups& 1 independent variable
A researcher would use an ANOVA test over a t-test when they are comparing more than two
groups. A t-test only compares two groups, and while the researcher could run multiple t-tests this
would violate the assumption of independence since the participants in each group will be tested
twice. Also, by conducting multiple t-tests, you increase the chances of a Type 1 error.
An ANOVA test will compare the multiple groups at once and will control for the above error by
keeping alpha at 0.05% and thus not increasing the chances of a Type 1 error.

46. Explain what is being compared in F-ratio of an ANOVA

using plain language.
47. What is a null-hypothesis?
The null hypothesis is essentially the "devil's advocate" position. That is, it assumes that
whatever you are trying to prove did not happen (hint: it usually states that something equals
zero). For example, the two different teaching methods did not result in different exam
performances (i.e., zero difference). Another example might be that there is no relationship
between anxiety and athletic performance (i.e., the slope is zero).
https://statistics.laerd.com/statistical-guides/hypothesis-testing-3.php

48. What are post-hoc tests used for?

Post-hoc tests are used for determining the source of the differences after an ANOVA test shows an
overall statistically significant difference in group means. While an ANOVA test can show an overall
statistically significant difference in the groups that you are testing, it cannot tell you which of those
groups differed. As a result, post-hoc tests are run since they have the ability to determine where
the differences between groups occurred. Like the ANOVA test, post-hoc tests also attempt to control
alpha.
There are three main and widely used post-hoc tests: Turkey (HSD) test; Scheff test; Least
Significant Test (LSD)
https://statistics.laerd.com/statistical-guides/one-way-anova-statistical-guide-4.php

49. Jane is conducting a correlational study for the first time.

She ran her analyses without plotting the data first. Explain to
Jane why plotting first is a good practice.
The correlation coefficient is estimated using a linear model.
Not all relationships between 2 variables can be described accurately using a linear model.
If you plot the data and see that the line appears to change direction, then you know a linear model won't yield an
accurate description of the relationship.It would be misleading (i.e., a correlation coefficient close to zero might not
actually indicate a weak relationship, which is how we would interpret it if we assumed a linear relationship)
(Example: Coffee & attention span. A little coffee -> greater attention span, a LOT of coffee-> short attention span; the
relationship is predictable but not linear)
Here's another classic example: https://en.wikipedia.org/wiki/Yerkes
Dodson_law#/media/File:HebbianYerkesDodson.svg

Exam-style questions
1. What procedures can increase a studys external validity?
a. Manipulating the conditions so they are as similar as possible between
groups
b. Randomly assigning individuals to treatment groups
c. Measuring other variables that need to be controlled
d. Encourage as many participants as possible to complete the instruments

2. Which of the following would provide you with the best

evidence that the scores from an attitudinal measure used in a study
can be used to draw meaningful and useful inferences?
a.

Test-retest reliability

b.

Inter-rater reliability

c.

Construct validity

d.

Standardized procedures

3. If a researcher reports the most common response given to an

item, which statistic would you expect to find?
a.

Standard deviation

b.

Mode

c.

Mean

d.

Percentile rank

4. What is the first step in the process of testing a hypothesis in a

quantitative study?
a.

Set the alpha level

b.

Compute the sample statistic

c.

Identify a null and alternative hypothesis

d.

Collect data

5. Of the factors listed below, which one best distinguishes

participatory action research from practical action research?
a.
b.
c.
d.

The intent to advocate for individuals in the community

The study of problems that need to be solved
The use of action plans to address issues
The use of quantitative and qualitative data

6. Below are listed four types of mixed methods research designs. Match the
design to the alternate name that best fits.
1.
2.
3.
4.
D

Exploratory
Convergent
Embedded
Explanatory

Description

a. Nested

Design #
Embedded

b. Two-phase
c. Triangulation

Convergent

d. Instrument development

Exploratory

8. What is the first step a researcher may use to code a transcript?

a.
b.
c.
d.

Dividing the text into segments

Reading through the data
Reducing overlaps among the codes
Labeling the segments of information with codes

9. In a research study, the researcher sits at a table on the side of

the school's lunchroom while the children have their lunch break. The
researcher records the interactions that he sees between the children
and the school staff. What form of observation is the researcher
engaged in?
a.
b.
c.
d.

A changing observational role

A day-in-life observational role
A nonparticipant observational role
A participant observational role

10. Of the designs available in mixed methods research, what is the

name of this particular design shown in the diagram? Provide a rationale

for why this type of design might be selected.

QUAL

quan