Effective Questionnaire

What is Questionnaire?
“A questionnaire is a set of questions to be
asked from respondents in an interview, with
appropriate instructions indicating which
questions are to be asked, and in what
experimental design”.
Problem 1: same response
Problem 2: No significant results from hypothesis
testing although logic says so
Problem 3: No result from factor analysis
 Steps in Questionnaire Design
• Preliminary decisions
• Question content
• Response format
• Question wording
• Questionnaire sequence
• Questionnaire pre-test, revision and final draft.
 Five Major Questions while Deciding
the Question Content
• What is the utility of the data collected?

• How effective is a question in eliciting the required data?

• Can the respondent answer the question accurately?

• Is the respondent willing to answer the question

accurately?

• What is the chance of the responses being influenced by

external events?
Guidelines for Developing the
Questionnaire:
• Shared vocabulary
• Unsupported assumptions
• Frame of reference
• Biased wording
• Adequate alternatives
• Double barreled questions
• Generalizations and Estimates
Avoid……………….
Questions such as,
“Were you involved in any extra-marital relationship in the
10 years of your marriage?”

“would you resort to stealing things in a supermarket if you

knew there were no hidden cameras?”

Are virtually sure to attract stereotyped responses or

refusals from participants.
 Effect of external events:
“how many times did you go shopping in the
past one week?”

the survey was conducted just after a week

of heavy rain in the particular city.
Consider the following questions to a lady.

“How often does your man accompany you to…?”

This will elicit varied responses and may even be
misinterpreted. The question assumes that every day the
lady has a spouse or a boyfriend, which is obviously not
the case. Consider another question.

“Would you favor a ban on overcrowding of buses?”

This is sure to provide an inflated estimate of the public’s
demand, unless the assumption “even if it means an
increase in bus fares?” is added. When the assumption
is explicit in the question itself, it tends to produce the
right estimates of the demand for products.
 Biased Wording:
• For example , a question to a factory employee, “would
you favor the replacement of manual labor by
machinery?” is sure to receive a negative response.

• A way of asking the question to read the sub-conscious

mind of the employee would be, “How do you think the
introduction of machines would affect laborers in a
factory?”
• Similarly, a question in a customer feedback form” How
satisfied are you with the service provided at our
restaurant?” is biased as the question implies that the
customer is already satisfied and asks them to grade the
service.

• The question should rather be phrased, “How satisfied or

dissatisfied are you with the service provided at our
restaurant?” thereby avoiding bias.
Double-barreled Questions:
• Questionnaires should avoid asking double-
barreled questions like;
“Do you like fuel-efficient cars with comfortable
seats?”

This is actually a combination of two questions.

• It does not distinguish between people who
prefer cars due to their fuel-efficiency and people
who prefer a car for its comfortable seats or
other competing reasons.
 All responses same:
• Make the questionnaire small

• Negative worded questions

• Ask 2 question, same meaning but

one positive and the other negative
 Major problem is sampling:

• Exactly no solution

• Formula not effective as how can you know the

population standard deviation
2
 σ  σ
2 2
z z
n     2
 e  e
• Best chance is to increase the sample size
 Sample Size
• Rule of 100: Gorsuch (1983) and Kline (1979, p. 40) recommended at least 100 (MacCallum,
Widaman, Zhang & Hong, 1999). No sample should be less than 100 even though the number of
variables is less than 20 (Gorsuch, 1974, p. 333; in Arrindell & van der Ende, 1985, p. 166);

• number of subjects should be the

Hatcher (1994) recommanded that the

larger of 5 times the number of variables, or 100. Even more subjects

are needed when communalities are low and/or few variables load on each factor (in David Garson,
2008).
• Rule of 150: Hutcheson and Sofroniou (1999) recommends at least 150 - 300 cases, more toward
the 150 end when there are a few highly correlated variables, as would be the case when collapsing
highly multicollinear variables (in David Garson, 2008).
• Rule of 200. Guilford (1954, p. 533) suggested that N should be at least 200 cases (in MacCallum,
Widaman, Zhang & Hong, 1999, p84; in Arrindell & van der Ende, 1985; p. 166).
• Rule of 250. Cattell (1978) claimed the minimum desirable N to be 250 (in MacCallum, Widaman,
Zhang & Hong, 1999, p84).
• Rule of 300. There should be at least 300 cases (Noru?is, 2005: 400, in David Garson, 2008).
• Significance rule. Lawley and Maxwell (1971) suggested 51 more cases than the number of
variables, to support chi-square testing (in David Garson, 2008).

• Rule of 500. Comrey and Lee (1992) thought that 100 = poor, 200 = fair, 300
= good, 500 = very good, 1,000 or more = excellent They urged researchers to
obtain samples of 500 or more observations whenever possible (in MacCallum, Widaman, Zhang &
Hong, 1999, p84).
COMMON MISTAKE-1
The sequencing is explained through the
following steps:
• Lead-in questions: ice breakers, for
cooperation
• Qualifying questions: evaluate the
respondent and qualify him/her for further
questioning
• Warm-up questions: making him/her think
of certain facts related to the survey
questions.
• Specific questions: questions that are
specific to the research objectives.
COMMON MISTAKE-2
Demographic Questions:
• These are a necessary part of every survey.

• These usually consist of a set of questions

related to age, sex, location, occupation, etc.

• These questions are kept to the end to avoid

interviewee resistance and to prevent the
interviewee’s attention from being diverted.
 Questionnaire Pre-Testing,
Revision and Final Draft
• Pre-testing refers to testing the questionnaire on a small
sample of respondents selected on a convenient basis
that is not too divergent from the actual respondents.

• See if all the responses are in the same category (all

strongly agree)

• Use cronbach alpha to determine the reliability and

validity of the questionnaire
CARE SHOULD BE TAKEN….
• Type of data generated

• Type of tool used

Likert scales are a common way to capture data about respondent's
opinion on surveys.

Technically, Likert scale data are ordinal. This means that the
response choices have a meaningful order, but the numbers
themselves are not meaningful. For example, consider a scale from
one to five with the following options: Strongly disagree, disagree,
neutral, agree, and strongly agree. Someone who chooses 'agree'
(score four) does not agree twice as much as someone who chooses
'disagree' (score two). So the numbers are not meaningful - only the
order is.

To statistically analyze ordinal data, non-parametric tests should be

used. Tests such as the Mann-Whitney U test or Wilcoxon signed rank
test can be used with ordinal data.

However, it is common to see ordinal data analyzed using parametric

tests, such as the t-test or an ANOVA.
 Way out
• Larger the sample size

• 7 points is better than 5 points

• Check normality assumption using Levene’s test

• Check for equal variance of residuals

• Choose stringent significance level (0.01)

• Run the nonparametric equivalent to your test. If you get

the same results, you can be confident about your
conclusions
 TYPES OF ATTITUDE SCALES
(beyond scope)
Attitude Scales

Single Item Scales Multi Item Scales

Itemized Rank Compara Constant Paired Pictorial Continu

Q-Sort
Category Order -tive Sum Compari- Scales -ous
Scales
Scales Scales Scales Scales son Scales
Scales

Likert Semantic Thurstone Associative Stapel

Scales Differential Scales Scales Scales
Scales
COMMON MISTAKE-3
• We see regression being used on non
metric data

• There is no significant difference in

terms of satisfaction with respect to
age
Satisfaction-dependent-not metric
 Type of tool to be used
• Metric data = • Non metric data =
Interval + Ratio Nominal + ordinal
 Conditions to use t-test &
ANOVA
• Randomness of data
• Normality of data
• Homogeneity of variance (for
ANOVA)
Parametric tests Non-parametric tests
• One sample t test • Kolmogorov-Smirmov

• Independent sample t • Mann-Whitney U test

test
• Paired sample t test • Wilcoxon Signed rank
test
• ANOVA • Kruskal-Wallis
LAST THING
P-VALUE
 Journal named “Nature”
• The 2016 journal metrics
for Nature are as follows:
• 2-year Impact Factor: 40.137
• 5-year Impact Factor: 43.769
• Immediacy index: 9.129
• Eigenfactor ® score: 1.43399
• Article Influence Score: 22.976
• 2-year Median: 24
• Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence
intervals, and power: a guide to misinterpretations. European Journal of
Epidemiology. 2016;31:337-350. doi:10.1007/s10654-016-0149-3.
• A p-Value to Die For, Donald Berry, Journal of the American Statistical
Association Vol. 112 , Iss. 519,2017
• Ronald L. Wasserstein, Nicole A. Lazar. (2016) The ASA's Statement on p-Values:
Context, Process, and Purpose. The American Statistician 70:2, pages 129-133.
• Basic and Applied Social Psychology (BASP) has declared that the null hypothesis
significance testing procedure (NHSTP) is 'invalid', and have banned it from future
papers submitted to the journal.
• Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med
2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
• Investigation of the false discovery rate and the misinterpretation of p-values, David
Colquhoun, R. Soc. open sci. 2014 1 140216; DOI: 10.1098/rsos.140216. Published
19 November 2014
• Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing
, Raymond HubbardR. Murray Lindsay, Theory & Psychology, Vol 18, Issue 1, pp. 69
- 88
 Way out
• Report and use table values
Hope that I have
created more
problems than
solutions….
THANK YOU