SRU6024

QUANTITATIVE RESEARCH METHODS

ASSIGNMENT 1 REPORT

DESIGNING A QUESTIONNAIRE

 TYPES

 SYSTEMATIC AND CORRECT WAY TO DESIGN A QUESTIONNAIRE

 CHARACTERISTICS OF GOOD ITEMS

 CHARACTERISTICS OF BAD AND GOOD QUESTIONNAIRE

GROUP 1

Submit BY:

MAZIN MAKY THAMER (M20161001043)

MOHAMAD SHAHIR BIN MOHAMAD SOBERI (M20161000636)

FAHIMATUN JAIYIDAH BT ABDUL RAHMAN (M20171000956)

1.0 DESIGNING A QUESTIONNAIRE

Introduction

Most of the social science researches are based on the information provided by the target
population chosen for the analyses. How to determine target or sample population is normally
the part of discussion in sampling design.

Questionnaire design is very crucial and important part of the research, because an
inappropriate questionnaire misleads the research, academics and policymaking. Therefore, a set
of adequate and appropriate questions in a sequential order is required in a questionnaire. A
questionnaire is defined as a document containing questions and other types of items designed to
solicit information appropriate to analysis (Babbie, 1990:377).

Questionnaire is equally used in survey research, experiments and other modes of

observation. Indeed, people ask different questions in their daily life to satisfy their queries.
Journalists, market researchers, observers and interested persons in different events ask a set of
questions to others. They form an opinion or conclusion according to the answers that they receive.
The use of the questionnaire or a set of questions is a similar process in the research. The research
questions are firmly constructed to receive answers related to the chosen variables for analysis.
Social science research is the basic tool to inaugurate, illustrate and predict the human behavior.
The findings of research contribute to policy making and enhancement of the academics. These
findings are based on the answers collected through different questions asked to the sample
population. Naturally, dubious and inconsistent questions produce ambiguous and inharmonious
answers that result in futile analyses and incoherent outcome of the research.

The design of questionnaire is a crucial stage in a quantitative research. Specifically, the

basic steps in quantitative research involve formulating the study objectives, selecting data
collection techniques, developing the instrument (questionnaire), pretesting the questionnaire with
expert judges, pilot testing the questionnaire with a sample from the study population, collecting
the main data from the study population, analyzing the gathered data and writing and disseminating
results (Agarwal, 2011).
What is a questionnaire?

A questionnaire is a research instrument consisting of a series of questions for the purpose of

gathering information from respondents which are often designed for statistical analysis of the
responses (data). In a survey research, the questions in the questionnaire are known referred to as
items (Agarwal, 2011). Questionnaire tend to be a easy way to gather information in a survey
research in that they are cheap, do not require as much effort from the researcher as verbal or
telephone surveys, and often have standardized answers that make it simple to compile data.
However, questionnaires are sharply limited by the fact that respondents must be able to read the
questions and respond to them. More so, while questionnaires are inexpensive, quick, and easy to
analyze, often the questionnaire can have more problems than benefits. For example, unlike
interviews, the people conducting the research may never know if the respondent understood the
question that was being asked. Also, because the questions are so specific to what the researchers
are asking, the information gained can be minimal (Kaplan & Saccuzzo, 2009).

A questionnaire is a research instrument consisting of a series of questions (or other types

of prompts) for the purpose of gathering information from respondents. The questionnaire was
invented by the Statistical Society of London in 1838. (Gault, 1907). Although questionnaires are
often designed for statistical analysis of the responses, this is not always the case.

Questionnaires have advantages over some other types of surveys in that they are cheap, do
not require as much effort from the questioner as verbal or telephone surveys, and often have
standardized answers that make it simple to compile data. However, such standardized answers
may frustrate users. Questionnaires are also sharply limited by the fact that respondents must be
able to read the questions and respond to them. Thus, for some demographic groups conducting a
survey by questionnaire may not be concrete

More generally, one key concern with questionnaires is that there may contain quite large
measurement errors (Alwin, 2007). These errors can be random or systematic. Random errors are
caused by unintended mistakes by respondents, interviewers and/or coders. Systematic error can
occur if there is a systematic reaction of the respondents to the scale used to formulate the survey
question. Thus, the exact formulation of a survey question and its scale are crucial, since they
affect the level of measurement error (Saris, & Gallhofer, 2007). Since questionnaire is regarded
as the heart of a survey and the results of a survey research depend crucially on it, efforts must be
made to minimize response errors or mistakes that could affect the credibility of a result by
designing the questionnaires in accordance with best practices (Krosnick & Presser, 2010).

Types of questionnaire

Types of questionnaire

Open-ended Close-ended

Leading
Questions

Likert
Questions

Dichotomou
s Questions
Survey Monkey
 There are two major types of questionnaire questions they are open-ended and closed-
ended questions. One of the first decisions a researcher must make when designing a survey
question is whether to make it open (permitting respondents to answer in their own words) or
closed (requiring respondents to select an answer from a set of choices). Although the vast majority
of survey questions are closed, some open questions play prominent roles in survey research, such
as those about the most important problem facing a teaching method. In order to analyze the
answers to open questions, they must be grouped into a relatively small number of categories.

Open-ended questions: These gives the respondents an opportunity to express their opinions in a
free-flowing manner. These questions don't have predetermined set of responses and the
respondent is free to answer whatever he/she feels right. By including open format questions in
your questionnaire, you can get true, insightful and even unexpected suggestions. Qualitative
questions fall under this category. Sometimes, a questionnaire could include an open-ended
question at the end of the questionnaire that seeks feedback and/or suggestions for improvements
from respondents.

Closed-ended questions: come in a multitude of forms, but are defined by their need to
have explicit options for a respondent to select from. There are a wide variety of closed-ended
question types for survey creators to choose from, including: Multiple choice, semantic
differential, drop down, check boxes, ranking, and many more. Each question type does not allow
the respondent to provide unique or unanticipated answers, but rather, they have to choose from a
list of pre-selected options.

In most cases of quantitative research, multiple choice questions are used, where
respondents are restricted to choose among any of the given multiple-choice answers. This are
referred to as closed-ended questions (Boynton, & Greenhalgh, 2004). There is no fixed limit as
to how many multiple choices should be given; the number can be even or odd depending on if
mid-point options are included depending on the research objectives.

One of the main advantages of including closed format questions in your questionnaire
design is the ease at performing preliminary analysis. These questions are ideal for calculating
statistical data and percentages, as the answers set is known. Closed ended questions can also be
asked to different groups at different intervals to efficiently track their opinion about a construct.
Closed-ended questions can be further classified into five types.

A. Leading Questions

Questions that force your audience for a particular type of answer are known as leading questions.
In a leading question, all the answers would be equally likely. An example of a leading question
would be a question with choices such as, fair, good, great, poor, superb, excellent etc. These
questions are meant to get an opinion from the audience in limited words.

Example:

My Mathematic teacher is… (a) Fair (b) Good (c) Excellent

B. Importance Questions: In importance questions, the respondents are usually asked to rate the
importance of a particular issue, on a rating scale of 1 to 5. These questions can help researcher to
understand things that hold significance to the respondents and in turn, allow researcher to make
critical decisions.

C. Likert Questions: Likert questions can help to ascertain how strongly respondents agree to a
particular statement.

Example of a Closed-Ended Likert Question

How do you feel about the Principal's performance this year?

(a) strongly approve (b) somewhat approve (c) neutral/no opinion (d) somewhat disapprove
(e) strongly disapprove

D. Dichotomous Questions: These are simple questions that ask respondents to answer in a yes
or no. One major drawback with dichotomous questions is that it cannot analyze the answers
between yes and no, there is no scope for a middle perspective.

Example of a Closed-Ended Dichotomous Question

Do you like your Mathematic teacher? (A) Yes (b) No

Survey Monkey: is an online survey development cloud-based software as a service company,
founded in 1999 by Ryan Finley. Survey Monkey provides free, customizable surveys, as well as
a suite of paid back-end programs that include data analysis, sample selection, bias elimination,
and data representation tools. It is an online survey software that helps you to create and run
professional online surveys, It is very powerful and a well-known online application.

Example: https://www.surveymonkey.com/r/22XZM8K

2.0 HOW TO CONSTRUCT GOOD ITEMS

The process of constructing good questionnaire items involves five major steps (Radhakrishna,
2007).

1. Background / Literature Review

2. Questionnaire Conceptualization
3. Format and Data Analysis
4. Estimate validity and reliability
5. Pilot test (back to estimate validity if value of reliability not satisfy >0.7)

1.
Background

2.
5.
Questionnair
Pilot test Conceptualization

4. 3.
Estimate validity
and reliability Format
Step 1: Background

In this initial step, the purpose, objectives, research questions, and hypothesis of the proposed
research are examined. Determining who is the audience, their background, especially their
educational/readability levels, access, and the process used to select the respondents (sample vs.
population) are also part of this step. A thorough understanding of the problem through literature
search and readings is a must. In this step we need to determine construct or dimension of our
study. Example, if we want to know about what good or bad method of teaching, it is base on
several constructs which is teacher planning, assessment and ability to delivered knowledge. Good
preparation and understanding of Step1 provides the foundation for initiating Step 2.

Step 2: Questionnaire Conceptualization

After developing a thorough understanding of the research, the next step is to generate
statements/questions for the questionnaire. In this step, content (from literature/theoretical
framework) is transformed into statements/questions. In addition, a link among the objectives of
the study and their translation into content is established. Researcher must indicate what the
questionnaire is measuring, that is, knowledge, attitudes, perceptions, opinions, recalling facts,
behavior change, etc. Major variables (independent, dependent, and moderator variables) are
identified and defined in this step. If we choose teaching planning as our construct, we need to
build the item that make sense due to that construct and will measure it. Example of item is, teacher
provide lesson plan before class or teacher has define the learning objective at early lessons, or
Teacher teach base on student need.

Step 3: Format and Data Analysis

In Step 3, the focus is on writing statements/questions, selection of appropriate scales of

measurement, questionnaire layout, format, question ordering, font size, front and back cover, and
proposed data analysis. Scales are devices used to quantify a subject's response on a particular
variable. The Scale , such as nominal scale , ordinal scale, interval and ratio. Example of nominal
scale is Gender, male or female, ordinal scale is such a likert scale. While the interval scale like
how your score in mathematics subject last year. For ratio example like, how frequent you go to
library a week. Understanding the relationship between the level of measurement and the
appropriateness of data analysis is important. For example, if ANOVA (analysis of variance) is
one mode of data analysis, the independent variable must be measured on a nominal scale with
two or more levels (yes, no, not sure), and the dependent variable must be measured on a
interval/ratio scale (strongly agree to strongly disagree).

Step 4: Estimate Validity and Reliability

As a result of Steps 1-3, a draft questionnaire is ready for establishing validity. Validity is
established using a panel of experts and a field test. Which type of validity (content, construct,
criterion, and face) to use depends on the objectives of the study. However, construct validity is
the most considered in quantitative research questionnaire. Construct validity refers to the degree
to which inferences can legitimately be made from the operationalization’s in a study to the
theoretical constructs on which those operationalization’s were based. When both convergent and
discriminant validities are satisfied, construct validity is said to be satisfied. Verifying survey items
for construct validity becomes important, especially when these items are self-developed and not
based on questionnaires used in past studies. The following questions are addressed in Step 4:

1. Is the questionnaire valid? In other words, is the questionnaire measuring what it intended
to measure?
2. Does it represent the content?
3. Is it appropriate for the sample/population?
4. Is the questionnaire comprehensive enough to collect all the information needed to address
the purpose and goals of the study?
5. Does the instrument look like a questionnaire?
After expert validity, researcher need to make changes, as appropriate, based expert opinion.

After that we need run the SPSS (Statistics package for social sciences) to get the value of
estimation of Cronbach alpha to get check for reliability of questionnaire item.. Reliability refers
to random error in measurement. Reliability indicates the accuracy or precision of the measuring
instrument (Norland, 1990). Now the questionnaire is ready to pilot test.

Step 5: Pilot Test

In this step, reliability of the questionnaire using a pilot test is carried out. The pilot test seeks to
answer the question, does the questionnaire consistently measure whatever it measures?

The use of reliability types (test-retest, split half, alternate form, internal consistency)
depends on the nature of data (nominal, ordinal, interval/ratio). For example, to assess reliability
of questions measured on an interval/ratio scale, internal consistency is appropriate to use. To
assess reliability of knowledge questions, test-retest or split-half is appropriate. If the value of
Cronbach alpha is below than 0.7, we need to back to the step number 4 to check for validity
mean , we need to revise to the expert or test to the respondent against either they are understand
our questionnaire or not.

The reliability coefficient (alpha) for questionnaire can range from 0 to 1, with 0 representing an
instrument with full of error and 1 representing total absence of error. A reliability coefficient
(alpha) of .70 or higher is considered acceptable reliability.

Example of item in questionnaire ready to do pilot test.

HOW TO CREATE A SURVEY MONKEY

SurveyMonkey makes it easy to create a survey from scratch or from a template. To create and
send a survey, you need to create the survey, add your questions, apply logic, customize the design,
preview the questions and send your survey. The basic steps to do this are summarized below:

How to create a survey

1) Go to www.surveymonkey.com
2) To create a new survey, click Create Survey in the upper-right corner of your account and
choose one of the options: ‘start from scratch, copy an existing survey, or search survey
and templates.

3) After you give your survey a name, you'll start with a single blank page where you can add
your questions and use the tools in the left sidebar to build your survey. If you select My
questions are already written when you're getting started, you can copy and paste survey
questions you've got ready to go and add them to the survey in bulk.
4) You can create a copy of a survey that's in your account. This way you don't have to start
from scratch or make edits to past surveys you've sent that already have recorded responses.
The survey questions, logic, theme, and other survey design settings are carried over to the
copy, but you'll need to create new a collector to send your survey.

How to add questions and pages

5) Once you create your survey, start by adding questions from the left sidebar. You can add
different question types from the BUILDER section of the sidebar, or choose pre-written
questions from the ‘Question bar’. After you add a question, take a look under the ‘Edit
and Options tabs to further customize it.
6) If your survey is long, break it up into multiple pages. Respondents will feel they are
making progress as they reach each new page, and it can help you organize your survey
into logical sections.
 7) If you started from a template, review the survey and see if there are opportunities to
personalize it. Most of the pre-written questions allow you to customize certain parts of the
question text to ensure that the question is relevant to your survey. You may also want to
add additional questions of your own, or remove questions that aren't useful to you.

How to apply logic

8) After your survey questions and pages are already in place, it's a good time to apply logic
to the survey. On the logic options you can control the behavior of your survey and enhance
the quality of your data.
9) For example, you can use ‘Question Skip Logic’ to control the path of respondents based
on their answers to previous questions. Or, you can use ‘Question Randomization’ to
reduce order bias.

How to customize the design

10) After the structure of your survey is in place, and your logic is working smoothly, you can
focus on fine-tuning the appearance of your survey with design and display options. You
can show or hide elements on the survey page, add a logo and create a custom theme.

How to preview and send your survey

11) Always preview and test your survey before sending it to respondents so you can detect
any mistakes or logic issues. Then once you're ready to send it out, create a collector to get
responses.

3.0 CHARACTERISTICS OF A BAD AND GOOD QUESTIONNAIRE

To construct a good questionnaire, certain considerations must be looked into by the researcher.
The researcher must take into account methodological recommendations. Several methodology
textbooks have offered various versions of recommendations. The most valuable advices pointing
to the characteristics of a good questionnaire as summarized by Krosnick & Presser, 2010 are as
follows:

1. Use simple, familiar words (avoid technical terms, jargon, and slang);

 I use computer during teaching in class

× I use integrating of technology and communication during teaching in class

2. Use simple sentences, example

Item Expert comment

Blogs help me to learn new terms and new Use simpler word like “word” instead of
ideas from classmates on blogs. “term”

3. Avoid words with ambiguous meanings, i.e., aim for wording that all respondents will interpret
in the same way;

Item Expert comment

It is interesting to share our personal idea with “our” is an ambiguous pronoun
others on blogs.

4. Strive for wording that is specific and concrete (as opposed to general and abstract);

Item Expert comment

Blogs make the class more interesting to me This item seems to evaluate interest
Use a “course blog” instead of “blogs”
Define course blog ?

5.Ask about one thing at a time (avoid double-barreled questions); and

 Excellent student easy to understand science concept

× Excellent student difficult to not understand science concept

6.Avoid “loaded” questions , minimize bias in question by using experts and field testing
.Example:

Do you treat your children with kindness like a good parent should?

Are you as interested in sport as most other red –blooded American men?

7. Minimize item with open ended answer such as rarely or frequently that will give different
meaning to respondent. Example:

How often do you speak Bahasa Melayu with?

Never

Rarely

Sometimes

Frequently

All the time

Recommendations to consider also contains advice about how to optimize question order:

1. Early questions should be easy and pleasant to answer, and should build rapport between the
respondent and the researcher.

2. Questions at the very beginning of a questionnaire should explicitly address the topic of the
survey, as it was described to the respondent in the introduction page.

3. Questions on the same topic should be grouped together.

4. Questions on the same topic should proceed from general to specific.

5. Questions on sensitive topics that might make respondents uncomfortable should be placed at
the end of the questionnaire.
 ELEMENTS OF GOOD QUESTIONNAIRE

1. It should have a very clear and precise title: this help to identify the domain of investigation
and also prepare the mind of the participants. It should be captivating enough to get the
participants attention to answer to the items. Example, teachers’ questionnaire on use of
smart phone in teaching and learning.
2. General introduction: this has a description of purpose of the study. The respondent is
assured of confidentiality of information, making clear that there no right or wrong answer.
Honesty answers are requested.
3. Specific instructions: this offers concise instructions on how to responding to the
questionnaire.
4. Questionnaire items: this is the main part of the questionnaire schedule, it should be clearly
separated from the aforementioned parts.
5. Additional information: it should include the full contact information of the researcher.
May include a promise that a copy of the summary of the final report would be sent to the
respondents on request.
6. Appreciation: the questionnaire should include a “Thank you”, or better “Thank you very
much”.

CONCLUSION

The development of the questionnaire for data collection is important to reduce measurement
errors which could arise through inappropriate questionnaire content, questionnaire design and
format, and respondent. Well-crafted conceptualization of the content and transformation of the
content into questions (Step 2) is essential to minimize measurement error (Radhakrishna, 2007).
Careful attention to detail and understanding of the process involved in developing a questionnaire
are of immense value to educators, graduate students, and faculties. Not following appropriate and
methodical procedures in questionnaire development, testing, and evaluation may undermine the
quality and use of data (Esposito, 2002). Anyone involved in educational research, must, at least
follow these five steps to develop a valid and reliable questionnaire items to enhance the quality
of research. Researchers who compose questionnaires should find useful guidance in specific
recommendations for the wording and organization of survey questionnaires. Educational
researchers should benefit general recommendations as provided by research methods books. First,
questionnaire designers should review questions from earlier surveys before writing their own.
This is a matter of efficiency and expertise (Lietz, 2010).

REFERENCES

Gault, RH (1907). "A history of the questionnaire method of research in psychology". Research in
Psychology. 14 (3): 366–383. doi:10.1080/08919402.1907.10532551.

Babbie, Earl, 1990. Survey Research Methods, Second Edition. California: Wadsworth Publishing
Company

Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. Handbook of survey
research, 2(3), 263-314.

Lietz, P. (2010). Research into questionnaire design. International Journal of Market

Research, 52(2), 249-272.

Agarwal, N. K. (2011). Verifying survey items for construct validity: A two‐stage sorting
procedure for questionnaire design in information behavior research. Proceedings of the
Association for Information Science and Technology, 48(1), 1-8.

De Bruijne, M., & Wijnant, A. (2014). Improving response rates and questionnaire design for
mobile web surveys. Public Opinion Quarterly, 78(4), 951-962.

Boynton, P. M., & Greenhalgh, T. (2004). Selecting, designing, and developing your
questionnaire. Bmj, 328(7451), 1312-1315.

Radhakrishna, R. B. (2007). Tips for developing and testing questionnaires/instruments. Journal

of extension.
Esposito, J. L. (2002 November). Interactive, multiple-method questionnaire evaluation research:
A case study. Paper presented at the International Conference in Questionnaire Development,
Evaluation, and Testing (QDET) Methods. Charleston, SC.

Kaplan, R. M., & Saccuzzo, D. P. (2009). Psychological testing: Principles, applications, and
issues. Belmont, CA: Wadsworth.

Alwin, D. F. (2007). Margins of error: A study of reliability in survey measurement (Vol. 547).
John Wiley & Sons.

Othman, T. (2013). Asas Penulisan Tesis Penyelidikan & Statistik (pp. 166-178). Selangor,
Malaysia: UPM.

Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation, and analysis of questionnaires for
survey research (Vol. 548). John Wiley & Sons.

Shahsavar, Z., Tan, B. H., & Aryadoust, S.V. (2010). Investigating the factor structure of
the blog attitude scale. Turkish Online Journal of Distance Education, Notes for editor,
11(4), 12-24.