Sampling, Questionnaire & Scaling Techniques

ANAND PRAKASH

Questionnaire

Flowchart for Instrument Design

Flowchart for Instrument Design Phase 1

Flowchart for Instrument Design Phase 2

Flowchart for Instrument Design Phase 3

Guidelines for Question Sequencing

Interesting topics early Classification questions later Sensitive questions later Simple items early Transition between topics

Overcoming Instrument Problems

Build rapport Redesign question process Explore alternatives Use other methods Pretest

A Good Questionnaire
Appears as easy to compose as a good poem But, usually the result of long, painstaking work

Developing a Questionnaire
No hard and fast rules Only guidelines

Major Decisions in Questionnaire Design

What should be asked? How should each question be phrased? In what sequence should the questions be arranged? What questionnaire layout will best serve the research objectives? How should the questionnaire be pre-tested? Should it be revised?

Questionnaire Organization
Logical flow Usually go from general to specific Ask sensitive questions later

Some Questionnaires

Customer Service Satisfaction Survey

PURPOSE: Our goal is to provide all of the computer users in our organization with the highest quality of service and responsive support. In order to do this, we need your help. Please take a moment to evaluate your most recent experience with our Help Desk Support Engineers. Please rate your satisfaction with the Help Desk Service form Very Satisfied to Very Dissatisfied. Ease of reaching the Help Desk. Time required to restore or repair the equipment or software. Prompt response by Help Desk Support Engineer. Technical understanding of your problem. Ability to resolve problem on initial visit. Do you have additional equipment or reinstallations planned for your site within the next six months? ____________________________ Please provide any additional comments that could help us improve our Help Desk service and support in the future: _______________ ______________________________________________________

1. 2. 3. 4. 5. 6. 7.

EMPLOYEE SURVEY
SURVEY OVERVIEW Every organization has a unique and individual culture. This culture affects every action of the organization and permeates every level. If an organization wants to function at its most effective level, it must pay close attention to its culture. Successful organizations develop a culture over time to achieve their vision, mission, and objectives, which move them closer to their vision. The following survey is designed to evaluate CornerRock Manufacturing's culture and pinpoint possible areas for improvement.

EMPLOYEE ORIENTATION Our organization is one that greatly values its people. How often do managers compliment employees for doing a good job? EQUAL OPPORTUNITY EMPLOYMENT Our organization does not discriminate against minorities when hiring. The most capable employees are promoted regardless of sex, race or religion. ORGANIZATION'S IMAGE How would you rate the reputation of our organization? To what extent do you consider our organization an industry leader? TRAINING AND CAREER DEVELOPMENT How often do you feel employees are well trained for their jobs? Our organization offers employees excellent opportunities for career development. COMPENSATION How often do you feel that you are fairly compensated for the job you perform? Our organization provides a strong pension plan. COMMUNICATION Employees within our organization receive regular communication from top management. Top management in our organization wants to know what employees think. EMPLOYEE INVOLVEMENT Employees are proud to be part of our organization. The work is very challenging within our organization.

Pyramid Corporation 360 Degree Leadership Assessment

The purpose of this 360 Degree Assessment is to identify improvement opportunities for our leadership personnel. This information will be used to measure the effectiveness of our leadership today and where they might be in the future. All information given in this assessment and subsequent interviews will be held confidential by our organization. Mark on 5 point scale 1 = Definite Strength 2 = Moderately Effective 3 = Average Performance 4 = Rarely Effective 5 = Definite Weakness LEADERSHIP EFFECTIVENESS This individual is sensitive to the influence his/her actions have on co-workers. This individual leads co-workers by example. This individual continually encourages co-workers to express their ideas and opinions. When conflict occurs, this individual resolves them in a constructive win/win manner. This individual continually develops the spirit of teamwork among co-workers. This individual always looks for new and creative methods to motivate co-workers. This individual clearly understands co-worker's roles in our organization.

1. 2. 3. 4. 5. 6. 7.

PLANNING 8. This Individual understands his/her job responsibility, accountability, and authority clearly. 9. This individual makes realistic plans and schedules and puts them in writing. 10. This individual uses his/her resources (workforce, time, money, etc.) productively. 11. This individual sees that co-workers have the necessary resources to do their jobs productively.

Health Insurance Satisfaction Survey

Please take a few minutes and tell us how we are doing. We appreciate your valuable time and comments. All your responses will be held confidential. 1. 2. 3. 4. 5. 6. Date: Time: How would you rate our agency on type of service provided: Purpose for insurance policy: Number of people covered by your insurance policy: How much time did it take our staff to collect the information and complete your claim?
5 minutes or less 6-15 minutes 16-30 minutes 31-45 minutes 46 minutes to 1 hour Over 1 hour

7. Would you recommend our health insurance services to your relatives and friends? Yes Maybe No

8. 9. 10. 11. 12. 13. 14.

How would you rate our agency on a 5 point scale meaning 1 = Great 2 = Very Good 3 = Good 4 = Fair 5 = Poor. Promptness Friendliness Patience Responsiveness Clear consistent communication How claims were handled How would you rate the value received for money spent for our health insurance services?

15. Is there anything we could do differently to improve our insurance coverage and/or service? http://www.surveytracker.com/surveys/samples.htm#360

KINGFISHER AIRLINES
[The Good Times Monitor]
Wed like to know a little more about your flight 1. Your flight number: IT________________________________ 2. Your Seat No.:______________________________________ 3. Takeoff city:________________ Landing city:_____________ 4. Flight date (dd)/(mm)/(yyyy): __________________________ 5. King Club No.:______________________________________ Wed like to know a little more about you Name (Mr./Mrs.)________________________________________ Address:______________________________________________ _____________________________________________________ City:______________________________ Pin Code:___________ Tel. No.:______________________________________________ Email ID:______________________________________________

KINGFISHER AIRLINES
[The Good Times Monitor]
How did you make your reservation? [ ] Through our website? [ ] Through a travel agent? [ ] Through the Kingfisher Airlines ticketing office? [ ] Through home delivery of ticket? In which city did you make your reservation? ________________ Had a good time? This section comprises of asking opinions for three dimensions on a four pointer scale as Wow, Good, Not bad, and Could be better. Various statements under this heading are:

KINGFISHER AIRLINES
[The Good Times Monitor]

1. Reaching Out
a) Accessibility of our call center members b) How were we during the reservation process?
i. Staff efficiency ii. Staff courtesy and friendliness

c) Overall experience

2. At the Airport 3. On board

KINGFISHER AIRLINES
[The Good Times Monitor]
1. 2. Reaching Out At the Airport
a) b) How did we handle your baggage?
i. ii. i. ii. iii. iv. v. vi. Time taken for security screening Assistance provided Waiting time at Check-in counter Time taken in to Check-in Staff greeting, helpfulness & warmth Staff presentability Staff efficiency Overall check-in process Clarity of boarding announcement Process at the boarding hall Boarding the aircraft

How did we check you in?

c)

Did we get you on board alright?

i. ii. iii.

d)

Overall airport experience

3.

On board

KINGFISHER AIRLINES
[The Good Times Monitor]
1. 2. 3. Reaching Out At the Airport On board
a) How did you like your crew?
i. ii. iii. iv. v. vi. Welcome while boarding Assistance on board Courtesy & professionalism Responsiveness to your needs Presentability Overall cabin crew rating Audio selection Kingfisher Radio Video selection Fun TV Magazine Hi-Blitz Clarity of captain's announcements Flight info in captain's announcement Cleanliness of cabin Cleanliness of washroom Temperature before takeoff Temperature after takeoff Did we make you feel at home

b)

Had an entertaining time?

i. ii. iii.

c)

Did we make ourselves heard?

i. ii.

d)

Did we make you comfortable?

i. ii. iii. iv. v. Too warm [ ] Just right [ ] Too cold [ ] Too warm [ ] Just right [ ] Too cold [ ]

KINGFISHER AIRLINES [The Good Times Monitor]

Lets Talk About Food Could you tell us what kind of meal you had? [ ] Vegetarian [ ] Non-vegetarian [ ] Other (please specify) __________________________________ Rate on the scale of 14, where 1 = Wow, , 4 = Could be better
a) b) c) d) e) Appropriateness of the meal for the time of flight Visual appeal The taste Meal portion Overall food rating

How was the Kingfisher Class Experience, overall? Wow [ ] Good [ ] Not bad [ ] Could be better [ ]

How can we make the Kingfisher Class Experience even better for you? [ ] Suggestion [ ] Query [ ] Compliment [ ] Concern __________________________________________________________________ __________________________________________________________________ __________________________________________________________________

Check Your Own Student Feedback Sheet

1. 2. 3. 4. 5. 6. Knowledge Gain Guidance & Advice Style of Teaching Atmosphere in Class Adaptability & Responsiveness General Impression

Check Your Own Student Feedback Sheet

1. Knowledge Gain
Could have learnt by reading text book Copies/Notes provided to supplement text book Explanation and examples given in the class A very all-round, practicable learning

2. 3. 4. 5. 6.

Guidance & Advice Style of Teaching Atmosphere in Class Adaptability & Responsiveness General Impression

Check Your Own Student Feedback Sheet

1. Knowledge Gain 2. Guidance & Advice
Questions not always answered Questions usually answered but partially Care taken to explain specifics Care taken to clarify satisfactory & wholistically

3. 4. 5. 6.

Style of Teaching Atmosphere in Class Adaptability & Responsiveness General Impression

Check Your Own Student Feedback Sheet

1. Knowledge Gain 2. Guidance & Advice 3. Style of Teaching
Mainly syllabus covering lectures (Speech) Limited use of OHP/PC for presentation Use of OHP/PC + cases/examples for teaching Balanced use of lectures, cases, and other acts

4. Atmosphere in Class 5. Adaptability & Responsiveness 6. General Impression

Check Your Own Student Feedback Sheet

1. 2. 3. 4. Knowledge Gain Guidance & Advice Style of Teaching Atmosphere in Class
Not teaching, one way communication by lectures Too light and open; not much learning Occasionally lively Usually very lively

5. Adaptability & Responsiveness 6. General Impression

Check Your Own Student Feedback Sheet

1. 2. 3. 4. 5. Knowledge Gain Guidance & Advice Style of Teaching Atmosphere in Class Adaptability & Responsiveness
Generally, a feeling of unapproachability Generally approachable, but time severely limited Generally approachable and usually tries to understand students view point Very approachable, responsive and flexible

6. General Impression

Check Your Own Student Feedback Sheet

1. 2. 3. 4. 5. 6. Knowledge Gain Guidance & Advice Style of Teaching Atmosphere in Class Adaptability & Responsiveness General Impression
Attend classes mainly to avoid TNG Would attend since helpful for examinations Would attend mostly since a learning experience Would positively attend all classes

Check Your Own Student Feedback Sheet

Are you willing to discuss your assessment with the concerned faculty?
Yes / No

Do you wish to keep this confidential?

Yes / No

Please use the reverse side for general comments.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24.

Surveys are all around us. They can be used to: Measure and improve customers satisfaction levels. Discover worker attitudes about issues affecting the work environment, quality, and productivity. Quickly evaluate opinions and attitudes. Provide data for long-range strategic planning. Enhance customer relations. Determine specific training needs for an organization. Determine the effectiveness of a Help Desk. Evaluate internal and external supplier quality Determine readiness for industry standards such as ISO 9000 and Malcolm Baldrige. Increase employee commitment through involvement and implementation of a common mission. Evaluate an employee's leadership effectiveness. Measure the quality of an education system. Evaluate and track Total Quality Team effectiveness. Ensure communication among all levels of an organization. Determine public perception of political candidates and issues. Determine how many and what types of people watch television programs. Determine the effectiveness and attractiveness of a web page. Evaluate customer satisfaction. Gather feedback about a restaurants service and aesthetics. Evaluate the effectiveness of care and facilities of a hospital. Measure satisfaction, cooperation, and readiness of the military. Discover new market possibilities. Gauge a lodger's enjoyment of the facilities, room cleanliness, etc. Discover opportunities for improvement.

Survey Responsibility Checklist

PLAN DO Develop a schedule for key dates of the survey project. Decide what demographics are relevant to your survey. Decide upon the survey sampling techniques. Decide what topics are to be addressed in the survey. Draft a cover letter to accompany the survey. Review the audience list for integrity and enter it into a software application Determine the survey delivery methods you will employ based on your audience list. Send letters to survey/interview respondents. Print surveys and prepare to present them to the respondents. Coordinate the administering of the surveys. Retrieve returned surveys from the mail, disk, e-mail, web, etc. Enter or read data from the returned surveys into software. Develop a list of survey objectives (expected outcomes). Develop a list of potential action steps reflecting the outcomes. Develop a list of potential contingency steps reflecting changes in the action steps or outcomes. Secure management's commitment and support. Assign or nominate a project administrator\manager\coordinator. Determine the survey delivery methods available. Develop a projected budget for all of the survey costs. Select members for a Survey Project Team. Select Team members to help design and administer the survey.

STUDY Analyze the data and develop a survey report. ACT Present the results to management. Present the feedback to respondents/audience. Establish an action plan. Establish a responsibility chart for action implementation and follow-up.

Reliability
Reliability can be defined as the extent to which measures are free from random error, XR. If XR = 0, the measure is perfectly reliable. In test-retest reliability, respondents are administered identical sets of scale items at two different times and the degree of similarity between the two measurements is determined. In alternative-forms reliability, two equivalent forms of the scale are constructed and the same respondents are measured at two different times, with a different form being used each time.

Reliability
Internal consistency reliability determines the extent to which different parts of a summated scale are consistent in what they indicate about the characteristic being measured. In split-half reliability, the items on the scale are divided into two halves and the resulting half scores are correlated. The coefficient alpha, or Cronbach's alpha, is the average of all possible split-half coefficients resulting from different ways of splitting the scale items. This coefficient varies from 0 to 1, and a value of 0.6 or less generally indicates unsatisfactory internal consistency reliability.

Validity
The validity of a scale may be defined as the extent to which differences in observed scale scores reflect true differences among objects on the characteristic being measured, rather than systematic or random error. Perfect validity requires that there be no measurement error (XO = XT, XR = 0, XS = 0). Content validity is a subjective but systematic evaluation of how well the content of a scale represents the measurement task at hand. Criterion validity reflects whether a scale performs as expected in relation to other variables selected (criterion variables) as meaningful criteria.

Validity
Construct validity addresses the question of what construct or characteristic the scale is, in fact, measuring. Construct validity includes convergent, discriminant, and nomological validity. Convergent validity is the extent to which the scale correlates positively with other measures of the same construct. Discriminant validity is the extent to which a measure does not correlate with other constructs from which it is supposed to differ. Nomological validity is the extent to which the scale correlates in theoretically predicted ways with measures of different but related constructs.

 Reliability is a necessary, sufficient, condition for validity.

but

not

BACK PAIN AND DISABILITY QUESTIONNAIRE (Revised Oswestry) Name Date

This questionnaire has been designed to give your health care provider information as to how your back pain has affected your ability to manage everyday life. Please answer every section and mark in each section only the ONE box which applies to you. I realize you may consider that two of the statements in any one section relate to you, but please just mark the box which most closely describes your problem today. Section 1 Pain Intensity The pain comes and goes and is very mild. The pain is mild and does not vary much. The pain comes and goes and is moderate. The pain is moderate and does not vary much. The pain comes and goes and is severe. The pain is severe and does not vary much. Section 6 Standing I can stand as long as I want without pain. I have some pain on standing but it does not increase with time. I cannot stand for longer than one hour without increasing pain. I cannot stand for longer than hour without increasing pain. I cannot stand for longer than 10 minutes without increasing pain. I avoid standing because it increases the pain straight away.

Section 2 Personal Care I would not have to change my way of washing or dressing in order to avoid pain. I do not normally change my way of washing and dressing even though it causes some pain. Washing and dressing increase the pain but I manage not to change my way of doing it. Washing and dressing increase the pain and I find it necessary to change my way of doing it. Because of the pain I am unable to do some washing and dressing. Because of the pain I am unable to do any washing and dressing without help.

Section 7 Sleeping I get no pain in bed. I get pain in bed but it does not prevent me from sleeping well. Because of pain my normal nights sleep is reduced by less than . Because of pain my normal nights sleep is reduced by less than . Because of pain my normal nights sleep is reduced by less than . Pain prevents me from sleeping at all.

Section 3 Lifting I can lift heavy weights without extra pain. I can lift heavy weights but it causes extra pain. Pain prevents me from lifting heavy weights off the floor. Pain prevents me from lifting heavy weights off the floor, but I manage if they are conveniently positioned (e.g. on a table). Pain prevents me from lifting heavy weights but I can manage light to medium weights if they are conveniently positioned. I can only lift very light weights at the most.

Section 8 Social Life My social life is normal and gives me no pain. My social life is normal but increases the degree of pain. Pain has no significant effect on my social life apart from limiting my more energetic interests, e.g. dancing, etc. Pain has restricted my social life and I do not go out very often. Pain has restricted my social life to my home. I have hardly any social life because of the pain.

Section 4 Walking
I have no pain on walking. I have some pain on walking but it does not increase with distance. I cannot walk more than 1 mile without increasing pain. I cannot walk more than mile without increasing pain. I cannot walk more than mile without increasing pain. I cannot walk at all without increasing pain.

Section 9 Traveling
I have no pain while traveling. I have some pain while traveling but none of my usual forms of travel make it any worse. I have extra pain while traveling but it does not compel me to seek alternate forms of travel. I have extra pain while traveling that compels me to seek alternative forms of travel. Pain restricts all forms of travel. Pain prevents all forms of travel except that done lying down.

Section 5 Sitting I can sit in any chair as long as I like. I can only sit in my favorite chair as long as I like. Pain prevents me from sitting more than 1 hour. Pain prevents me from sitting more than hour. Pain prevents me from sitting more than 10 minutes. I avoid sitting because it increases my pain right away.

Section 10 Changing Degree of Pain My pain is rapidly getting better. My pain fluctuates but overall is definitely getting better. My pain seems to be getting better but improvement is slow at present. My pain is neither getting better nor worse. My pain is gradually worsening. My pain is rapidly worsening.

Oswestry Disability Questionnaire (revised)

Purpose: * Acute and Chronic population of low back pain sufferers * Discriminate between chronic and acute low back pain * An evaluative measure in clinical trials * Used to predict different rates of improvement + Measured 0 5 by degree of difficulty + Very specific questions + No specific frame of reference + Capacity related questions + Score by summing all items = percentage score Pain intensity Walking Sleeping Personal care Lifting Sitting Standing Sex / social life Travelling

Face Validity:

Content Validity:

Oswestry Disability Questionnaire (revised)

Content Validity: Omits: bending twisting emotional state kneeling turning sudden movement

Sex life reduced response rates (Hudson-Cook et al. 1989) Scoring issues: 11% is a cut off score (Erhard et al. 1994) 00 - 20% 20 - 40% 40 - 60% 60 - 80% 80 - 100% Minimal Disability Moderate Disability Severe Disability Crippled Bed Bound or Exaggerating

Stratford et al. 1988

Modified Oswestry Low Back Pain Disability Questionnaire

This questionnaire has been designed to give your therapist information as to how your back pain has affected your ability to manage in everyday life. Please answer every question by placing a mark in the one box that best describes your condition today. We realize you may feel that 2 of the statements may describe your condition, but please mark only the box that most closely describes your current condition.

Pain Intensity I can tolerate the pain I have without having to use pain medication. The pain is bad, but I can manage without having to take pain medication. Pain medication provides me with complete relief from pain. Pain medication provides me with moderate relief from pain. Pain medication provides me with little relief from pain. Pain medication has no effect on my pain. Personal Care (e.g., Washing, Dressing) I can take care of myself normally without causing increased pain. I can take care of myself normally, but it increases my pain. It is painful to take care of myself, and I am slow and careful. I need help, but I am able to manage most of my personal care. I need help every day in most aspects of my care. I do not get dressed, I wash with difficulty, and I stay in bed. Lifting I can lift heavy weights without increased pain. I can lift heavy weights, but it causes increased pain. Pain prevents me from lifting heavy weights off the floor, but I can manage if the weights are conveniently positioned (e.g., on a table). Pain prevents me from lifting heavy weights, but I can manage light to medium weights if they are conveniently positioned. I can lift only very light weights. I cannot lift or carry anything at all. Walking Pain does not prevent me from walking any distance. Pain prevents me from walking more than 1 mile. (1 mile = 1.6 km). Pain prevents me from walking more than 1/2 mile. Pain prevents me from walking more than 1/4 mile. I can walk only with crutches or a cane. I am in bed most of the time and have to crawl to the toilet.

Sitting I can sit in any chair as long as I like. I can only sit in my favorite chair as long as I like. Pain prevents me from sitting for more than 1 hour. Pain prevents me from sitting for more than 1/2 hour. Pain prevents me from sitting for more than 10 minutes. Pain prevents me from sitting at all. Standing I can stand as long as I want without increased pain. I can stand as long as I want, but it increases my pain. Pain prevents me from standing for more than 1 hour. Pain prevents me from standing for more than 1/2 hour. Pain prevents me from standing for more than 10 minutes. Pain prevents me from standing at all. Sleeping Pain does not prevent me from sleeping well. I can sleep well only by using pain medication. Even when I take medication, I sleep less than 6 hours. Even when I take medication, I sleep less than 4 hours. Even when I take medication, I sleep less than 2 hours. Pain prevents me from sleeping at all. Social Life My social life is normal and does not increase my pain. My social life is normal, but it increases my level of pain. Pain prevents me from participating in more energetic activities (e.g., sports, dancing). Pain prevents me form going out very often. Pain has restricted my social life to my home. I have hardly any social life because of my pain. Traveling I can travel anywhere without increased pain. I can travel anywhere, but it increases my pain. My pain restricts my travel over 2 hours. My pain restricts my travel over 1 hour. My pain restricts my travel to short necessary journeys under 1/2 hour. My pain prevents all travel except for visits to the physician / therapist or hospital. Employment / Homemaking My normal homemaking / job activities do not cause pain. My normal homemaking / job activities increase my pain, but I can still perform all that is required of me. I can perform most of my homemaking / job duties, but pain prevents me from performing more physically stressful activities (e.g., lifting, vacuuming). Pain prevents me from doing anything but light duties. Pain prevents me from doing even light duties. Pain prevents me from performing any job or homemaking chores.

Quebec Back Pain Disability Scale

Purpose: * Acute and Chronic population of low back pain sufferers * Assess level of functional disability * Designed as discriminative, evaluative and predictive + Response on rating scale 0 - 5 + Very specific questions + Today as frame of reference + Performance related questions + Score by summing all items = percentage score Mobility Sitting Lifting Travelling Standing Bending Sleeping Running

Face Validity:

Content Validity:

Quebec Back Pain Disability Scale

Content Validity:

Omits: twisting emotional state sex life

turning sudden movement

Characteristics of Good Measurement

1. 2. 3. Utility / Practicality
Is it useful? Is it dependable? Split Half Does it do what it is supposed to? Can it identify patients with a condition? Can it identify those that do not have the condition? Can it measure differences over time?

Reliability Internal Consistency

4.
5.

Validity

Sensitivity

6.
7.

Specificity

Responsiveness

Consider the dart boards shown below.

The marks you see on the targets were left by darts thrown at them.

If the goal was to hit the Bullseye with each dart Then the results were consistent but offtarget

If the goal was to hit the Bullseye with each dart Then the results were both consistent and accurate

If the goal was to hit the Bullseye with each dart Then the results were neither consistent nor accurate

Reliability and Validity

Reliability has to do with consistency Validity has to do with accuracy

If the goal was to hit the Bullseye with each dart We have reliability but not validity

If the goal was to hit the Bullseye with each dart

We have both reliability and validity

If the goal was to hit the Bullseye with each dart

We have neither reliability nor validity

Reliability and Validity

To have validity we must first have reliability
i.e. reliability is a prerequisite for validity

Reliability is a necessary but not sufficient condition for validity

Reliability is a necessary but not sufficient condition for validity

We can have reliability without having validity

In order to be valid the measurement must be reliable

We cannot have validity without first having reliability

Reliability is a prerequisite for validity

Without reliability, we cannot have validity

Measurement reliable but not valid

Measurement reliable and valid

Measurement neither reliable nor valid

Reliable Not Valid

Reliable Valid

Not Reliable Not Valid