Académique Documents
Professionnel Documents
Culture Documents
Maureen S. Heaphy
The Transformation Network, Inc.
Gregory F. Gruska
Omnex
Surveys are experiencing an increased popularity in many fields of study including quality, organizational development, and marketing. This is partially due to the Malcolm Baldrige National Quality Award criteria. More organizations are now asking How do we know our efforts are effective?. Whether this means are customers delighted with a product or service, do employees feel their opinion is important, or are teams getting the support they need, organizations are turning to surveys to obtain the answer. Unfortunately with many surveys, data is first collected then someone tries to make sense of it all. Instead of being a science, the construction and conducting of a survey is slighted. The 80/20 rule should be applied. That is, planning should be 80% of effort and conducting and analyzing should be 20% of the effort. Remember the Deming cycle (also referred to as the Shewhart cycle) of Plan Do Study Act. If the planning stage is thorough then the analysis should have been decided upon before any data is even collected.
Decision to conduct survey Formulation of questions What analysis will be done what to do about non returns and non responses Validating questions pilot with interview Conducting survey Analyze results Take action
Figure 1 General Steps of a Survey
Planning
The first step in planning is to make the decision whether a survey is the correct tool to use. It has been said that 'A person, given a hammer, discovers that many things need pounding'. This is true about the use and misuse of surveys as well as other statistical tools. A written statement of purpose should be developed to aid in making in determining whether to conduct a survey or not. Determine what decisions are going to be made based on the survey results and by whom. Critical review of some economic surveys showed many were useless because they started with no particular hypothesis did not have structure hence could not provide any conclusions
All too often, professional assistance is called in after a survey has been conducted and help is requested in making sense out of a poorly constructed survey. To avoid that, get professional assistance before decision is even made to conduct survey. The focus here is on preventing problems, not reacting to them.
Terminology
Some definitions are appropriate at this point. The target population refers to those people or entities that needs to be surveyed to address the identified issue whereas the population refers to the entire group of interest. The sample are those people or items selected to be included in the study. There may be random and stratified samples Random - everyone in the population has an equal chance of being selected Stratified - the population is divided into groups (strata) and then random samples are select from each group. This requires subject matter knowledge, not statistics. For example, if a company is considering adding to their employee benefit program and they want input from their employees, the company may decide to use a stratified sample. Perhaps age and length of service are the two factors used in creating the strata. This would be based on subject matter not statistical knowledge.
1990 The Transformation Network, Inc and The Third Generation, Inc.
Uncertainties
Recognize that in any survey there are uncertainties or errors. The types of survey errors are listed in Figure 2. Sampling Error sample too small sample not representative Non sampling Error Random error misinterpretation of question attitude Bias non response bias response bias (untruth) administrative Figure 2 Survey Errors A statistician can provide assistance in this area by calculating the associated errors with certain sample sizes. This information, combined with the subject matter experts opinion of expected differences, can be used in selecting the appropriate sample sizes. This raises a whole new issue of analytic versus enumerative situations which is beyond the cope of this paper. See Gruska (bibliography) for additional information on analytic versus enumerative.
A third point addresses the issue of personal or impersonal wording of questions. That is, do you want general perceptions or individual opinion? For example, if you asked Are the reference materials in the library sufficient? you should be clear if you want to know if the respondent finds them sufficient for their own use or does the respond think they are sufficient for people in general in the department. Just as there are things to do, there are things to avoid. In developing questions avoid leading or biased questions, double negatives, and multiple questions in one. The double negative comes about when a question has a negative in it such as Managers should not be required to review their employees reports and then the answers to choose from are agree or disagree. If someone disagrees that managers should not have to do the task then they are agreeing that mangers should have to do the task. The confusion here should be apparent now. The issue of having multiple questions in one deserves additional explanation. If a survey is being done as a coarse filter to assist in determining where a closer look is needed, then multiple questions may be combined into one, BUT the instructions must be clear on how to answer it. For example, consider a survey that has the following question: We listen to our customers, respond to their requirements and have well trained customer contact employees who have the authority to resolve problems quickly. The answers to choose from range from strongly agree to strongly disagree. If a company does listen to their customers but the customer contact employees are not authorized to resolve issues quickly, should the respondent pick a middle of the road answer since half is yes and half is no? Or do both criteria need to be met to agree with the questions? Without further instructions, it is not clear how to answer.
Tips
In developing questions for a survey there are some common sense tips of things to do. In formulating the questions use terms common to the target population, keep the choices short, provide operational definitions, place easy, interesting or short questions at the beginning of the survey and be sensitive to gender words. Before developing a survey some general observations should be considered. One such consideration is that people have a tendency to agree with statements. This was demonstrated by having two contradictory statements in a survey and a certain percentage of respondents agreed with both statements. Another consideration is that strong negatives have strong effects. It is best to avoid words such: control, forbid, restrict, restrain, oppose. More people are willing to not allow than are willing to forbid. For example, if you asked people if public demonstrations should be allowed regarding abortions and later asked if public demonstrations should be forbidden, more people are willing to say not allow than are willing to say forbidden
Likert Scale
A common response scale was used in the above example. It is called a Likert scale, developed by Rensis Likert. It is used for assessing opinions and is usually 5 or more response categories. For example, the choices might be stated as follows: strongly agree agree undecided disagree strongly disagree
Generally, it is recommended that a N.A. - Not Applicable choice be made available in addition to the Likert scale. This semi-variable approach does provide more information than the simple attribute (yes-no) question. Because many people are reluctant to make a firm decision or stand, the center response may be selected by default. Some surveys will use an even number of responses (4 or 6) to force a position. Traditional variables analysis must be used with caution since the Likert scale is not usually an interval scale.
An important step in the survey process that is often overlooked is the activity of validating questions. This could be accomplished by conducting a pilot study. The steps involved are as follows: select participants to participate in pilot have participants complete the survey have the team that designed survey review results participants individually interviewed by design team During the interview conducted for the pilot a typical discussion would include questions such as What did the question mean to you?, How would you say it?, What did ______ make you think of?, What was it you had in mind when you said _____?. For example, if terms such as empowerment or robust designs were used in the survey, you don't want to find out after the survey is completed that some people interpreted the words different than intended. A classic example is taken from an automotive company survey conducted back in the late 1970's. Survey results showed that door closing effort was a major complaint but yet the automotive company felt it had resolved that problem. A follow up clinic was set up in a major city and people who had complained about the door closing efforts were invited to participate. People were asked to arrive at a dealership for one-half hour so the engineers could measure the door closing effort and talk to the owners. In return the owners would get a car wash and tank of gas for their efforts. The engineers quickly discovered that the real complaint was the door to fender gap was unacceptable. Since the survey did not have any questions related to fit of sheet metal and since the door was involved, people checked the only item listed relating to doors. One result of the pilot study should be an estimate of the expected response rate of the full survey.
Behavioral Assessments
With American management transforming their behavior to coaches there are many behavioral surveys being done. Often times this involves a pre and post survey; i.e. before and after some event such as training. Proceed with extreme caution before conducting a behavioral assessment. An example is a survey chat your peers complete regarding your behavior before and after you attend a leadership class on Leadership for Diversity or some such tide. It is expected that you will have higher scores showing increased sensitivity to issues 3 to 6 months after completing the class. There could in fact be no observable change in the metric (numeric answer) but it may occur for a variety of reasons. Perhaps your behavioral did change (so your scores should have been higher), but this is offset because the awareness level of your peers completing the survey changed (so they became harsher in scoring). Consider a particular issue such as a question pertaining to trust in the survey. The behavioral pattern is discussed in class so the participants are more observant of those patterns and their value system has changed. You and your peers have participated in the class. Suppose your behavior does improve but your peers are harsher in their evaluations. The net result is no noticeable change in your score. There are ways to handle this complication, and the first step is to be aware of this situation during the planning stage. The discussion above did not include the other source of variation, namely the measurement system repeatability. If a survey is completed today and again tomorrow the answers may change some what even though nothing in the system has changed. But that is a topic for another paper.
Response Rates
The percent of questionnaires returned will vary depending on numerous factors such as: Interest level strong feelings Perceived impact action will be taken based on results Length of questionnaire time it takes to complete delay responding indefinitely
Pilot
3
Level of obligation response is required Certain factors may dominate in the decision to respond. For example, suppose everything was done by the book but the response is low. It may be due to a low interest level or people felt nothing would be done with results. On the other hand, a questionnaire that is considered to be too long by many experts may have a high response rate because the subject is of such great interest.
In Closing
Be sure the planning stage of a survey is given full attention. Don't wait and then try to make sense of the data after the survey has been conducted. You may find yourself with data that cannot be converted into information. As a last note, when a survey is conducted, the expectations of the respondents as well as the clients are often raised. If someone indicated they were unsatisfied with the work environment, they now expect something will be done about the situation. Feedback of the plans based on the survey results to the respondents is important. The Plan Do Study Act cycle should be used on an ongoing basis.
viding services in the development, implementation an supporting training in process control and Total Quality Management (TQM) using statistical methods and employee involvement. Assistance is also provided to companies wanting to use the Malcolm Baldrige National Quality Award criteria for self improvement. She worked for several divisions of General Motors over a 13 year period. Her last position was Director of Statistical Methods. She has advance degrees in Statistics from University of Michigan and Industrial Engineering Operations Research from Wayne State University. She is an officer of the Automotive Division of ASQ, is certified by the society as both a Quality and Reliability Engineer. Maureen has been an examiner for the Malcolm Baldrige National Quality Award, and a judge for the Michigan Quality Leadership Award. Mr. Gruska, a Fellow of ASQ, is the president of the Third Generation, Inc. an Engineering and Management services firm, which assists in the development, implementation and supporting training in Continuous Improvement using Statistical Methods. Prior to this, he served as a Corporate consultant in Statistical Management at the General Motors Corporation. Mr. Gruska has authored papers in the areas of non-normal and multivariate data analysis, measurement systems analysis, and statistical process control. He is an officer of the Statistics Division of the American Society for Quality, is certified by the society as a Quality Engineer and is a licensed Professional Engineer (CA-Q). Greg has been an examiner and a judge for the Michigan Quality Leadership Award.
Bibliography
Cochran, William G., Sampling Techniques, John Wiley and Sons, 1963. Converse, Jean M., Stanley Presser, Survey Questions - Handcrafting the Standardized Questionnaire, SAGE Publications, 1986, Beverly Hills, CA. Deming, Dr. W. Edwards, Out of the Crisis MIT, 1986. Some Theory of Sampling, Wiley, 1950; Dover, 1960.
Research, Wiley, 1960. Sample Design in Business
Gruska, Gregory F., Analytic and Enumerative Studies Class notes from Deming Seminar for Teachers and Consultants NYU March 1990,91. , Enumerative vs. Analytic Transactions - Quality Concepts '90, ESD/ASQC Automotive Division, 1990. Orlich, Donald C., Designing Sensible Surveys Redgrave Publishing Company, 1978, Pleasantville, New York.
About the authors Maureen Heaphy, a Fellow of ASQ, is a principal consultant in The Transformation Network, Inc. pro-
Focus groups are used widely because they provide useful information and offer the researcher a number of advantages. This information and the advantages of the technique come at a price, however. Advantages of Focus Groups Focus groups provide a number of advantages relative to other types of research: 1. 2. Focus groups provide data from a group of people much more quickly and at less cost than would be the case if each individual were interviewed separately. They also can be assembled on much shorter notice than would be required for a more systematic, and larger survey. Focus groups allow the researcher to interact directly with respondents. This provides opportunities for the clarification of responses, for follow-up questions, and for the probing of responses. Respondents can qualify responses or give contingent, answers to, questions. In addition, it is possible for the researcher to observe nonverbal responses such as gestures, smiles, frowns, and so forth, which may carry information that supplements (and, on occasion, even contradicts) the verbal response. The open response format of a focus group provides an opportunity to obtain large and rich amounts of data in the respondents' own words. The researcher can obtain deeper levels of meaning, make important connections, and identify subtle nuances in expression and meaning. Focus groups allow respondents to react to and build upon the responses of other group members. This synergistic effect of the group setting may result in the production of data or ideas that might not have been uncovered in individual interviews. Focus groups are very flexible. They can be used to examine a wide range of topics with a variety of individuals and in a variety of settings. Focus groups may be one of the few research tools available for obtaining data from children or from individuals who are not particularly literate. The results of a focus group are easy to understand. Researchers and decision makers can readily understand the verbal responses of most respondents. This is not always the case with more sophisticated survey research that employs complex statistical analyses.
3. 4. 5. 6. 7.
Limitations of Focus Groups Although focus groups are valuable research tools and offer a number of advantages, they are not a panacea for all research needs and they do have their limitations. Many of these limitations are simply the negative sides of the advantages listed above: 1. The small numbers of respondents that participate even in several different focus groups and the convenience nature of most focus group recruiting practices significantly limit generalization to a larger population. Indeed, persons who are willing to travel to a locale to participate in a one- to two-hour group discussion may be quite different from the population of interest, at least on some dimension, such as compliance or deference. The interaction of respondents with one another and with the researcher has two undesirable effects. First, the responses from members of the group are not independent of one another, which restricts the generalizability of results. Second, the results obtained in a focus group may be biased by a very dominant or opinionated member. More reserved group members may be hesitant to talk. The "live" and immediate nature of the interaction may lead a researcher or decision maker to place greater faith in the findings than is actually wan-anted. There is a certain credibility attached to the opinion of a live respondent that is often not present in statistical summaries. 'Me open-ended nature of responses obtained in focus groups often makes summarization and interpretation of results difficult. The moderator may bias results by knowingly or unknowingly providing cues about what types of responses and answers are desirable.
2.
3. 4. 5.
Thus, we see that focus groups offer important advantages, but that these same advantages have associated dangers and limitations. As noted above, focus groups are used most often as a preliminary stage in a larger research program that includes a larger, more representative survey of the population, or as a means for adding insight to the results obtained from a survey. We should not overlook, however, the cases in which focus groups alone may be a sufficient basis for decision making. One example in an applied research setting would be the identification of flaws or serious problems with a new product or program that would necessitate redesign. Another would be a situation in which there is reason to believe that the group of people - or population - of interest is relatively homogeneous, at least with respect to the issue at hand. In such cases, a small number of respondents is all that is needed to generalize to the larger population. Reynolds and Johnson (1978) provide a useful example of the complementary use of focus groups and survey research. It is true that focus groups yield qualitative data obtained from relatively small numbers of respondents who interact with one another; yet, this is exactly their purpose. There are those who would use focus groups to explore all manner of research questions. This view, however, is as inappropriate as the view that dismisses the focus group as having no utility MODERATOR BIAS IN FOCUS GROUP INTERVIEWS One important aspect of moderator preparation is developing an understanding of the sources and nature of biases that can affect the validity of the focus group data, as well as an understanding of the steps that might be taken to cope with these biases. Moderator bias can be introduced both intentionally and unintentionally. Three different sources of bias that threaten moderator objectivity are: Personal bias: The all-too-human predisposition to welcome and reinforce the expression of points of view which are consonant with our own Unconscious needs to "please the client": The predisposition to welcome and reinforce the expression of points of view which are consonant with those of our clients, those for whom we are doing the research The need for consistency: The predisposition to welcome and reinforce the expression of points of view which are internally consistent. Some examples of how these different sources of bias are manifested in practice include : Most often, by greeting favorable comments with appreciative nods, smiles or reinforcing comments, and by responding to unfavorable comments with indifference, perplexed stares, or body movements which reflect discomfort.
By being patient, permissive and encouraging when someone finds it difficult to articulate a favorable thought, but by providing no such assistance to one who finds it difficult to express an unfavorable position. By initiating a round of questioning with a favorably-inclined respondent, so that a favorable view will set a precedent and context for subsequent inquiries. By failing to probe for contrary sentiments when favorable comments are expressed, but by probing actively when unfavorable comments are articulated. By more actively directing questions to those who seem most likely to hold favorable views, and by ignoring those who seem most likely to hold unfavorable views. By "turning on the charm" so that respondents will tend to go along with the position you have unconsciously conveyed you want to hear. By permitting "out of context" favorable comments, while telling those who offer an unfavorable view out of context that "we'll talk about that later." In periodic summaries of group positions, understating or omitting "minority" points of view
Possessing the appropriate training and experience does not guarantee a bias-free focus group session. The researcher and research sponsor need to take an active role to understand the pressures being applied on the moderator, and to work closely with him or her both during the preparation and post-interview phases to avoid biasing the outcome of a group.
Getting Started
An important aspect of moderator training and preparation involves learning how to deal with situational variables such as disruptive focus group participants, emergent leaders, different focus group sizes, deadlines, and other resource constraints. Personal characteristics, educational background and training, and amount of moderating experiences are important considerations in selecting a moderator. However, there is no one best style for leading a focus group, nor is there a single best "type" of moderator. Rather, both the moderator and the strategy for conducting the interview must be matched with the purpose of the research and the characteristics of the group. One of the many ways in which moderators differ from one another is in the amount of' time they devote to introductory remarks. One very experienced moderator opens each group by explicitly making each of the following points: His name The subject under discussion is He would like to use participants' first names and would like them to use his first name The role that he plans to play, as moderator (keeping the discussion focused on the topic, etc.) Participants are free to speak when they have something to say People should not speak at the same time The group is being tape recorded There is a one-way vision mirror There are observers They are encouraged to talk to one another and not only address the moderator They are encouraged to be candid in their assessments of the concept The moderator has not vested interest in the success of' the concept per se They will not be quoted by name in the report The client will remain anonymous This introduction often takes five minutes or more. Another equally experienced moderator uses some variant of the following: Good evening. Im your moderator. My name is ___________. You will notice from the microphones (pointing) that we are recording our discussion tonight and there is (pointing) a one way mirror. Behind it are some observers. They're there so that they can make notes and talk among themselves without getting in our way. Our topic tonight is The average time required is 40 seconds.
There is really nothing to recommend one approach over another, since neither approach facilitates nor handicaps the ensuing discussion. The principal objection to a long introduction is the time it takes. The 120 minutes normally allocated to a typical group interview are a precious commodity that should not be squandered on anything that does not further the research objectives. The information covered in the longer introduction outlined is not irrelevant, but can be communicated nonverbally by what the moderator does rather than by what is said.
Type of Questions
Main research questions Leading Questions Testing Questions Steering Questions Obtuse Questions
Purpose/Usage Situation
Focus discussion on issues directly at the purpose of the session. Exactly how you are going to ask these questions should be thought out beforehand. Useful for carrying a discussion toward deeper meaning and are especially useful if the group seems hesitant to pursue it. Formulate the questions using the group's words and ideas and by asking, "Why?" Used to test the limits of a concept. Use the group's words and ideas to formulate the question, this time feeding the concepts back to participants in a more extreme, yet tentative form, as though you may have misunderstood. Used to nudge the group back onto the main research questions, following its frequent excursions into what it wants to talk about. Often the discussion will go into territory uncomfortable to the group. To further pursue topics into such areas, you need to back the questions off one level of abstraction, allowing the group to discuss other people's reactions or opinions, not necessarily their own: "Why do you suppose somebody else would feel this way?" Questions that have a factual answer and permit the group to answer without personal risk. These questions can be useful for neutralizing emotionally charged groups or discussions. Used to ask for opinions surrounded by personal feelings. Feel questions ask participants to take risks and expose their personal feelings. They are the most dangerous and most fertile of question types. The rule to remember here is that every person is entitled to his or her feelings, and no one else can disagree with or discount them, though many will try. Used to get a group talking, comfortable with each other, or refocused on a key question. They generally take the form, "Please take the 'index card in front of you and write down the single idea that comes to mind regarding this issue." Often the best question is no question. Many group leaders tend to fill in every void in the discussion. Simply waiting for a response allows those who are a little slower or uncertain to formulate their ideas. Source: Wheatley & Flexner (1988)
Example Interview Guide for New Car Purchasers 1. All of you here been involved with purchasing a new automobile. As a way to get started, let's talk about the factors that influenced your decision to buy the car that you purchased (if not raised by the group, probe for the importance of each of the following):
Dealership Sales Personnel Type of vehicle desired and purpose it would serve Friends, relatives, or other significant individuals Prior experience with make or model or with dealer Service expectations Special packages (convenience items, appearance items, safety items toys, etc.) Price Advertising
Do you feel you got a good deal in your last purchase? (Probe: Why or why not'? What makes you think this?) 2. If you could change the vehicle in any way, what would you change? (Probe: Why? would this change make a difference?) Why
3. Some of you bought American automobiles, while others bought foreign cars. Do you see any differences in American versus foreign-made cars? (Probe: Why do you think these differences exist?) 4. What do other people think of your vehicle? (Probe: What kind of comments? Were this comments pleasant or unpleasant? Why?) 5. What do you expect of the automobile after the sale? (Probe: How many of you feel these expectations have been met?) 6. Is there anything else about your purchase experience that you would like to share that we have not yet touched upon?
10
Source: Adapted from John M. Hess (1968), in R. L. Ring (Ed.), New Science of Planning, p.194. Reprinted with permission.
11
5.
candid because the focus is on the group rather than the individual; the respondent soon realizes that the things he or she says are not necessarily being identified with him or her." Spontaneity: Since no individual is required to answer any given question in a group interview, the individual's responses can be more spontaneous, less conventional, and should provide a more accurate picture of the person's position on some issue. In the group interview, people speak only when they have definite feelings about a subject and not because a question requires a response.
Sponsor Advantages 1. 2. 3. Serendipity: It is more often the case in a group rather than an individual interview that some idea will "drop out of the blue." The group also affords the opportunity to develop it to its full significance. Specialization: 'Me group interview allows the use of a more highly trained, but more expensive, interviewer, since a number of individuals are being "interviewed" simultaneously. Scientific scrutiny: The group interview allows closer scrutiny. First, the session itself can be observed by several observers. This affords some check on the consistency of the interpretations. Second, the session itself may be tape-recorded or even videotaped. Later detailed examination of the recorded session allows additional insight and also can help clear up points of disagreement among analysts. Structure: The group interview affords more control than the individual interview with regard to the topics that are covered and the depth with which they are treated, since the "interviewer" in the role of moderator has the opportunity to reopen topics that received too shallow a discussion when initially presented. Speed: Since a number of individuals are being interviewed at the same time, the group interview permits the securing of a given number of interviews more quickly than do individual interviews.
4. 5.
12