I want to accomplish three different things in this chapter. First, I want to introduce you to the three major kinds of surveys and discuss the tradeoffs in deciding which kind of survey to do. Second, we will talk about response rate, why it is important, and how it is calculated. Third, we will discuss how to minimize three kinds of bias that can creep into surveys: interviewer bias, question bias, and questionnaire bias. We have already talked about the other kind of bias, sample bias, in the last chapter. And you should already know that our goal is to minimize each kind of bias.

I. Types of Surveys

Although there are practically an infinite number of different variations of different surveys that one can perform, virtually all fall into one of three major categories: face-to-face, telephone, and self-administered. Let us consider each in turn and then do some comparisons.

1. Face-to face. The classic face-to-face survey is the door-to-door survey. However, any survey in which an interviewer asks questions to the respondent in person is a face-to-face survey. This can be done in a work setting, in schools, or in people's homes. A few years ago I helped in a survey of workers who worked in a set of office buildings. We surveyed them in the morning as they entered the buildings (using a combination of a cluster sample and a systematic sample). But the interviews were face-to-face.

2. Telephone surveys. This is the most common form of survey today. It is a compromise between the other two kinds in many ways. But it is getting more difficult to do well, largely because of telemarketing and devices such as caller id and answering machines, and most recently. cell phones. Happily for those of us in the survey business, the "Do Not Call List" has helped reduced some of the resistance for legitimate surveys we were seeing a few years ago when people were getting several marketing calls every evening. But that is about all that has improved. Cell phones are today the greatest challenge to telephone surveys.

3. Self-administered surveys. Any questionnaire that the respondent fills out him or herself is a self-administered survey. Your teacher evaluations are self-administered. It works well when you have a captive audience. One of the most common forms is the mail survey, which has a few advantages but many disadvantages, as we shall see. The most modern version of self-administered surveys are on-line surveys, or computer surveys. These may be the surveys of the future as soon as we figure out how to get a representative sample that has the important characteristics of a random sample. I expect the first step will be creating stratified samples and then supplementing with some telephone surveys to add in demographic groups that do not have computer access, mainly elderly and poor people.

II. Response Rate

Response rate is an important consideration in any survey. It can be measured in several ways. Ultimately, what we want is the percentage of units in the population targeted for selection who actually completed surveys. However, that number is not always easy to find.

For example, suppose you call a telephone number several times but never get an answer in a survey you are doing of registered voters. If this is a business number or if no one living there is a registered voter, it is not a member of the population and therefore should not count as a nonresponse in the response rate. Suppose you do reach the household but are unable to determine whether the potential respondent is registered before she refuses the interview? Again, you cannot be sure how to count this person. All this yields several possible estimates of response rates. You could calculate the percentage of those actually contacted and determined to be potential respondents who actually completed the interviews. That would be a liberal estimate of response rate. Or you could use a conservative estimate, the percentage of the units originally chosen that responded, only discounting those which were determined to be ineligible (businesses or people not registered to vote in our example). Otherwise, if we can't reach them, we assume that they were part of the target sample. Generally we should use the conservative estimate.

Response rates are important when certain kinds of units are more likely to be surveyed than other kinds of units. Say for example, wealthy people use answering machines or caller identification to screen their calls and are therefore less likely to be interviewed. That creates a bias in which wealthy people are less likely to be chosen. You can see how that creates a problem. However, if refusals and non-responses are randomly distributed across all groups, then a low response rate is not a problem. A random sample (those contacted) minus a random sample (those who refused) equals a random sample. Unhappily, random refusals and non-responses are usually not the case.

You can see how much distortion this can create with a little "pretend" example. Suppose we draw a sample of 100 likely voters. Suppose that in the population of all voters 30% will actually vote for Ross Perot, the Reform Party candidate back in 1992. But suppose that these Perot voters are so distrusting that two thirds of them refuse to talk to interviewers. Suppose that those who will vote for the other candidates do not refuse. So in our initial target sample or 100, 20 of the 30 Perot voters we try to interview refuse to be interviewed. That leaves us with 10 Perot voters. We then replace them with 20 new target voters to be interviewed. Of these, 6 should be Perot voters (.3 x 20 = 6). But 2/3's of these 6, or 4 also refuse to be interviewed. So that gives us 2 more Perot voters in our sample. We replace those 4 and about 1 of these new potential interviews is also a Perot voter. But she refuses and is replaced by a non-Perot voter. So we end up with 12 Perot voters, or 12% of the sample, when an unbiased estimate that would have come with a 100% response rate would have given us 30% Perot voters. That is a pretty big distortion! And that is pretty much what happened in surveying for the presidential election in 1992, though our numbers are inflated a bit. Perot did get undercounted and pollsters knew it, but were not sure how big this sample bias was.

Sometimes researchers will talk about refusal rates. This is just the opposite of the response rate. To put it in algebraic form, 100% - Response Rate = Refusal Rate. This may be conceptualized a little differently however. It could be the percentage who actually refused, and would not count those who were never successfully contacted. In that case the proper term to use would be non-response rate.

So here is the bottom line on response or refusal rates. Make sure you explain exactly how you calculate whatever rate you use. That is critical for replication! I generally prefer to just use the response rate and use a conservative estimate. .

What do you do when you have refusals or units that were target for interviewing but you were unable to reach? You have two choices. You can just use what you have and end up with a smaller sample--this is called sampling without replacement. Or you can replace them with other units--sampling with replacement. Mathematically speaking, as long as the additional units are chosen so that each unit in the population has an equal chance of being chosen, it makes little difference.

Knowing likely response rates affects how you determine your target sample size when planning a survey. For example, if you expect a response rate of 50% in a mail survey and you want a sample of 300, you should select 600 in order to end up with 300 at the end. The same would be true of other kinds of samples, with one interesting exception. In the "plus one" method of telephone survey sample selection, you just go ahead and start with 300. Then for a refusal of a number, you just add another digit. So if 642-3341 refuses or is unable to be reached, you replace it with 642-3342. The only time this does not work is if you get into a group of nonworking numbers. If this happens a number of times, you may have to go back and choose another sample to add on to your original sample. But that is ok so long as it is also a probability sample in which each member of the population has an equal chance of being chosen. A random sample plus a random sample equals a random sample!

Since refusals are bad, how do we maximize response rates? The answer depends on the kind of survey one is doing.

In telephone surveys:

• Call each number at least 4 times at different times and days to maximize the chance of getting someone to answer
• Assume that the respondent wants to do the interview and you must just contact them at the best time for them--so if they sound harried, make a appointment to call back at a more convenient time.
• Let respondent know that the survey is important and will benefit them in some way--"help provide better government or services for your home."
• Associate survey with a respected name if at all possible--"sponsored by USC Aiken or Aiken City government."
• Personal plea for pity--"a student who is working my way through college--appreciate your helping me get my job done" works very well for us.
• Use "fallback statements" to answer questions they may have--must be pre-prepared for interviewers.
• Give positive reinforcement to early screening questions--"Registered voter? Yes. "That's great!!" (Source: Trade Secrets of Telephone Interviewing from telemarketing firms on National Public Radio, 12/1/98). Then move to neutral reinforcement later.
• Ask easy questions first to get respondent committed to the interview.
• Use downward voice inflection ad the end of questions--makes them sound more like a command rather than a question. (Source: Trade Secrets of Telephone Interviewing from telemarketing firms on National Public Radio, 12/1/98)
• Confidentiality if possible
• Say "not selling anything"
• Leave message to return call on answering machine using first name only, and make it sound interesting and friendly--"Hi, this is Gwen! Could you call me back at ###-#### either tonight before 9 pm or tomorrow evening? [downward inflection] Thanks--talk to you then!!"

In mail surveys:

• personally hand address the mailing envelope
• sign letters by hand
• use letterhead of prestigious firm
• opening sentence must give the respondent a reason to respond and make them feel important--"We all want lower taxes and I need your help to make this happen."
• pre-addressed pre-stamped return envelope included
• follow-up post card reminder within a week
• follow-up questionnaire in registered letter in 2 weeks
• short questionnaire with no open-ended questions--just boxes to be checked
• confidentiality if possible
• monetary incentives to participate--a dollar bill enclosed--"We know your time is valuable so we want to pay you a small token for five minutes of your time." OR, better yet, an entry to win a large prize for all returned completed questionnaires--a large screen tv is cheaper than one dollar bills in a thousand envelopes.

In face-to-face surveys:

• several efforts to contact at different times
• well-dressed with name tag with prestigious label.
• make eye contact
• same verbal reinforcements and opening as in telephone interviewing
• not selling
• confidentiality if possible
• can also use financial incentives or prizes for participation

III. Tradeoffs in Different Interviewing Formats

The table below compares the three most common types of public opinion surveys in terms of several important parameters that anyone planning a survey should consider. As you can see, there is no perfect survey. I should point out that these are very rough estimates, because the actual numbers will depend a great deal on the population that is being surveyed and on the number and complexity of questions.
 
 

As you can see from the table, telephone interviews are generally the interview of choice. Mail interviews are too slow, must be very short, must only use very simple questions that require no more than checking a box, have a very low response rate, and take a long time to do. However, mail surveys are cheap. They work well for surveys of the membership of social/business/interest groups when asking everyone's opinion is politically important. Not doing this sometimes leads members to protest that they did not get a chance to have a say.  At the other extreme, face-to-face interviews are generally just too costly, unless you have a captive audience and do not have to do a lot of traveling to get the interviews.

However, things are changing as people become more resistant to telephone calls because of the excesses of telemarketing and other "junk" calls. Response rates have fallen from the 70 percent range to the 40 percent range in just the last decade. We used to get 70% in our telephone surveys here where we have about everything working in our favor, but recently we have fallen into 50s and low 60s. The "Do Not Call List" should help things here over the next few years, but the growing tendency for people to use cell phones as their only phone may begin to cause significant problems. The General Social Survey (which we have available to us) is a door-to-door survey. That is why they are able to ask so many questions. The surveys of the future may involve specially selected Web target populations to represent the general population supplemented by some traditional telephone surveys.

IV. Question Construction

Bias can result from unreliable and/or invalid questions. Sloppily worded questions that are subject to multiple interpretations are unreliable and an create bias in unknown ways. More typically, invalid questions that push the respondent toward a certain kind of answer creates bias in predictable ways.

1. Avoid loaded terms or phrases:

    Do you support the Reagan administration's request for a major increase in the U.S. defense budget to counter the massive Soviet arms buildup? (Republican National Legislative Action Survey, 1982)

    Do you favor or oppose Republican efforts to make huge increases in the current defense budget? (DNC 1996 Presidential Poll)

    Do you agree with Barack Obama and Nancy Pelosi's efforts to impose massive tax hikes on the American people? (Republican National Committee 2010 Congressional District Survey)

     Do you believe that American business and industry will be able to compete in the world economy if the Obama Administration bends to pressure from radical environmentalists and implements draconian regulations on emissions, energy consumption and transportation beyond what is required  in other industrialized nations. (Republican National Committee 2010 Congressional District Survey)

2. Avoid "motherhood and apple pie" or silly questions:

    Do you feel it is very important to institute common-sense legal reform to stop excessive legal claims, frivolous lawsuits and overzealous lawyers? (National Republican Congressional Committee poll, 1995)

    Agree or Disagree? Sex crimes, such as rape and attacks on children, deserve more than mere imprisonment; such criminals ought to be publicly whipped, or worse. (Adorno, The Authoritarian Personality). This particular example was found to be biased toward lower educated people. Well educated people knew that agreeing with outdated modes of punishment was a sign of lack of education, and were therefore more likely to disagree, however they really felt.

3. Avoid response set bias by reverse wording several agree/disagree format questions when they occur in groups. Resonse set bias can occur in two ways. "Acquiescence" response set bias occurs in lower educated people who tend to feel intimidated and and as a result try to please the interviewer by agreeing with all statements. Occasionally a well-educated person will be a "nay sayer" and disagree with almost everything just for the sake of argument. A good interviewer can tell when this is happening and can terminate the interview or it can be thrown out as not useable.

4. Avoid touchy matters on which the respondent is unlikely to tell the truth. People want to be liked and generally give the socially desirable response. That is why so many more people say they voted than actually did vote.

    How often do you smoke pot?

    Did you drink alcohol on a daily basis during your pregnancy?

    Do you cheat on tests in school?  

5. A closely related problem is questions that call for subjective self-evaluations--avoid them. Again social desirability comes into play here. Make them as objective as possible.

    Do you generally vote when there is an election? (poor)
    Did you vote in the election last month? (better)

    Are you a good student? (poor)
    What is your grade point average at present? (better)

    Are you a racist? (very poor)
    Agree or disagree? Blacks don't get ahead in life because they don't work as hard as people in other ethnic groups. (better--this is called "social racism," by the way)

6. Avoid long questions--they are unreliable--shoot for 15 words or less. This is a tough rule to always follow. It is especially tough on "balanced alternative questions" that try to give two sides to an issue and then ask which side the person more agrees with. These questions usually say something like: "Some people say that ... while other say that .... Which side do you agree with?" regardless, the 15 word rule should be a goal. Short is especially important when the questionnaire is self-administered. Longer questions can work if a skilled interviewer reads them and carefully emphasizes the key ideas.

The example below is an actual question sent to my by a former student who worked for the GAO and recognized the proposed question as problematic. I was being pushed by Republican members in the U.S. House. Here is it. By the time you get to the end, you forget what was at the beginning.

There is a philosophical debate as to whether GAO should be an organization consisting of many highly specialized employees with great expertise in narrow issue areas, supplemented by a few generalists with a broad range of knowledge, or an organization with a few highly specialized employees in each issue area, working with a large number of generalists who have a broad knowledge of many issue areas. Proponents of the specialist approach point to the high level of expertise in very difficult technical fields that is fosters, providing GAO with an institutional knowledge and memory unparalleled in government. Proponents of the generalist approach argue that high degrees of specialization lead to organizational inflexibility, while a large number of generalists will give GAO flexibility to respond to changing demands. Which philosophy best reflects you views?
    ___ many specialists, few generalists
    ___ many generalists, few specialists
    ___ equal number of specialists and generalists
    ___ don't know/no opinion (House Republican Conference member survey, 1996)

7. Avoid compound questions. A compound measure consisting of several questions is fine, but keep each question separate.

    Do you believe that Social Security and Medicare should be targets for major cuts in the battle to balance the nation's budget? (1996 DNC Platform Poll) This should be two separate questions.

8. Use filter questions or built-in filters to eliminate "door-step' opinions. You only want to ask people about things on which they have an opinion. So eliminate them if they have not "read or thought about the debate on whether patients should have the right to sue HMO's" or give them a graceful way to choose the answer " I have not had time to think about that issue."

9. When possible, use questions that have been well-tested by other researchers. If this is not possible, then pretest or use a focus group to see how questions are being interpreted. that is why we pretest all our questionnaires on family members and do a few trial surveys before we print a final version of the survey.

V. Questionnaire Construction

Just as individual questions can cause problems, so can the overall layout of a questionnaire. Here are a number of areas of concern.

1. Branching instructions or interviewer instructions. Anytime you have a question that is contingent on another question, you must make sure that the instructions are clear as to when to ask it. Avoid these altogether in self-administered questionnaires if at all possible. I like to place interviewer instructions in brackets and in caps to distinguish them from questions themselves. [IF "YES," GO TO QUESTION 6]

2. Include codes when possible and use higher numbers for higher amounts of whatever it is that you are measuring, if it is at the ordinal or interval level of measurement.

For example, consider the following question designed to measure support for sending more troops into the war in Afghanistan. Note that higher support has higher codes for this ordinal measure.

How to you feel about sending more American troops to fight the war in Afghanistan? Do you  

5) strongly support sending more troops         4) moderately support             3) have mixed feelings         2) moderately oppose   1) Strongly oppose        0) or have you not had time to think about this issue

3. Stick to close-ended questions where possible--that greatly simplifies coding and analysis later on. If a question is open-ended, make sure that the interviewer writes down the answer as completely and accurately as possible.

4. Multiple response questions should be coded as a series of dichotomous yes/no questions.

    What kinds of movies do you like?
        horror?     __ yes       __ no
        mystery?   __ yes       __ no
        romance?  __ yes       __ no
        action?      __ yes       __ no
        other? [SPECIFY] ____________________

5. Separate answer boxes/blanks so that it is clear which one goes with each question.

    If the election were held today, for whom would you vote in the 2000 presidential election?   __ Dole __Buchanan __Bush __ Forbes __ Bradley __ Gore
(This is a very bad layout!)

6. Question interaction. If you are asking open-ended questions and close-ended questions on the same subject, you should use the open-ended questions first, because the answers to the close-ended questions can suggest answers for the open-ended questions. Separate question that can influence each other, or randomize the order so as to minimize consequences.

So for example, if we are asking about whether a respondent thinks the U.S. health care system needs reform, we should ask that question before we ask about a series of proposals designed to address problems. Asking that series of questions suggests to the respondent that the system is indeed broken and in need of reform. This is precisely what unscrupulous people do in designing "push polls," which we will talk about shortly.  

7. Keep questionnaire as short as possible. Tired respondents give unreliable answers. Many clients I have dealt with don't really know what they need to know. They often use surveys as a defense against decision making and against hard thinking before doing the survey. So spend a lot of time forcing them to do some hard thinking about exactly what it is that they really need to decide and how answers to very specific questions will help them decide. The rule I use is "actionability." Don't ask a question unless you know how an answer will help you take some action. Don't just ask it because it is interesting, or it might be nice to know. This is a hard rule for even me to follow in that I am interested in finding out about a lot of things. We will try to follow this rule in designing our survey. We need all questions to help us in testing some hypothesis which we think important.

VI. Survey Ethics

When you ask people for their opinions and they let you into their lives, you have certain professional obligations to them. You are a representative of your profession. Being ethical is an important of the reputation of those in your profession. Here is a partial list of survey ethics.

• Don't unnecessarily pry into personal lives--remember actionability.
• Don't waste people's time--it is valuable, just like yours is.
• Promise confidentiality if at all possible and keep it if you promise--id codes are for the purpose of figuring out who responded and recalls--don't use them beyond that.
• Don't force people to answer questions they object to.
• Use objective unbiased questions--a special word here--DO NOT use "push polls." This is a particularly insidious poll used by candidates whose purpose it is to suggest nasty things about opponents that may well be untrue and then asking how that might affect the respondent's opinion of the opponent to see how effective the smear is. The purpose is not to measure but to change public opinion. I consider it sufficient reason to vote against someone who does this. Public education and condemnation and negative reaction is the best antidote.

VII. Interviewer Training

The best way to avoid interviewer bias is to train and properly reward interviewers for their hard work. What follows is an outline of the steps in interviewer training. I will go over the specifics in class. This is what we will do in preparing for our survey.

Steps in Telephone Interviewer Training

I. General Training

   A. What makes a good interviewer
      1. Use of instruction sheets (call-sheet, fall-back, and questionnaire)
      2. Quality over quantity
      3. properly completed interviews (4 to 1 ratio)
      4. pleasant neutral voice -- not telephone therapy
      5. probing without bias to maximize reliability and validity

   B. Handling refusals -- being politely persuasive
      1. Assume that every respondent needs incentives
      2. Good timing -- reschedule if seems bad
      3. Pleading -- personalize -- help me as a poor student
      4. Individual questions -- allow to skip uncomfortable questions
      5. provide reassurance with sponsor's name and number

   C. Role playing on minimizing refusals

   D. Supervision
      1. Monitoring
      2. Validate completeness by examining questionnaires
      3. Verify by calling selected respondents to ensure interview properly done

   E. Productivity (4/1 completion-refusal ratio) and Payment (1.5 x min. wage or 2/1 base/productivity incentive system)
 

II. Specific Training

    A. Purpose of survey

    B. How telephone numbers selected

    C. Use of Call-Sheet
       1. Call-backs
       2. When to call (7-9:30 pm Sun, Mon-Thu, Sun afternoon)

    D. Introduction and respondent selection at each household

    E. Fall-back statements

    F. Questionnaire
        1. Read all questions and answers
        2. Review all special instructions
        3. Recording answers
        4. Role playing practice