My goal in this chapter is to give you enough information so that you could design your own sample if you were doing a survey. the odds are very good that several of you will be asked to do this kind of thing in your work life in the years to come.

Here is a general outline of what I am going to talk about. First, if we are doing a survey, we must decide what kind of units to observe. Second, we need to decide how many to observe--that is, sample size. Third, we need to know precisely how to choose those units--sample selection.

How we answer the second and third questions will have a big impact on the error in our findings. In general there are two kinds of error in survey research: sampling error and bias. Sampling error is good. It is good because if we choose our units correctly, we can control it. That is, we can estimate what it will be and make it smaller if we feel we need to before we do the survey.

On the other hand, bias is bad. We can never be sure how big it is. And it is hard to eliminate. All we can do is try and minimize it as much as possible. Bias can enter survey research in three ways. 1) Sample bias can wreck findings if the units for study do not represent the population. Therefore our goal is to minimize sample bias and maximize the chance that the sample represents the population. 2) Question and questionnaire bias can also cause unknown amounts of error in findings.  3) Interviewer bias is when those asking the questions cause also distort results. You should know that while we can never be sure of how much error is caused by bias, sometimes people create bias on purpose in order to find support for some goal they have.

In this chapter we will sampling error and sample bias. In a later chapter on surveys and how to write questions, we will focus in minimizing question and questionnaire and interviewer bias.

What kinds of units to observe?

Generally speaking, one should choose units that are as close as possible to those specified in the hypothesis. So if the hypothesis is about citizens, then the units observed should be citizens. If it is about cities, then cities should be observed.

Sometimes, as I showed you earlier, you can substitute aggregate data for individual data. For example, if you need to know about voting behavior, ideally you would do a survey of voters or an exit poll. But you may be able to substitute precinct data--IF you know the characteristics of the precincts and the precincts are homogeneous--that is, all of the people in the precinct are pretty much alike in ways that are important in your study (e.g. white, middle class, working class, Hispanic, or whatever). Then you can compare precincts with different characteristics and assume that the votes represent how people with these different characteristics are voting. But the more heterogeneous the aggregate units are, the less you can assume about who in each unit is doing what.

How Many? Sample size and sampling error

At this point, I will just give you some formulas. In class we will go over them and I will show you how they are applied and a little about how they are related to each other.

Here is the formula of sampling error for proportions AFTER a survey is completed:

     where p is the proportion in decimal form, e.g. .35 of the sample say they voted
              n is the sample size
              N is the population size

This is great, except that it is only good after the survey is over! What we really need is a formula that will help us plan a survey and keep sampling error in an acceptable range. if we make certain worst case assumptions (like the variability thing that Corbett talks about), this formula can be reduced to the following (I will show you in class how we get to this one):

Here is the formula for sampling error for proportions BEFORE a survey is performed:

        Sampling Error for Proportions = [1/sq rt (n)]   [(N-n)/(N-1)]

        where n is the sample size
              N is the population size

You will note that p is not in this formula. That is because we do not know what p is until after it is done!

The last factor in the formula, [(N-n)/(N-1)], is call the Correction Factor. It can also be dropped if the sample is small relative to the population. The usual practice here is that if the sample is LESS than 10% of the population, you can drop this factor. Why? the ansser is that in this case it does not make much difference in the error estimate. So in a sample of 300 out of a population of 10,000 people, the factor is inconsequential and can be ignored. On the other hand, if you are doing say, 350 out of the 3200 USCA students, then it should be kept.

When can these formulas be used? They can only be used when each unit in the population has an equal chance of being chosen (when we have a sample that approximates a random sample, which we discuss below). Otherwise, the formulas are inappropriate and have little to do with actual error.

How accurate are these formulas? They are all calculated at what is called the 95% significance level. That means that a one in twenty chance exists that the actual proportion for the population will be outside the + or - confidence interval. Another way to look at it is that 19 of 20 times you will be within the sampling error of the truth. Of course, except in the case of exit polls that ask for whom you voted, we do not know what the truth is! So we must just have faith in the laws of sampling and do the best we can to minimize bias--the other source of error.
 

How to choose units? Types of Samples

Once you decide how many, you have to come up with a scheme for choosing them. One could have a whole course on sample selection--indeed, such courses exist at the graduate level. For our purposes, we will keep things relatively simple.

Generally, all sample designs fall into one of two categories: haphazard samples or probability samples. Haphazard samples are pretty terrible in the sense that they ignore the problem of bias and consequently usually have a great deal of bias in them. There are several special kinds of haphazard samples.

Convenience samples are composed of units that the researcher finds convenient to find and survey. The "person-on-the-street" interviews that one sees so frequently are convenience samples that are rarely representative of all the diverse people who exist. The people interviewed are chosen because they are easily found (often at malls these days) and easily interviewed. Years ago in the early days of survey research even the professional pollsters used a form of convenience sampling. They called it "quota sampling," and thought it was a way of being more scientific. What they did was to try and force the sample to look like the general population in specified ways: so many women, so many in each age group, and so on. But after they did that, the interviewers went out and found people in each group who were convenient to find. It was still a convenience sample and was likely biased in that certain kinds of people are more convenient to find than others.

Self-selected samples are even worse, if that is possible. Here the interviewer waits for the sample to come to her. Usually the motivation is caring enough to participate. Straw polls in which one calls in or clips something out of a newspaper and then sends it in, or uses a form on a Web page, are all versions of straw polls. The bias is that those with the most intense feelings are the most likely to volunteer themselves. That usually means a bias in favor of more extreme opinions.

The infamous Literary Digest survey of 1936 used a haphazard sample. It had elements of both a convenience sample and a self-selected sample. It was convenient for the magazine to get addresses from auto registrations and from telephone books. Then those who could afford the magazine or had strong feelings about it self-selected themselves to respond. Despite this, the survey got it right in the previous two elections. The reason is that the parties were not aligned nearly as much along economic lines in 1928 and 1932. The splits were more regional. And in 1932, everybody, whether rich or poor, was unhappy with Hoover. But in 1936 the poor had moved to the Democratic Party and the wealthy were very angry with FDR, feeling that he was a traitor to his economic class. So the vote was more along class lines. And as a result, the sample had very few lower class people in it.

Probability Samples are those in which one knows the probability of each unit in the population being chosen. If and only if the probability of each unit in the population being chosen is equal, we can use the sampling error formulas that were given above. Sometimes people will call this a simple random sample. Simple random samples do have this characteristic, but in fact they refer to a particular method of choosing a probability sample. So let's start with that kind of sample.

A simple random sample requires a complete listing of all units in the population. Each unit is given a unique number:1, 2, 3, and so on. Then a random number table or generator from a computer program is used to select the numbers of the units to be chosen. I will provide you with a random number table and go over some examples in class. Unfortunately, often one does not have a numbered list of all units in the population, so this is not really a practical method most of the time. Choosing the numbers out of a bowl or basket (like in the old draft lotteries) is a variation on this method. We will do an exercise on this in class with M&M's to show how well the formulas work.

A systematic sample is a very practical way of choosing a sample when you have a list but it is not numbered. Say, for example, you have a customer list for a bank, of say 2500 people. If you want a sample of 400, you then choose every k^th one, where k is N/n, or in this case, 2500/400 = 6.25, which we then round down to 6 so that we don't run out of people in the list at the end to get the 400. The first one chosen would be randomly chosen from the first 6, and then you count every 6^th one from there.

If you have a really long list with pages--like a phone book--you can do a slight alternative on this by choosing the r^th name (where r is some low random number so that you do not have to count very much) on every k^th page (or column), so that you get the desired number and get all the way through the phone book.

There is a slight bias in this in that people at the end may have a lower probability of being chosen. In the example of 400 from a list of 2500, choosing every 6^th leaves roughly the last 100 with no chance of being chosen. the best way around this is to go ahead through the list and choose a few extra so that all had an equal chance. In this case you would end up with a sample of about 416.

In telephone interviewing firms often use a technique called random digit dialing. This technique uses a random number table (usually a computer program) to generate telephone numbers so that all possible numbers has an equal chance of being chosen. A real plus here is that unlisted numbers are just as likely to be selected. A systematic sample from a phone book misses unlisted numbers. This works pretty well if everyone in the population has the same exchange or it is a national sample with thousands of possible exchanges and hundreds of area codes. But it does not work very well in most other cases because you have a lot of unused exchanges or area codes and therefore a lot of generated numbers that are not usable. So what we usually do is to combine this method with stratification, our next topic.

When you stratify a sample, you make sure that you select certain proportions of the sample in each strata of the population. This is only used in combination with other techniques. So for example, in random digit dialing, we might stratify Aiken City area numbers into several strata corresponding to each of the exchanges in the area (641, 642, 648, 649 and so on). Then once you select the number in each strata you want, you then generate the other 4 digits using random numbers. How do you know how many from each strata (exchange in this case) to use? You may have to take a systematic sample from the phone book to see what the proportion of numbers in the phone book each exchange is. One need only assume that unlisted numbers have the same proportions.

Stratification is often used in deciding whom to talk to once one reaches a working number. If someone calling you says that they need to speak with the oldest/youngest/second oldest male/female in the household, you know that this is professional survey that is stratifying by age and gender. That is typical. An easy and useful variation on this which chooses all genders and ages proportionally is the "Last Birthday" method. You simply ask to speak to the person (adult) in the household who has had the most recent birthday. This may sound pretty silly, but if you think about it, it is a quick and excellent off-the-wall way of randomly choosing a member of the household with whom to talk.

Even if you stratify by exchange, one still often ends up with many numbers that are not usable. One way around this is to take advantage of the fact that the phone company assigns numbers in clusters, so that if one is good, the numbers around it are usually also good. The "Plus One" method of  number selection in telephone surveys works in two steps. First, you select a sample using the systematic sample technique or some variation thereof. Then, in order to allow for unlisted numbers to be chosen (which can run as high as half of all households in some urban areas), you simple add one (or two or three or four or five) to the number. And you have a sample of mostly working numbers so that all numbers in the population are roughly equal in chances of being chosen. This is a very good method, especially when you are having to work through several listings, say all those in Aiken County!

Cluster Samples are used in face-to-face surveys in order to save time and money. It would be impractical to travel to 400 widely scattered homes throughout South Carolina. So instead, using census tracts and block maps, we might randomly choose 25 tracts and then choose two blocks in each tract so that we have 50 blocks to travel to. Then we do 8 interviews in each block area (choosing houses systematically). Often this is combined with some stratification so that we get the right mix of urban, suburban, and rural, for example. Needless to say, this is a very complicated technique that requires a great deal of planning.

Weighted Samples are chosen in situations where one needs to compare subgroups, some of which would be too small to use if the groups were chosen in proportion to their actual size. Say, for example, that one wishes to compare the attitudes of social science majors (history and political science, sociology, psychology) with all other majors here at USC Aiken. If we were to choose a sample of 200 using some method that made each student equally likely to be chosen, we would end up with something in the range of only about 20 social science majors in our sample at the most, since they number only 200 out of roughly 2,000 students. Obviously one could not make a very good comparison with such a small subsample with such a larger subsample! The sampling error for the roughly 20 social science majors would be about + or - 22% (using the formula 1/ sq rt of n). If making some kind of comparison was the major point of the study, we might wish to weight the two subgroups equally, taking a systematic sample of 100 of each group. That would have about a + or - 10% sampling error for each subgroup (again, using the formula 1/sq rt of n).

Let's take this one step further. Suppose we make our comparison. Then we decide that we would also like to make some projection to the student body as a whole. For example, suppose we compared attitudes toward a career in public service, and found that 60% of the social science majors would consider a career in public service as compared to only 25% of the other majors. Now we would like to know how the whole student body feels about a career in public service. How would we answer this question?

We must re-weight the subsamples so that they are in correct proportion to what they should have been. Here is the method you use. 1) Compute the number of each subgroup that should have been in the subsample were it proportional to the population. 2) Divide that by the number that actually was in each subgroup. This gives you the re-weighting factor for each subgroup. 3) Re-weight the subsamples by  multiplying the number of units in each subgroup by the re-weighting factor. 4) Add these re-weighted units together. 5) Re-compute the proportion of the whole sample that gave that particular response.

Yes, I know that sounds like IRS Form 1040 instructions on computing the capital gains tax. But I think an example will make it clear--and it should even make intuitive sense--something that has always eluded me in the form 1040. Let's go back to our make-believe example.

If you remember, 60% of the social science majors (out of the subsample of 100) would consider a career in public service as compared to only 25% of the other majors (also out of a subsample of 100). So the "number that was actually in each subgroup" was 100 each. However, there SHOULD have been 20 social science majors and 180 other majors, based on the fact that social science majors were 10% of the population (200 out of 2,000, if you remember). So 20 and 180 are the "should have been" numbers for each subsample. Using these numbers, the re-weighting factor for the social science majors is 20/100 = .2, and the re-weighting factor for the other majors is 180/100 = 1.8. This means that when we re-weight, the social science majors, who were over-counted, will be recounted for .2 persons each, and the other majors, who were actually under-counted, will be re-weighted as 1.8 persons each. The rest can be pretty easily set up in a table.
 
 

Group	% yes	# yes	x  re-weight fact	new #  yes	new % yes
Soc Sci maj	60%	60 of 100	60 x .2	12 of 20
other maj	25%	25 of 100	25 x 1.8	45 of 180
totals				57 of 200	57/200 = 29%

You can see that had we just added together the original numbers (which is inappropriate), we would have had 60 + 25 = 85 or 85/200 =  43% saying they would consider a career in public service. That obviously overestimates the truth for the whole student body because too many social science majors were in the sample, and social science majors are far more likely to consider a public service career. The other majors should have far more influence on the total because they are a greater part of the overall student body. The percentage computed using the re-weighted students does exactly that. It is much closer to the percentage that would consider a career in public service among other majors.  

Weighted and Re-weighted Samples--an easy way to Re-weight

Here is another way to do this that is simpler. Suppose you have a population in which subgroup B is small, say 10% of the population. Further suppose you want to compare the opinions of the larger subgroup A with the smaller subgroup B. To make the sampling errors about the same for both groups, you might weight the sample so that instead of having only 10% of the sample be in group B, you would weight them equally with 50% in subgroup A and 50% in subgroup B.

To get some actual numbers to illustrate, suppose you find that 25% of subgroup A favors something and 80% of subgroup B favors that same thing. Further suppose that the size of the samples for each subgroup was 200. That makes the total sample 400.

Now you want to reweight the entire sample of 400 so that you can find the percentage of everyone who favors whatever you asked about. 
 

 You just compute the reweighting factor(rwf) for each cubgroup by dividing the proportion in the actual population by the proportion in the weighted sample. Then you multiply that times the numbers you found that gave some response in the two subsamples to get what those numbers would have been had we sampled in the correct proportions. Add the numbers together and divide by the total sample size to get the actual proportion who would have given that response.

After that the only thing left is to calculate the sampling error for the sample of 400 in which p = .305. And you know how to do that!

Assignment for next class:

1. How large a sample would you need in order to end up with a sampling error or no more than + or - 5%, if the population is 50,000 (which is about the number of adults over 18 in Aiken County)?

2. Suppose you chose a sample of 500 adults out of this population of 50,000 in question number 1. And further suppose that your survey found that 30% said that they were registered to vote. What is the sampling error for that 30% figure?

3. Suppose that instead of 30%, 45% said that they were registered to vote? What is the sampling error for that figure?

4. Suppose that the % who said they were registered to vote is 50%. What is the sampling error for that finding?

5. One more--suppose the percentage is 55%--what is the sampling error for that figure?

6. Compare the numbers you found for questions 2-5. Do you see a pattern? Describe the pattern. What does it suggest about how to estimate sampling error when you don't know what the proportion (p in the furmula) will be? (This is what Corbett was talking about when he had that confusing discussion on "variability," by the way.)

7. Suppose you were asked to do a survey of business majors here at USC Aiken. There are 450 business majors.
    a. What would the sampling error be if you were to do a sample of 150 of them?
    b. If you have a list that can be computer generated with their names and telephone numbers, how would you design the survey of 150? Describe the best process.
    c. Suppose your friend, who is a business major, offers to give out the questionnaire in his classes, which have just over 180 students in all of them combined. Then what would the sampling error be if he got 180 questionnaires completed?

8. Take a page from the current Aiken City area telephone directory. Pick the page that comes closest to the middle two digits in your social security number. Use a systematic sample of 100 to produce estimates of the relative number of numbers at each exchange on that page.

9. Suppose you use a weighted sample in doing a survey of male and female USCA students so that the number of males and females are equal. You know that 65% of the student body is female. The question is whether students want to start intercollegiate football here at USCA. Each subgroup has 200 students in it and was chosen using a sytematic sample so that each male and female student had an equal chance of being chosen within their subgroup. The finding is that 45% of the females in the sample say "yes" to football and 65% of the males say "yes."
    a. Based on these figures, is there likely to be a real difference in how males and females feel about football in the actual population of all students?
    b. Reweight the two subsamples and recombine them so that we can project how the student body as a whole feels about football. Account for sampling error.