II.

Sampling and Experimentation:

Planning and conducting a study (10% –
15%)
A. Overview of methods of data collection
1. Census
A census is the procedure of systematically acquiring and recording information about all
members of a given population. From 1950 onward, in the US census forms were mailed to
every address on record with the United States Post Office, including the Armed Services Postal
System. Beginning in 1970, it was made illegal to fail to return a completed census form.
Researchers rarely survey the entire population for two reasons: the cost is too high, and the
population is dynamic in that the individuals making up the population may change over time.

2. Sample Survey
Sampling is selection of a subset within a population, to yield some knowledge about the
population of concern. The three main advantages of sampling are that (i) the cost is lower, (ii)
data collection is faster, and (iii) since the data set is smaller, it is possible to improve the
accuracy and quality of the data.

3. Experiment
Experiments are performed when there are some controlled variables (like certain treatment in
medicine) and the intention is to study their effect on other observed variables (like health of
patients). One of the main requirements to experiments is the possibility of replication.

4. Observational study
An observational study is appropriate when there are no controlled variables and replication is
impossible. This type of study typically uses a survey. An example is one that explores the
correlation between smoking and lung cancer. In this case, the researchers would collect
observations of both smokers and non-smokers and then look for the number of cases of lung
cancer in each group.

B. Planning and conducting surveys

1. Characteristics of a well-designed and well-conducted survey
a. A good survey must be representative of the population.
b. To use the probabilistic results, it always incorporates a chance, such as a random
numbers generator. Often we don’t have a complete listing of the population, so we have
to be careful about exactly how we are applying “chance”. Even when the frame is
correctly specified, the subjects may choose not to respond or may not be able to respond.
c. The wording of the question must be neutral − subjects give different answers depending
on the phrasing.

1
 d. Possible sources of errors and biases should be controlled.
The population of concern as a whole may not be available for a survey. Its subset of items
possible to measure is called a sampling frame (from which the sample will be selected). The
plan of the survey should specify a sampling method, determine the sample size and steps for
implementing the sampling plan and, finally, sampling and data collecting.

2. Populations, samples and random selection

The population definition depends on the problem studied. In Kazakhstan each year about 4,000
newborn babies are abandoned. Kazakhstan has 75,000 children placed in state-run institutions
(orphanages). Suppose you need to design a policy to fight this problem. To find its roots, who
would you survey?
Note also that the population from which the sample is drawn may not be the same as the
population about which we actually want information. For instance, we might study rats in order
to get a better understanding of human health, or we might study records from people born in
2010 in order to make predictions about people born in 2011. A good sampling procedure should
include elements of randomness.

3. Sources of bias in sampling and surveys

Survey results are typically subject to some error. The term "error" here includes systematic
biases as well as random errors.

4. Sampling methods, including simple random sampling, stratified random

sampling and cluster sampling

a. Nonprobability sampling
Nonprobability sampling is any sampling method where some elements of the population have
no chance of selection or where the probability of selection can't be accurately determined. The
selection of elements is based on some criteria other than randomness. These conditions give rise
to exclusion bias, caused by the fact that some elements of the population are excluded.
Nonprobability sampling does not allow the estimation of sampling errors. Information about the

2
relationship between sample and population is limited, making it difficult to extrapolate from the
sample to the population.
Example: We visit every household in a given street, and interview the first person to answer the
door. In any household with more than one occupant, this is a nonprobability sample, because
some people are more likely to answer the door (e.g. an unemployed person who spends most of
their time at home is more likely to answer than an employed housemate who might be at work
when the interviewer calls) and it's not practical to calculate these probabilities.
One example of nonprobability sampling is convenience sampling (customers in a supermarket
are asked questions). Another is quota sampling, when judgment is used to select the subjects
based on specified proportions. For example, an interviewer may be told to sample 200 females
and 300 males between the age of 45 and 60.
In addition, nonresponse effects may turn any probability design into a nonprobability design if
the characteristics of nonresponse are not well understood, since nonresponse effectively
modifies each element's probability of being sampled.

b. Probability sampling
Probability sampling is one in which every unit in the population can be randomly selected, and
the probability of selecting can be accurately determined.

c. Simple random sampling

In a simple random sample (“SRS”), all samples of a given size have an equal probability of
being selected and selections are independent. The frame is not subdivided or partitioned. The
sample variance is a good indicator of the population variance, which makes it relatively easy to
estimate the accuracy of results.
However, SRS can be vulnerable to sampling error because the randomness of the selection may
result in a sample that doesn't reflect the makeup of the population. For instance, a simple
random sample of ten people from a given country will on average produce five men and five
women, but any given trial is likely to overrepresent one sex and underrepresent the other.
Systematic and stratified techniques, discussed below, attempt to overcome this problem by
using information about the population to choose a more representative sample.
In some cases, investigators are interested in research questions specific to subgroups of the
population. For example, researchers might be interested in examining whether cognitive ability
as a predictor of job performance is equally applicable across racial groups. SRS cannot
accommodate the needs of researchers in this situation because it does not provide subsamples of
the population. Stratified sampling, which is discussed below, addresses this weakness of SRS.

3
 d. Systematic sampling
Systematic sampling relies on dividing the target population into strata (subpopulations) of equal
size and then selecting randomly one element from the first stratum and corresponding elements
from all other strata. A simple example would be to select every 10th name from the telephone
directory, with the first selection being random. SRS may select a sample from the beginning of
the list. Systematic sampling helps to spread the sample over the list.
As long as the starting point is randomized, systematic sampling is a type of probability
sampling. 'Every 10th' sampling is especially useful for efficient sampling from databases.
However, systematic sampling is especially vulnerable to periodicities in the list. Consider a
street where the odd-numbered houses are all on one side of the road, and the even-numbered
houses are all on another side. Under systematic sampling, the houses sampled will all be either
odd-numbered or even-numbered. Another drawback of systematic sampling is that even in
scenarios where it is more accurate than SRS, its theoretical properties make it difficult to
quantify that accuracy.
Systematic sampling is not SRS because different samples of the same size have different
selection probabilities - e.g. the set {4,14,24,...} has a one-in-ten probability of selection, but the
set {4,13,24,34,...} has zero probability of selection.

e. Stratified sampling
Where the population embraces a number of distinct categories, the frame can be organized by
these categories into separate "strata." Each stratum is then sampled as an independent sub-
population. Dividing the population into strata can enable researchers to draw inferences about
specific subgroups that may be lost in a more generalized random sample. Since each stratum is
treated as an independent population, different sampling approaches can be applied to different
strata. However, implementing such an approach can increase the cost and complexity of sample
selection.
A stratified sampling approach is most effective when three conditions are met:
i. Variability within strata are minimized
ii. Variability between strata are maximized
iii. The variables upon which the population is stratified are strongly correlated with the
desired dependent variable (beer consumption is strongly correlated with gender).

f. Cluster sampling
Sometimes it is cheaper to 'cluster' the sample in some way, e.g. by selecting respondents from
certain areas only, or certain time-periods only. Cluster sampling is an example of 'two-stage
random sampling': in the first stage a random sample of areas is chosen; in the second stage a
random sample of respondents within those areas is selected. Cluster sampling works best when
each cluster is a small copy of the population.
This can reduce travel and other administrative costs. Cluster sampling generally increases the
variability of sample estimates above that of simple random sampling, depending on how the
clusters differ between themselves, as compared with the within-cluster variation. If clusters
chosen are biased in a certain way, inferences drawn about population parameters will be
inaccurate.

4
 g. Matched random sampling
In this method there are two samples in which the members are clearly paired, or are matched
explicitly by the researcher (for example, IQ measurements or pairs of identical twins).
Alternatively, the same attribute, or variable, may be measured twice on each subject, under
different circumstances (example: the milk yields of cows before and after being fed a particular
diet).

C. Planning and conducting experiments

1. Characteristics of a well-designed and well-conducted experiment
A good a statistical experiment includes:
i. Stating the purpose of the research, including estimates regarding the size of treatment
effects, alternative hypotheses, and the estimated experimental variability. Experiments
must compare the new treatment with at least one standard treatment, to allow an
unbiased estimate of the difference in treatment effects.
ii. Design of experiments, using blocking (to reduce the influence of confounding variables)
and randomized assignment of treatments to subjects.
iii. Examining the data set in secondary analyses, to suggest new hypotheses for future study.
iv. Documenting and presenting the results of the study.
Example. Experiments on humans can change their behavior. The famous Hawthorne study
examined changes to the working environment at the Hawthorne plant of the Western Electric
Company. The researchers first measured the productivity in the plant, then modified the
illumination in an area of the plant and found that productivity improved. However, the study is
criticized today for the lack of a control group and blindness. Those in the Hawthorne study
became more productive not because the lighting was changed but because they were being
observed.

2. Treatments, control groups, experimental units, random assignments and

replication

a. Control groups and experimental units

To be able to compare effects and make inference about associations or predictions, one typically
has to subject different groups to different conditions. Usually, an experimental unit is subjected
to treatment and a control group is not.

b. Random assignments
The second fundamental design principle is randomization of allocation of (controlled variables)
treatments to units. The treatment effects, if present, will be similar within each group.

c. Replication
All measurements, observations or data collected are subject to variation, as there are no
completely deterministic processes. To reduce variability, in the experiment the measurements
must be repeated. The experiment itself should allow for replication, to be checked by other
researchers.

5
 3. Sources of bias and confounding, including placebo effect and blinding
Sources of bias specific to medicine are confounding variables and placebo effects, among
others.

a. Confounding
A confounding variable is an extraneous variable in a statistical model that correlates (positively
or negatively) with both the dependent variable and the independent variable. The methodologies
of scientific studies therefore need to control for these factors to avoid a false positive (Type I)
error (an erroneous conclusion that the dependent variables are in a causal relationship with the
independent variable).
Example. Consider the statistical relationship between ice cream sales and drowning deaths.
These two variables have a positive correlation because both occur more often during summer.
However, it would be wrong to conclude that there is a cause-and-effect relation between them.

b. Placebo and blinding

A placebo is an imitation pill identical to the actual treatment pill, but without the treatment
ingredients. A placebo effect is a sham (or simulated) effect when medical intervention has no
direct health impact but results in actual improvement of a medical condition because the
patients knew they were treated. Typically all patients are informed that some will be treated
using the drug and some will receive the insert pill, however the patients are blinded as to
whether they actually received the drug or the placebo.

c. Blocking
Blocking is the arranging of experimental units in groups (blocks) that are similar to one another.
Typically, a blocking factor is a source of variability that is not of primary interest to the
experimenter. An example of a blocking factor might be the sex of a patient; by blocking on sex
(that is comparing men to men and women to women), this source of variability is controlled for,
thus leading to greater precision.

4. Completely randomized design, randomized block design and matched pairs

a. Completely randomized designs

Completely randomized designs are for studying the effects of one primary factor without the
need to take other nuisance variables into account. The experiment compares the values of a
response variable (like health improvement) based on the different levels of that primary factor
(e.g., different amounts of medication). For completely randomized designs, the levels of the
primary factor are randomly assigned to the experimental units (using, for example, a random
numbers generator).

b. Randomized block design

A randomized block experiment is a collection of completely randomized experiments, each run
within one of the blocks of the total experiment. A matched pairs design is its special case when
the blocks consist of just two elements (measurements on the same patient before and after the
treatment or measurements on two different but in some way similar patients).