Chapter 6

Experiments in the Real World

Chapter 6 1

Example: The case of fickle mice

• In the real world experiments don’t always go smoothly. Even if they do we can’t always take a firm stand on the findings.

• Is our behavior coded into our genes? To find out, knock out a gene in one group of mice and compare their

behavior with a control group of normal mice.

Mice have the same genetic makeup before the experiment, and each mouse is randomly assigned to one group.

Any difference during the experiment is then attributed to the knocked-out gene.

• “No sooner has one group of researchers tied a gene to a behavior when along comes the next study, proving that the link is spurious or even that the gene in question has exactly the opposite effect.” (published in the journal Science)

Chapter 6 2

Example: The case of fickle mice

• To find out what goes wrong, scientists conducted the same experiments with the same genetic strain in three different labs (Oregon, Alberta, New York)

• The results were often different.

• It appears that very small differences in the lab environments have big effects on the behavior of the mice

Chapter 6 3

Equal treatment for all

• A sampler should know exactly what information she wants and must compose questions that extract that information from that sample.

• An experimenter must know exactly what treatments and responses he wants information about, and he must construct apparatus needed to apply the treatments and measure the responses.

• This is what is referred to as “designing and experiment”.

• The logic of randomized comparative experiment assumes that all the subjects are treated alike except for the treatments that the experiment is designed to compare.

• Any other unequal treatment can cause bias.

Chapter 6 4

Example: Feeding rats

• Does a new cereal provide good nutrition?

• Compare the weight gains of young rats fed the new product and rats fed a standard diet.

• Rats are placed in large racks of cages. Rats in upper cages grow a bit faster than the ones in bottom cages.

• If the experimenters put rats fed the new product at the top, the experiment is biased in favor of the new product.

• Solution: assign rats to cages at random.

Chapter 6 5

Completely randomized design

In a completely randomized experimental design, all the experimental subjects are allocated at random among all the treatments.

Chapter 6 6

Placebo, and Experimenter Effects

• The problem:

– people may respond differently when they know they are part of an experiment.

• The solution:

– use placebos, control groups, and double-blind studies when possible.

Chapter 6 7

Double-blind experiments

• The powerful placebo

• Want to keep balding men keep their hair? Give them placebo! One study found out that 42% of balding men maintained or increased the amount of hair on their heads when they took a placebo.

• Another study told 13 people who were very sensitive to poison ivy that the stuff being rubbed on one arm was poison ivy. It was a placebo, but all 13 broke out in rash.

• The stuff rubbed on the other arm really was poison ivy, but the subjects were told it was harmless, and only 2 developed rash.

Chapter 6 8

Double-blind experiments

• Knowing that they are getting “just a placebo” might weaken the placebo effect and bias the experiment in favor of the other treatments.

• Also, if the doctors and other medical personnel know if a certain subject is receiving “just a placebo” they might expect less.

• Doctor’s expectations change how they interact with patients and even the way they diagnose a patient’s condition.

Chapter 6 9

Double-blind experiments • In a double-blind experiment, neither the subjects nor the

people who work with them know which treatment each subject is receiving.

Chapter 6 10

Refusals, nonadherers, and dropouts • Sample surveys suffer from nonresponse due to failure to

contact some people selected for the sample and refusal or others to participate.

• Subjects who participate but don’t follow the experimental treatment, called nonadherers, can also cause bias.

• Experiments that continue over an extended period of time also suffer dropouts, subjects who begin the experiment but do not complete.

Chapter 6 11

Can we generalize?

• A well-designed experiment tells us that changes in the explanatory variable cause changes in the response variable.

• It tells us that this happened for specific subjects in the specific environment of this specific experiment.

• Can we generalize our conclusions from our little group of subjects to a wider population? First step is to make sure that our findings are statistically significant,

that they are too strong to occur just by chance.

The treatments, the subjects, and/or the environment may not be realistic.

Chapter 6 12

Can We Generalize?

• The problem: lack of generalizability due to:

unrealistic treatments

unnatural settings

sample that is not representative of population

• The solution: Researchers should use natural settings with a properly

chosen sample.

• Good experiments combine statistical principles with understanding of a specific field of study.

Chapter 6 13

Example: Center brake lights • Cars sold in the U.S. since 1986 are required to have a center brake

light. This requirement was justified by randomized comparative experiments with fleets of rental and business cars.

• The experiments showed that the third brake light reduced rear-end collisions by 50%.

• After a decade in actual use, the Insurance Institute found only a 5% reduction in rear-end collisions.

• What happened? At the time the first experiment was carried out most cars did not have break lights, so it caught the attention of the drivers.

• Now that almost all the cars have the light, it no longer captures the attention.

• The experiments conclusions did not generalize as good as expected, because the environment changed.

Chapter 6 14

Experimental design in the real world

• A completely randomized design can have any numbers of explanatory variables, and they might interact in their effect on the response variable.

Chapter 6 15

Interacting Variables

• The problem: effect of explanatory variable on response variable

may vary over levels of other variables.

• The solution: measure and study potential interacting variables. does the relationship between explanatory and response

variables change for different levels of these interacting variables? if so, report results for different groups defined by the

levels of the interacting variables.

Chapter 6 16

Example: Effects of TV advertising • What are the effects of repeated exposure to an advertising

message? The answer may depend both on the length of the ad and on how often it is repeated.

• In an experiment, all subjects viewed a 40-minute TV program that included ads for a digital camera. Some subjects saw a 30-second commercial; others, a 90-second version.

• The same commercial was repeated 1, 3, or 5 times during the program.

• After viewing, all subjects answered questions about their recall of the ad, their attitude toward the camera, and their intention to purchase it (the response variables).

• Explanatory variables: length of the commercial, with 2 levels,

Repetitions, with 3 levels.

Chapter 6 17

Chapter 6 18

Matched pairs and block designs

• One common design that combines matching with randomization is the matched pairs design, which compares just two treatments.

• Assign one of the treatments to each subject randomly.

• The order of treatments can influence the subject’s response, so we randomize the order for each subject.

Chapter 6 19

Example: Coke versus Pepsi • Pepsi wanted to demonstrate that Coke drinkers prefer Pepsi when

they taste both colas blind.

• The subjects, all of whom said they were Coke drinkers tasted both colas from glasses without brand markings.

• Since the response may depend on which cola is tasted first, the order of tasting should be chosen at random for each subject.

• When more than half the Coke drinkers chose Pepsi, Coke claimed that the experiment was biased.

• Pepsi glasses were marked M and Coke glasses were marked Q. Coke claimed that the results could just mean that people like the letter M better than the letter Q.

• Matched pairs design is OK, but any distinction other than Coke vs Pepsi should have been avoided.

Chapter 6 20

Chapter 6 21

•A block design combines the idea of creating equal groups

by matching with the principle of forming treatment groups

at random.

•They control the effect of some outside variables by

bringing them into the experiment to form the blocks.

Example: men, women, and advertising

• Women and men respond differently to advertising. An experiment to compare the effectiveness of three TV commercials for the same product will want to look at the reactions of men and women, as well as assesses the overall response to the ads.

• Randomly assign subjects to three treatment groups without regard to their sex.

• A better design considers women and men separately. Randomly assign women to three groups, one to view each commercial. Then separately assign men at random to three groups.

Chapter 6 22

Chapter 6 23

Key Concepts

• Double-Blind Experiment

• Difficulties and Disasters

• Experimental Designs

– Completely Randomized Design

– Matched Pairs Design

– Block Design

Chapter 6 24

Exercise 6.13

Chapter 6 25

• Comparing corn varieties. New varieties of corn with altered amino acid content may have higher nutritional value than standard corn, which is low in the amino acid lysine. An experiment compares two new varieties, called opaque-2 and floury-2, with normal corn. The researchers mix corn-soybean meal diets using each type of corn at each of three protein levels: 12% protein, 16% protein, and 20% protein. They feed each diet to 10 one-day old male chicks and record their weight gains after 21 days. The weight gain of the chicks is a measure of the nutritional value of their diet.

a) What are the individuals and the response variable in this experiment?

b) How many explanatory variables are there? How many treatments? Use a diagram to describe the treatments. How many experimental individuals does the experiment require?

c) Use a diagram to describe a completely randomized design for this experiment. (Don’t actually do the randomization.)