Studies and experiments
Problem-Solving and Data Analysis is about 15% of SAT Math (5–7 of 44 questions), and evaluating statistical claims is a small part of it. You read a short description of a study and choose the conclusion that is supported, or the flaw that makes a conclusion unsafe.
Two kinds of study
In an observational study, researchers only record what already happens: they do not change anything. In an experiment, researchers give different treatments to different groups and compare the results. A treatment can be a medicine, a teaching method, a fertilizer or any other condition the researchers decide. Only an experiment can show that the treatment caused a difference, and only if it is done correctly.
| Observational study | Experiment | |
|---|---|---|
| Researchers… | record what people already do | give each group a treatment |
| Compare groups… | as they are | made by chance (random assignment) |
| Can show cause? | No: only an association | Yes, if assignment was random |
Two different random steps
The SAT keeps two ideas apart. Random sampling means the participants were picked by chance from the population. It lets you generalize: you may apply the result to the whole population. Random assignment means chance decided who got the treatment and who got the control. It lets you claim cause and effect, because chance makes the groups alike, on average, at the start, so the treatment is the only systematic difference between them. A study can have one, both, or neither.
| Random sampling? | Random assignment? | What you may conclude |
|---|---|---|
| Yes | Yes | Cause and effect, for the whole population |
| No | Yes | Cause and effect, but only for the participants |
| Yes | No | An association, for the whole population |
| No | No | An association, for the participants only |
Correlation is not causation
If two things go up together, they are correlated. That does not mean one causes the other. A third factor, called a confounding factor, may drive both. Cities with more ice cream sales also have more sunburns, but ice cream does not burn skin: hot, sunny weather explains both. In an observational study people choose their own group, so the groups usually differ in other ways too. Words such as “causes”, “leads to” and “because of” are unsafe after an observational study.
Bias: when a sample misleads
A sample is biased when it leaves out or over-represents some members of the population. Volunteers differ from everyone else: people with strong opinions answer surveys more often. Convenience samples (the first people you meet, your own friends) are biased too. The result applies only to the group that was really sampled. If the sample came from one school, you may not generalize to all schools; if it was all adults in one city, not to adults in other cities.
Choosing the supported conclusion
Ask the same three questions every time, then match the answer choice to your findings. The correct choice is usually the one that is careful: it names the right group and does not claim more than the design allows.
- The average sleep of all 2,400 students at the school is probably close to 7.8 hours.
- Sleeping 7.8 hours causes students at the school to do well in class.
- The average sleep of students in all schools in Uzbekistan is 7.8 hours.
- The 300 students would sleep longer if they went to bed earlier.
- Sampling: random, from the 2,400 students. So we may generalize to those 2,400 students.
- Assignment: there was none. It is a survey, so no cause and effect.
- B claims a cause, C names a bigger population than the one sampled, and D claims an effect nobody tested.
- Only A stays inside the school and says “probably close to”.
- The new routine causes higher flexibility in all adults.
- The new routine caused higher flexibility in these 80 volunteers.
- People with higher flexibility are more likely to choose the new routine.
- The new routine has no effect on flexibility.
- Assignment: random, so a cause-and-effect conclusion is allowed.
- Sampling: the 80 were volunteers from one club, not a random sample, so the result applies to them only.
- A generalizes to all adults, which is not allowed. C is wrong because the participants did not choose their group.
- D contradicts the data. Only B has both the cause and the correct group.
- Eating breakfast causes adults in the region to weigh less.
- Skipping breakfast makes people gain weight.
- Among adults in the region, eating breakfast every day is associated with lower average body weight.
- Adults all over the world who eat breakfast weigh less.
- Sampling: random from the adults of the region, so we may generalize to them.
- Assignment: none. People chose whether to eat breakfast, so only an association can be claimed.
- A and B claim a cause. D names the whole world.
- C is an association about the right population.
- Confusing the two random stepsRandom sampling → generalize to the population. Random assignment → cause and effect. Ask which one the study really used.
- Claiming cause after an observational studyIf the researchers did not assign the treatment, avoid “causes”, “leads to” and “makes”. Choose “is associated with”.
- Generalizing past the population sampledFind the exact group that was sampled (one school, one city, volunteers) and keep the conclusion inside it.
- Thinking a large sample fixes everythingA big sample of volunteers is still biased. What matters is how the participants were chosen, not only how many.
- Choosing the most exciting answerThe right choice is often the careful one. Strong words (“proves”, “all”, “always”) should make you suspicious.
Desmos is not needed for the reasoning in this lesson, because the question is about the design of the study. It helps only with the arithmetic that sometimes comes with it, such as comparing two group averages or scaling a sample result up to a population.
- Type a proportion such as 18/120*3000 to scale a sample result up to the population
- Type the difference between two group means as one expression
- Do not spend time on Desmos if the question has no numbers
Use it for arithmetic only. The conclusion itself comes from reading the design of the study carefully.
Open Desmos ↗