Week 4 · Regression: predicting y from x, simple linear regression y = a + bx and the least squares method · 3 lessons
2
The best straight line: the errors between the points and the line, and the idea of least squares
Statistics Sec 3 Term 1, Unit 1, Lesson 2
GoalMeasure the errors between the dots and a line, and explain why the best line makes the sum of the squared errors smallest.
New words
error (gap) · the vertical distance y − ŷ between a dot and the lineŷ (y-hat) · the value of y given by the line at that xleast squares · choose a and b so that Σ(y − ŷ)² is as small as possible
Explanation
Through a cloud of dots we can draw many straight lines, and we want the best one. For each dot, the error is the vertical gap between the dot and the line: y − ŷ, where y is the observed value and ŷ is the value on the line at the same x. A dot above the line has a positive gap, a dot below it a negative gap. We cannot simply add the gaps, because positive and negative gaps cancel and a very bad line can have the sum 0. So we square them, which makes every gap positive and punishes big gaps more, and add: Σ(y − ŷ)². The least squares method chooses the line whose total is smallest. (Finding it exactly uses calculus; in the next lesson we use its result.) The picture shows five pupils: weeks of training x and the distance of a throw y in metres. The short vertical segments are the gaps from the line ŷ = 2.9 + 1.3x; their squares add up to 6.4, and no other straight line does better.
Worked examples
x
y
ŷ
y − ŷ
(y − ŷ)²
3
8
6.8
1.2
1.44
5
8
9.4
−1.4
1.96
7
11
12.0
−1.0
1.00
9
16
14.6
1.4
1.96
11
17
17.2
−0.2
0.04
Σ = 0
Σ = 6.4
The gaps of the line ŷ = 2.9 + 1.3x. At x = 3 the line gives 6.8, the pupil threw 8, so the gap is 1.2; the other gaps are −1.4, −1, 1.4 and −0.2. The squares are 1.44, 1.96, 1, 1.96 and 0.04, with total 6.4. The gaps themselves add to 0.
Two other lines for the same dots. Line ŷ = 4 + x has gaps 1, −1, 0, 3, 2 and squares totalling 1 + 1 + 0 + 9 + 4 = 15. The line through the first and last dots, ŷ = 8 + 1.125(x − 3), has gaps 0, −2.25, −1.5, 1.25, 0 and squares totalling 8.875. Both totals are larger than 6.4, so the line ŷ = 2.9 + 1.3x is the better one.
Why square? The line ŷ = 4 + x has gaps that add to 5, but a different bad line could have gaps +3 and −3 and a total of 0. Squares give 9 + 9 = 18 and expose it. The best line need not pass through any dot; here it passes through none of the five dots.
Class activity
Draw the five dots on squared paper (3, 8), (5, 8), (7, 11), (9, 16), (11, 17). Lay a ruler where you think the best line is, and read the five gaps. Then do the same with the ruler at a different slope. Which total of squares is smaller?
Practice
1
For the line ŷ = 1 + 2x, find the gaps y − ŷ of the dots (3, 8) and (4, 7).
At x = 3: ŷ = 7, gap 8 − 7 = 1. At x = 4: ŷ = 9, gap 7 − 9 = −2.
2
Two lines give total squared gaps of 12 and 9 for the same dots. Which line is better by least squares?
The line with total 9, because it is smaller.
3
For the line ŷ = 2.9 + 1.3x find ŷ and the gap at the dot (9, 16).
ŷ = 2.9 + 11.7 = 14.6, so the gap is 16 − 14.6 = 1.4 (the dot is above the line).
4
Why can we not choose the best line as the one with gaps adding to 0?
Many lines, even bad ones, have gaps that cancel to 0. Only the sum of squares measures how close the line is overall.
5
Must the best line pass through one of the dots?
No. It only has to make Σ(y − ŷ)² smallest; it may miss every dot.
