Regression: the line

# Regression: the line

**Topic 12 · Thursday, March 4**

]

---

# Where are we?

.box-inv-1.small[Summarize data]

.box-inv-1.small[Visualize data]
]

.box-inv-6.small[Does one score go with another?]

.box-inv-6.small[Use one score to predict another]

???

Same map. Topic 11 asked whether two scores go together. Today we use one to predict the other. Regression takes three sessions: the line today, whether the slope is real next Tuesday, and groups next Thursday.

---

# Plan for today

.box-1.medium.sp-after-half[A line for prediction]

.box-6.medium.sp-after-half[What the line minimizes]

.box-3.medium.sp-after-half[Finding a and b]

???

Three blocks of about 20 minutes. Each ends with students doing something: try, predict, try. The running question is the one from Topic 11: does a student's baseline math score predict their endline score?

---

# A line for prediction

---

---

# A line sums up the cloud

.box-inv-1.small[The line shows the pattern, describes it in two numbers, and lets us predict]

.source[Data: Muralidharan, Singh, and Ganimian (2019). 539 students. Dots are nudged so equal scores don't hide each other.]

???

Same dots as Topic 11. Topic 11's simulator already drew a line through a cloud, and now we ask what that line is.

Regression finds the straight line that best fits a set of dots. By the end of block 2, "best" will mean something exact.

---

# A line needs two numbers

.box-1.small[Greek means population: β is the slope for everyone, and b is our estimate of it]

???

The rule from Topic 3 still holds. A Latin letter or a hat means an estimate, so never call b "beta".

Topic 7 used β for the chance of a Type 2 error. That's a different β, and the context tells them apart. The population line gets plain numbers, such as Y = 2.00 + 0.50 X in the simulator, because α is taken too.

---

# Our line, read in points

.pull-left[
<img src="12-regression-line_files/figure-html/anatomy-plot-1.png" alt="The same line, with the dots faded. A red dot marks where the line meets a baseline of 0, at 8.87. A gold step shows that moving 10 points to the right, from a baseline of 10 to 20, raises the line by 7.6 points." width="100%" style="display: block; margin: auto;" />
]

.box-inv-1.small.sp-after-half[**a = 8.87:** a student with 0 at baseline is predicted to score 8.87 at endline]

.box-inv-1.small.sp-after-half[**b = 0.76:** one more point at baseline goes with 0.76 more points at endline, on average]

.box-1.small[Say "goes with", not "causes"]
]

???

Both numbers are in points. b compares students: among students who are one point apart at baseline, the one ahead tends to be 0.76 points ahead at endline. It doesn't say what would happen to one student who gained a point. We're still in the prediction column.

The numbers on this slide match 2024: 8.87 and 0.76.

---

# It predicts an average, not a student

.box-1.small[8.87 + 0.76 × 10 = **16.47**. Yet these 59 students ended anywhere from 7 to 32]

???

The arithmetic uses the rounded line, the way students will do it. Software keeps more decimals and gets 16.51.

The prediction is our best guess, and it's close to what these students did on average. Any one student can land far from it, and block 2 is about exactly that gap.

---

# Try it: Use the line

.box-8.medium.sp-after-half[Ŷ = 8.87 + 0.76X]

.box-inv-8.small.sp-after-half[Two students scored 6 and 16 at baseline. Predict each one's endline score]

.box-inv-8.small.sp-after-half[How far apart are the students at baseline? How far apart are the predictions?]

.box-inv-8.small[Work alone, compare with a neighbor, then vote with hands: is the gap bigger, smaller or the same?]

???

About 6 minutes, with calculators.

---

# Try it: Answer

.box-8.small.sp-after-half[10 points apart at baseline, 7.60 apart in prediction. The gap is 10 × b]

.box-inv-8.small[The slope is all you need to compare two students. a only sets the height]

???

Remember the smaller gap. It comes back at the end of today as regression to the mean.

---

# What the line minimizes

---

---

# A residual is how far the line missed

.pull-left[
<img src="12-regression-line_files/figure-html/residual-plot-1.png" alt="The scatter plot with faded dots and the line. One student at a baseline of 13 and an endline of 30 sits above the line, joined to it by a vertical red segment. Another at a baseline of 15 and an endline of 12 sits below the line, joined to it the same way." width="100%" style="display: block; margin: auto;" />
]

.box-inv-6.small[Above the line, it's positive. Below the line, it's negative]
]

???

These are two real students. The first did 11.25 points better than their baseline predicts. The second did 8.27 points worse.

The residual is what the line can't explain. It might be a good day, a strong teacher or a lucky guess, and the data don't say which.

---

# Residuals add to 0, so we square them

.box-inv-6.small.sp-after-half[All 539 residuals add up to 0, like deviations from a mean]

.box-6.small[So we square them, as in Topic 3: Σ(Y − Ŷ)² = 14,314, the SS<sub>residual</sub>]

???

Deviations from the mean add up to zero, and that's why variance squares them. The same thing happens here, for the same reason. In fact, the mean is the one number that minimizes squared deviations, and the regression line does the same job for a whole line of numbers.

---

# Loose dots leave big squares

.source[Simulator after Kristoffer Magnusson, rpsychologist.com. On your phone: [the simulator page](https://educ15.de-barros.com/resource/simulator-correlation.html).]

???

**Point to the readout:** r is 0.30, and the sum of squared residuals is 2,252. Each pink square is one student's squared miss.

**Drag a point with a big square toward the line.** Its square shrinks, and the total falls. Then drag it away and watch the square grow, and the line chase it. A vertical line is a residual, and the springs picture from 2024 fits here: each dot pulls on the line, and far dots pull hardest.

The tool always shows the best line for the dots, so students can't move the line itself. The next slide and the plot after it do that job.

Students can't drag on this screen. Those with a phone or laptop can open the simulator's Correlation and regression page on the course site, where Residuals and Squared residuals are checkboxes.

---

# Tight dots leave small squares

???

Same 12 baseline scores, with r up from 0.30 to 0.90. Flip back and forth between the two slides. The total falls from 2,252 to 470.

So the squares are a way to see fit. Small squares mean the line fits well, and r is high. That's where r² comes from, later today.

---

# Least squares picks the smallest total

.box-6.small[Regression chooses the a and b that make Σ(Y − Ŷ)² as small as it can be]

???

Back to our 8 students, where each square is big enough to see. Two of them share a point, so one square is darker. All three lines go through the two means.

The name is ordinary least squares, or OLS. It's the same regression. No other straight line through these 8 students gets below 58.5.

---

# Think of the line as held by springs

.box-6.small[Every student's spring pulls on the line. Least squares is where they balance]

???

Vertical springs, one per student. A spring stores energy with the square of its stretch, so the resting place of the line is the one with the smallest total of squared residuals.

---

# The best slope sits at the bottom

.box-inv-6.small[Every slope has a total. b = 2.25 is the only one at the bottom of the curve]

.source[Data: Muralidharan, Singh, and Ganimian (2019). Students 25 to 32. Each line goes through the two means.]

???

This is what "the squares shrink as the line fits" looks like. Walk along the curve from the left: the line gets steeper, the total falls, it bottoms out at 58.5, and then it rises again. Tilt the line past the best slope, and the squares grow back.

A computer doesn't try slopes one at a time. There's a formula for the bottom of the curve, which we use in block 3.

---

# We measure misses up and down

.box-inv-6.small[We're predicting Y, so a miss is a distance in Y]

???

The right-hand picture looks natural, and it's a common wrong guess.

This is why it matters which score is X and which is Y. Swap them, and the misses run sideways instead, so you get a different line. With our students, predicting baseline from endline has a slope of 0.33, not 1 / 0.76. r is the same in both directions, but regression isn't.

---

# Predict it: Which line wins?

.box-inv-8.small[
Line A misses four students by 0, 0, 0 and 8 points. Line B misses them by 2, 2, 2 and 2. Each line misses by 8 points in all. Which one does least squares prefer?
]

.options[
.option[
**A** Line A
]
.option[
**B** Line B
]
.option[
**C** It's a tie
]
.option[
**D** We can't tell without the slope
]
]

???

About 6 minutes. Vote, turn to a neighbor, vote again, then show the answer.

Andy: create this poll in Poll Everywhere, then add the `.poll-bar` button from EDUC 265's 06-01 with its presenter id.

---

# Predict it: Answer

.options[
.option.wrong[
**A** Line A
]
.option.correct[
**B** Line B
]
.option.wrong[
**C** It's a tie
]
.option.wrong[
**D** We can't tell without the slope
]
]

.box-inv-8.small.sp-after-half[
Line A: 0² + 0² + 0² + 8² = 64. Line B: 2² + 2² + 2² + 2² = 16
]

.box-8.small[
Squaring punishes one big miss. That's why one outlier can drag the line, as it dragged r in Topic 11
]

???

Least squares would rather miss everyone a little than one person a lot. This is the price of squaring, and a reason to plot the data first.

---

# Finding a and b

---

---

# We already have what we need

.box-inv-3.small[From Topic 11, our 8 students: SP = 54, SS<sub>X</sub> = 24, M<sub>X</sub> = 10.5, M<sub>Y</sub> = 22.5]

<div class="math-display">\[
a = M_Y - b M_X = 22.5 - 2.25 \times 10.5 = −1.125
\]</div>

.box-inv-3.small[SP: how X and Y move together. SS<sub>X</sub>: how X moves alone]

.box-3.small[Their line: Ŷ = −1.125 + 2.25X]

???

These formulas give the bottom of the curve we just saw. Students need to use them, not derive them.

The 2024 slides wrote "beta = 2.25" here. It's an estimate, so it's b.

In Lab 9, a spreadsheet's `SLOPE` and `INTERCEPT` do this for all 539 students.

---

# The line runs through the two means

.box-inv-3.small[A student at M<sub>X</sub> is predicted to land at M<sub>Y</sub>. That's where a's formula comes from]

???

This matches the line from 2024, so students now know where it came from. The same is true for all 539 students: 8.87 + 0.76 × 11.06 is about 17.32, the mean of Y.

---

# Don't trust the line past the data

.box-inv-3.small[A score below 0 is impossible. Predict only inside the scores you saw]

???

A prediction is only worth trusting inside the range of X we observed. These 8 students run from 8 to 13, so the line says nothing about a score of 0.

With all 539 students, a = 8.87 is safer, but it still leans on few students: only 2 scored 0 at baseline.

---

# b is r, rescaled into points

<div class="math-display">\[
b = r \times \frac{\hat{\sigma}_Y}{\hat{\sigma}_X}
\]</div>

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Students </th>
   <th style="text-align:center;"> r </th>
   <th style="text-align:center;"> σ̂<sub>Y</sub> </th>
   <th style="text-align:center;"> σ̂<sub>X</sub> </th>
   <th style="text-align:center;"> r × σ̂<sub>Y</sub> / σ̂<sub>X</sub> </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> All 539 </td>
   <td style="text-align:center;"> 0.502 </td>
   <td style="text-align:center;"> 5.97 </td>
   <td style="text-align:center;"> 3.92 </td>
   <td style="text-align:center;"> <strong>0.76</strong> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Our 8 </td>
   <td style="text-align:center;"> 0.822 </td>
   <td style="text-align:center;"> 5.07 </td>
   <td style="text-align:center;"> 1.85 </td>
   <td style="text-align:center;"> <strong>2.25</strong> </td>
  </tr>
</tbody>
</table>

.box-3.small[r has no units. b is in points of Y per point of X. They always share a sign]

.book-tag[In the book: σ̂<sub>Y</sub> and σ̂<sub>X</sub> are written s<sub>Y</sub> and s<sub>X</sub>]

???

So a slope isn't a new idea. It's the correlation, stretched by how spread out Y is compared with X. If r is 0, b is 0. That's why a test of r and a test of b will agree on Tuesday.

The two groups show why b alone can't tell you how strong a relationship is. Our 8 students have b = 2.25 partly because their baseline scores are so close together.

---

# Regression to the mean is in the slope

.pull-left[
<img src="12-regression-line_files/figure-html/rtm-plot-1.png" alt="Scatter plot of all students in standard deviation units, with faded dots. A dashed line has a slope of 1. The solid regression line has a slope of 0.50, flatter than the dashed line. At 2 SDs on baseline, the dashed line is at 2 and the regression line is at about 1." width="100%" style="display: block; margin: auto;" />
]

.pull-right[
.box-3.small.sp-after-half[In SDs, the slope is r, and r is below 1 unless the dots fall on a perfect line]

.box-inv-3.small.sp-after-half[So a student 2 SDs above the mean is predicted only 1.0 SD above it]

.box-inv-3.small[The 42 students with 17 or more at baseline were 1.96 SDs above the mean. At endline: 0.98]
]

???

This is the smaller gap from the first try-it. Part of any high score is a good day, and good days don't repeat. So high scorers tend to land closer to the mean the second time, and low scorers do too. No program is needed. That's where the word "regression" comes from: Galton saw it in the heights of parents and children.

It's why you can't judge a program by looking only at the lowest scorers and how far they rise.

The rule is about SDs, not points. Our b of 0.76 happens to be below 1, but the 8 students' b of 2.25 isn't, because their X barely spreads. In SDs, their slope is 0.82.

---

# r² is how much the squares shrink

<div class="math-display">\[
r^2 = \frac{SS_Y - SS_{residual}}{SS_Y} = \frac{180 - 58.5}{180} = 0.675
\]</div>

.box-inv-3.small[A flat line ignores X, and its squares total SS<sub>Y</sub>. For our 8, the best line cuts that by 67.5%]

???

This keeps Topic 11's promise. The flat line predicts M<sub>Y</sub> for everyone, so its squares are SS<sub>Y</sub>: that's the "Flat" total on the curve. The regression line's squares are SS<sub>residual</sub>. r² is the share of the squares that goes away when we use X.

With all 539 students, (19,143 − 14,314) / 19,143 = 0.252, the same 0.252 as Topic 11.

The bars redraw the book's split of SS<sub>Y</sub> into SS<sub>regression</sub> and SS<sub>residual</sub>, with our numbers. Tuesday uses this split.

---

# Try it: Five students from Topic 11

.pull-left[
<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:center;"> Baseline X </th>
   <th style="text-align:center;"> Endline Y </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:center;"> 13 </td>
   <td style="text-align:center;"> 14 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 10 </td>
   <td style="text-align:center;"> 21 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 11 </td>
   <td style="text-align:center;"> 24 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 11 </td>
   <td style="text-align:center;"> 14 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 10 </td>
   <td style="text-align:center;"> 22 </td>
  </tr>
</tbody>
</table>
]

.box-inv-8.small[SP = −15, SS<sub>X</sub> = 6]

.box-8.small[Find b and a. Then find the residual for the student with X = 11 and Y = 24]
]

.box-inv-8.small[Work alone, compare with a neighbor, then vote with hands: is the residual positive or negative?]

???

About 6 minutes, with calculators. These are the five from Topic 11, whose r was −0.65. The numbers on the right are given to save time.

---

# Try it: Answer

.box-inv-8.small[a = 19 − (−2.5 × 11) = **46.5**]
]

.box-inv-8.small[Y − Ŷ = 24 − 19 = **+5**]
]

.box-8.small.sp-after-half[r was negative here, so b is too. And X = 11 is M<sub>X</sub>, so Ŷ is M<sub>Y</sub>: no multiplying needed]

???

Common slips: the sign of b, and writing Ŷ − Y for the residual.

---

# Each b estimates β

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Students </th>
   <th style="text-align:center;"> n </th>
   <th style="text-align:center;"> r </th>
   <th style="text-align:center;"> a </th>
   <th style="text-align:center;"> b </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Five from the try-it </td>
   <td style="text-align:center;"> 5 </td>
   <td style="text-align:center;"> −0.65 </td>
   <td style="text-align:center;"> 46.50 </td>
   <td style="text-align:center;"> −2.50 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Students 25 to 32 </td>
   <td style="text-align:center;"> 8 </td>
   <td style="text-align:center;"> 0.82 </td>
   <td style="text-align:center;"> −1.125 </td>
   <td style="text-align:center;"> 2.25 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Everyone with both scores </td>
   <td style="text-align:center;"> 539 </td>
   <td style="text-align:center;"> 0.50 </td>
   <td style="text-align:center;"> 8.87 </td>
   <td style="text-align:center;"> 0.76 </td>
  </tr>
</tbody>
</table>

.box-inv-8.small.sp-after-half[Same test, three groups, three slopes. Like M and r, b bounces from sample to sample]

.box-8.small[Tuesday's question: could we see a slope like this if β were 0?]

???

This is the bridge to Topic 13, and it's the same step we took with r in Topic 11. A slope is a statistic, so it has a sampling distribution. That's step 7a of the blueprint, and the four steps come back with it. Don't test anything today.

---

# What you can do now

.box-inv-8.small[Read a and b from a line, in the units of the scores]

.box-inv-8.small[Predict Y from X, and say why that's an average, not a promise]

.box-inv-8.small[Find a residual, and say what least squares minimizes]

.box-inv-8.small[Compute b and a by hand from SP, SS<sub>X</sub> and the means]

.box-inv-8.small[Link b to r, r² to the squares, and r to regression to the mean]

---

# Next steps

Correlation and regression. Bring a calculator and a laptop
]

Mastery and Problem Set for Chapter 14 are due
]

.box-3.small[For Tuesday, March 9: read the rest of section 14-6, for Topic 13]

.picture-small img {
  display: block;
  margin: 0 auto;
  max-width: 100%;
  max-height: 230px;
}

.anteater img {
  display: block;
  margin: 0.15em auto 0;
  max-height: 105px;
}

/* This chunk sits at the end because any output before the first slide's
   "class:" line stops remark from reading that slide's properties */

/* Long titles get two smaller lines inside the 110 px title bar */
.remark-slide-content.long-title h1 {
  font-size: 1.05em;
  line-height: 1.05;
  margin-top: -100px;
  margin-bottom: 55px;
}

.source {
  position: absolute;
  bottom: 18px;
  left: 60px;
  font-size: 0.55em;
  color: #47505a;
}

.source a {
  color: #244a70;
  text-decoration: underline;
}

.book-tag {
  margin-top: 0.7em;
  font-size: 0.65em;
  color: #47505a;
  text-align: right;
}

table.deck-table {
  width: 92%;
  margin: 0.4em auto;
  border-collapse: collapse;
  font-size: 0.72em;
}

table.deck-table.tight td,
table.deck-table.tight th {
  padding: 0.12em 0.6em;
}

table.deck-table th {
  border-bottom: 2px solid #244a70;
  padding: 0.3em 0.6em;
}

table.deck-table td {
  border-bottom: 1px solid #c9d1d9;
  padding: 0.3em 0.6em;
}

.options {
  display: grid;
  grid-template-columns: 1fr 1fr;
  gap: 0.35em 1em;
  margin-top: 0.6em;
  margin-bottom: 0.8em;
  font-size: 0.65em;
}

.option {
  background-color: #f3f3f3;
  border: 2px solid #cccccc;
  padding: 0.35em 0.6em;
}

.option.correct {
  background-color: #b18323;
  border-color: #b18323;
  color: white;
}

.option.wrong {
  opacity: 0.6;
}

.poll-respond {
  display: block;
  font-size: 0.5em;
  text-align: center;
  margin-top: 0.1em;
}

/* The printed PDF keeps the question and drops the answer slide */
@media print {
  .remark-slide-container:has(.poll-answer) {
    display: none !important;
  }
}
</style>