t-tests for two samples and for gains

# t-tests for two samples and for gains

**Topic 9 · Thursday, February 18 and Tuesday, February 23**

]

---

# Where are we?

.box-inv-1.small[Summarize data]

.box-inv-1.small[Visualize data]
]

.box-inv-3.small[Did the program cause the gap?]

???

Same map. So far, one mean at a time. Now two groups, and that's where explanation starts: a lottery lets us read a gap between groups as the program's effect. Two sessions, six blocks.

---

# The four steps stay, for a gap

.box-inv-8.small.sp-after-half[Step 1: State a hypothesis about the difference between two populations]

.box-inv-8.small.sp-after-half[Step 2: Set the criteria: what's unexpected among differences?]

.box-8.small.sp-after-half[Step 3: Observe a sample from each population, and compute their difference]

.box-inv-8.small[Step 4: Reject or maintain H₀]

???

The same four steps from Topics 7 and 8. Every step now talks about a difference, not one mean.

---

# Plan for today

.box-1.medium.sp-after-half[From one mean to two]

.box-6.medium.sp-after-half[The two-sample t]

.box-3.medium.sp-after-half[Mindspark's test]

.box-inv-8.small[On Tuesday: what if the same people take the test twice?]

???

Three blocks today, three on Tuesday.

---

# From one mean to two

---

---

# Lottery winners scored higher

.box-inv-1.small[The gap is 18.68 − 15.99 = **2.69 points**. Could that be sampling error?]

.source[Data: Muralidharan, Singh, and Ganimian (2019). 266 Mindspark students and 273 control students.]

???

The lottery from Topic 1. In Topic 6, we compared this gap with each group's own standard error, and promised a real test, with a standard error for the gap itself. That's today.

---

# The blueprint gets a new step

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Blueprint step </th>
   <th style="text-align:center;"> Panel 2: one sample gives </th>
   <th style="text-align:center;"> Panel 3: the pile of </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1. One score </td>
   <td style="text-align:center;"> z </td>
   <td style="text-align:center;"> z-scores </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2. One mean, σ known </td>
   <td style="text-align:center;"> z </td>
   <td style="text-align:center;"> z-scores of means </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3a. One mean, σ unknown </td>
   <td style="text-align:center;"> t </td>
   <td style="text-align:center;"> t-scores of means </td>
  </tr>
  <tr>
   <td style="text-align:left;"> <strong>4. Two means</strong> </td>
   <td style="text-align:center;"> <strong>t for M<sub>1</sub> − M<sub>2</sub></strong> </td>
   <td style="text-align:center;"> <strong>t-scores of differences</strong> </td>
  </tr>
</tbody>
</table>

.box-1.small[Same three panels: population, one sample, the statistic across samples]

???

Each step is the same idea. Panel 3 tells us how unexpected panel 2's number is. Today panel 1 has two populations, and panel 2 has one sample from each.

---

# Step 4: two populations, two samples

???

Step 4 on the blueprint. Here's the world H₀ describes, with numbers close to Mindspark's: both group means are 16, and both SDs are 5.8. The two curves sit on top of each other.

Panel 2 has two samples. Even with identical populations, their means differ. Panel 3 records that one difference.

---

# H₀ now says the gap is zero

<div class="math-display">\[
H_0\!: \mu_1 - \mu_2 = 0
\qquad\qquad
H_1\!: \mu_1 - \mu_2 \neq 0
\]</div>

.pull-left[
.box-inv-1.small[H₀: the two population means are the same. Across samples, gaps average zero]
]

.pull-right[
.box-inv-1.small[H₁: the two populations differ. Gaps pile up around some other value]
]

.box-1.small[A gap of 2.69 under H₀ can only be sampling error. How unusual would that be?]

???

Two tails stay our default, from Topic 8. The population means are still Greek: nobody has seen them.

---

# Predict it: How far does a gap fall?

.box-inv-8.small[
H₀ is true, with σ = 5.8 and n = 10 in each group. One group's mean typically falls 1.83 points from its μ. How far does the gap M₁ − M₂ typically fall from 0?
]

.options[
.option[
**A** About 0.9: half as far
]
.option[
**B** About 1.8: the same
]
.option[
**C** About 2.6: a bit farther
]
.option[
**D** About 3.7 or more: at least twice as far
]
]

???

About 6 minutes. Vote, turn to a neighbor, vote again, then show the answer.

Andy: create this poll in Poll Everywhere, then add the `.poll-bar` button from EDUC 265's 06-01 with its presenter id.

---

# Predict it: Answer

.options[
.option.wrong[
**A** About 0.9: half as far
]
.option.wrong[
**B** About 1.8: the same
]
.option.correct[
**C** About 2.6: a bit farther
]
.option.wrong[
**D** About 3.7 or more: at least twice as far
]
]

.box-inv-8.small.sp-after-half[
It's 1.83 × √2 = 2.59. Both means wobble, so their gap wobbles more than either one
]

.box-8.small[
But not twice as much: sometimes the two errors cancel out
]

---

---

# The gaps pile up around zero

???

Same world, 551 draws. The center is zero, because H₀ is true. Point at the spread: about 2.6 points is a typical miss. That's the standard error of the difference.

Mindspark's gap of 2.69 wouldn't be strange with groups of 10. But Mindspark's groups aren't that small.

---

# A gap has its own standard error

<div class="math-display">\[
\sigma_{M_1 - M_2} = \sqrt{\frac{\sigma^2}{n_1} + \frac{\sigma^2}{n_2}}
\]</div>

.pull-left[
.box-inv-1.small[Variances add, even when the means subtract. Each sample brings its own error]
]

.pull-right[
.box-inv-1.small[Here: √(33.64 / 10 + 33.64 / 10) = 2.59. With n = 200 each, it's 0.58]
]

.box-1.small[We're assuming both populations share one σ. Keep that in mind]

???

σ_M from Topic 6 was σ / √n. Square it and it's σ² / n. Here there are two of those, one per sample.

This is why two groups need more people than one. To get the same standard error as one mean with n students, each group needs 2n students. That's four times as many in total, as Topic 7 promised.

---

# When the groups differ, the pile moves

???

Group 1's mean is now 18.7, so d is 0.46. The group mean boxes are on the simulator page, and Lab 7 uses them. They're hidden on a slide. H₁ moves the pile away from zero. With groups of 10, it's still wide, so many samples can't tell the two worlds apart.

---

# The two-sample t

---

---

# In real research, panel 1 goes dark again

???

Just like Topic 8. We can't see σ, so we estimate it, and z becomes t. With two groups of 10, df is 18: each group gives 9.

---

# The fraction keeps its shape

<div class="math-display">\[
t = \frac{M - \mu}{\hat{\sigma}_M}
\qquad \text{becomes} \qquad
t = \frac{(M_1 - M_2) - (\mu_1 - \mu_2)}{\hat{\sigma}_{M_1 - M_2}}
\]</div>

.pull-left[
.box-inv-6.small[Top: the gap we saw, minus the gap H₀ claims. Under H₀, that second part is 0]
]

???

Same logic as every test so far: what we saw, over what sampling error alone would give us.

---

# Pool the two estimates of σ²

<div class="math-display">\[
\hat{\sigma}_p^2 = \frac{SS_1 + SS_2}{df_1 + df_2}
\]</div>

.pull-left[
.box-inv-6.small[We assume one σ², so we pool: add up the squared deviations and the df]
]

.box-6.small[The p is for "pooled". With equal n, it's the plain average of the two variances]

???

Topic 3's σ̂² is SS / df. This is the same recipe with two groups: all the squared deviations, over all the degrees of freedom.

The assumption has a name, homogeneity of variance. The test also needs independent observations, and roughly normal populations unless the samples are large. Software can drop the equal-variance assumption, in a version called Welch's t. With Mindspark, it barely changes t.

---

# Then the bottom of the fraction, and df

<div class="math-display">\[
\hat{\sigma}_{M_1 - M_2} = \sqrt{\frac{\hat{\sigma}_p^2}{n_1} + \frac{\hat{\sigma}_p^2}{n_2}}
\qquad\qquad
df = (n_1 - 1) + (n_2 - 1)
\]</div>

.box-6.small[Three moves: pool the variance, find the standard error, compute t]

???

The most common slip is using n1 + n2 − 1 for df. There are two means, so two df are spent.

---

# Side by side with Topic 8

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:center;"> One sample </th>
   <th style="text-align:center;"> Two samples </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Estimate </td>
   <td style="text-align:center;"> \(M\) </td>
   <td style="text-align:center;"> \(M_1 - M_2\) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Under H₀ </td>
   <td style="text-align:center;"> \(\mu\) </td>
   <td style="text-align:center;"> \(\mu_1 - \mu_2 = 0\) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Variance </td>
   <td style="text-align:center;"> \(\hat{\sigma}^2 = SS / df\) </td>
   <td style="text-align:center;"> \(\hat{\sigma}^2_p = (SS_1 + SS_2) / (df_1 + df_2)\) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Standard error </td>
   <td style="text-align:center;"> \(\hat{\sigma}_M = \sqrt{\hat{\sigma}^2 / n}\) </td>
   <td style="text-align:center;"> \(\hat{\sigma}_{M_1 - M_2} = \sqrt{\hat{\sigma}^2_p / n_1 + \hat{\sigma}^2_p / n_2}\) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> df </td>
   <td style="text-align:center;"> \(n - 1\) </td>
   <td style="text-align:center;"> \((n_1 - 1) + (n_2 - 1)\) </td>
  </tr>
</tbody>
</table>

.box-6.small[Every row is the same idea, built for two groups]

---

# Try it: 9 fifth graders

.box-inv-8.small.sp-after-half[Mindspark group: n = 3, M = 20.67, SS = 12.67. Control group: n = 6, M = 16.33, SS = 239.33]

.box-8.small.sp-after-half[Find the pooled variance σ̂²<sub>p</sub>, then the standard error σ̂<sub>M₁−M₂</sub>]

.box-inv-8.small[Is σ̂²<sub>p</sub> nearer 6.33 or 47.87, the two groups' own estimates? Why? Work alone, compare with a neighbor, then we vote]

???

About 6 minutes, with calculators. Vote with hands on the last question. The groups' variances look very different, but 3 students can't tell us much about a population's σ².

---

# Try it: Answer

.box-8.small.sp-after-half[It's nearer 47.87, the control group's estimate. That group has more students, so it counts for more]

.box-inv-8.small[A plain average would have given 27.10. Pooling doesn't treat 3 students like 6]

???

If someone wants t: (20.67 − 16.33) / 4.24 = 1.02. With t(7), the cutoff is 2.365, so we'd maintain H₀. Nine students can't settle much.

---

# Mindspark's test

---

---

# Steps 1 and 2: set up the test

.box-3.small.sp-after-half[At endline, do math scores differ across the two groups?]

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Step </th>
   <th style="text-align:left;"> In words </th>
   <th style="text-align:left;"> In symbols </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1. State H₀ </td>
   <td style="text-align:left;"> The two population means don't differ </td>
   <td style="text-align:left;"> μ<sub>Mindspark</sub> − μ<sub>control</sub> = 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2. Set the criteria </td>
   <td style="text-align:left;"> α = 0.05, two tails, and σ is unknown </td>
   <td style="text-align:left;"> df = (266 − 1) + (273 − 1) = 537 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:left;"> Reject H₀ if t is beyond the cutoffs </td>
   <td style="text-align:left;"> t(537) beyond ±1.964 </td>
  </tr>
</tbody>
</table>

???

The df matter here: 266 + 273 − 2 = 537. With this many people, the cutoff is nearly 1.96. The book's table stops short of 537, so use the next lower df, or the z cutoff.

---

# Step 3: three moves

<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:center;"> n </th>
   <th style="text-align:center;"> M </th>
   <th style="text-align:center;"> SS </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Mindspark </td>
   <td style="text-align:center;"> 266 </td>
   <td style="text-align:center;"> 18.68 </td>
   <td style="text-align:center;"> 9107.5 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:center;"> 273 </td>
   <td style="text-align:center;"> 15.99 </td>
   <td style="text-align:center;"> 9060.0 </td>
  </tr>
</tbody>
</table>

.box-inv-3.small[Pool: σ̂²<sub>p</sub> = (9107.5 + 9060.0) / 537 = **33.83**]

.box-inv-3.small[Standard error: σ̂<sub>M₁−M₂</sub> = √(33.83 / 266 + 33.83 / 273) = **0.50**]

.box-3.small[Compute t: (18.68 − 15.99) / 0.50 = **5.37**]

???

In 2024's deck, df appeared as 573 on two slides. It's 537. Sums of squares like these come from a spreadsheet, with the computational formula from Topic 3. Lab 7 does this on the full data.

---

# Step 4: our t is far past the cutoff

.box-3.small[5.37 is beyond 1.964, so we reject H₀. p is below 0.001]

???

If the two populations were the same, a gap this large would almost never happen. So we reject H₀. That doesn't say how big the effect is, so that comes next.

---

# How big is the gap? g = 0.46

<div class="math-display">\[
d = \frac{\mu_1 - \mu_2}{\sigma}
\qquad \text{is estimated by} \qquad
g = \frac{M_1 - M_2}{\hat{\sigma}_p}
\]</div>

.box-inv-3.small.sp-after-half[σ̂<sub>p</sub> = √33.83 = 5.82, so g = 2.69 / 5.82 = **0.46**]

.box-3.small[The groups sit about half an SD apart. Remember Topic 7: that's large for education]

???

Same as Topic 8: g is in people's SDs, not in standard errors, so n isn't in the formula. Use the pooled SD, not the standard error of the gap.

Topic 7's table from Kraft put the median effect at 0.10 SD. Please keep ignoring the book's Table 8.2 here.

---

# The interval for a gap

.box-inv-3.small.sp-after-half[Estimate ± margin of error: 2.69 ± 1.964 × 0.501 = [1.71, 3.68]]

.box-3.small[The interval leaves out 0, so it agrees with the test: reject H₀]

???

Same move as Topic 8: estimate plus or minus a cutoff times a standard error. In H₀'s world, the gap is 0, and 0 isn't inside.

Starting from unrounded numbers, the interval is 1.707 to 3.676. By hand with 2.69 and 0.50, it comes out at 1.71 to 3.67.

---

# How we report it

.box-3.medium.sp-after-half[In words]

.box-inv-3.small.sp-after-half[At endline, Mindspark students scored 2.69 points higher in math than control students. The difference is statistically significant at the 0.05 level, t(537) = 5.37, g = 0.46, with a 95% confidence interval of [1.71, 3.68]]

.box-3.small[The test, the effect size and the interval: all three, as in Topic 8]

???

Because a lottery split the groups, this gap can be read as the program's effect, for these students. We'll come back to why on Tuesday.

---

# Try it: One Mindspark center

.box-inv-8.small.sp-after-half[Students at the Govindpuri center. Mindspark: n = 34, M = 19.50, SS = 1408.5. Control: n = 35, M = 16.89, SS = 1455.5]

.box-8.small.sp-after-half[H₀: μ₁ − μ₂ = 0. α = 0.05, two tails. The cutoff is t(67) = ±1.996]

.box-inv-8.small[Find t and g, then decide. Work alone, compare with a neighbor, then we vote]

???

About 6 minutes, with calculators. Vote with hands on the decision: reject or maintain. Common slips: using n in place of df when pooling, or using the standard error as the bottom of g.

---

# Try it: Answer

.box-inv-8.small[σ̂<sub>M₁−M₂</sub> = √(42.75 / 34 + 42.75 / 35) = **1.57**]
]

.box-inv-8.small[g = 2.61 / √42.75 = **0.40**]
]

.box-8.small[A similar g to the whole study's 0.46, and a different decision. With 69 students, we can't tell]

???

The same lesson as Topic 8: significant means hard to explain by sampling error. It doesn't mean big, and "maintain" doesn't mean no effect.

---

# Before Tuesday

.box-inv-8.medium.sp-after-half[Friday, February 19: Lab 7, two samples. Bring a calculator and a laptop]

.box-inv-8.medium.sp-after-half[Sunday, February 21 at 6 PM: Mastery and Problem Set for Chapter 10]

.box-8.medium[Read Chapter 11 for Tuesday. What if the same people take the test twice?]

???

Lab 7 does today's two-sample t on the full data, in a spreadsheet. Today's formula works when the two groups hold different people. But everyone at endline also took the baseline test, and so far we've ignored that.

---

# Session 2: Tuesday, February 23

.box-inv-8.small.sp-after-half[Thursday in one line: a gap between two groups gets its own standard error, and a pooled σ̂²]

.box-5.medium.sp-after-half[One group, two scores]

.box-7.medium.sp-after-half[Why measure twice?]

.box-4.medium[Who gets the credit?]

???

Three blocks today. There's no new formula, and the last block is the point of the whole topic.

---

# One group, two scores

---

---

# Some studies test people twice

.box-inv-5.small[Different people in each group]

.box-inv-5.small[One score per person]

.box-inv-5.small[Compare two groups: Thursday]
]

.box-inv-5.small[The same people, before and after]

.box-inv-5.small[Two scores per person]

.box-inv-5.small[Compare each person with themselves: today]
]

???

The book's Chapter 11 also covers matched pairs: different people, paired up on something like a baseline score. They work the same way, with one number per pair.

---

# Two scores become one: the gain

<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:left;"> Student </th>
   <th style="text-align:center;"> Before (baseline) </th>
   <th style="text-align:center;"> After (endline) </th>
   <th style="text-align:center;"> Gain = after − before </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Student 1 </td>
   <td style="text-align:center;"> 13 </td>
   <td style="text-align:center;"> 18 </td>
   <td style="text-align:center;"> 5 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student 2 </td>
   <td style="text-align:center;"> 9 </td>
   <td style="text-align:center;"> 15 </td>
   <td style="text-align:center;"> 6 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student 3 </td>
   <td style="text-align:center;"> 12 </td>
   <td style="text-align:center;"> 16 </td>
   <td style="text-align:center;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student 4 </td>
   <td style="text-align:center;"> 8 </td>
   <td style="text-align:center;"> 14 </td>
   <td style="text-align:center;"> 6 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student 5 </td>
   <td style="text-align:center;"> 7 </td>
   <td style="text-align:center;"> 20 </td>
   <td style="text-align:center;"> 13 </td>
  </tr>
</tbody>
</table>

.box-5.small[Now each student has one number. We've seen what to do with one column of numbers]

???

These are the first five students in the file who were in the Mindspark group. Their gains are positive, but gains can be negative.

---

# A paired test is a one-sample t on gains

<div class="math-display">\[
t = \frac{M - \mu}{\hat{\sigma}_M}
\qquad \text{where} \qquad
\hat{\sigma}_M = \frac{\hat{\sigma}}{\sqrt{n}}
\]</div>

.pull-left[
.box-inv-5.small[There's no new formula. This is Topic 8's, and now X is a person's gain]
]

.pull-right[
.box-inv-5.small[M is the mean gain, and n is the number of people. H₀ says μ = 0, so df = n − 1]
]

.box-5.small[H₀: on average, nothing changes between before and after]

.book-tag[In the book: a gain is D, and our M and σ̂<sub>M</sub> are M<sub>D</sub> and s<sub>M<sub>D</sub></sub>]

???

Here's the good news of the day. Whatever you can do with a one-sample t, you can do with gains: g, intervals, one tail or two. n counts people, not scores. A common slip is using 2n.

---

# All 266 Mindspark students

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Step </th>
   <th style="text-align:left;"> Mindspark students' gains </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1. State H₀ </td>
   <td style="text-align:left;"> μ = 0: no change on average </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2. Set the criteria </td>
   <td style="text-align:left;"> α = 0.05, two tails. t(265) beyond ±1.969 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3. Compute t </td>
   <td style="text-align:left;"> M = 7.82, σ̂ = 4.97, σ̂<sub>M</sub> = 4.97 / √266 = 0.305<br>t = 7.82 / 0.305 = <strong>25.68</strong> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4. Decide </td>
   <td style="text-align:left;"> Reject H₀: 25.68 is far beyond 1.969 </td>
  </tr>
</tbody>
</table>

.box-5.small[Mean gain: 7.82 points. g = 1.57, and the 95% interval is [7.22, 8.42]]

???

That's an enormous t. Hold on to it. Is 7.82 points what Mindspark did? We come back to that in the last block.

---

# Step 3b: each person gets a line

???

Step 3b on the blueprint, between one mean and two means, because that's what it is: one mean, of gains. The settings are close to Mindspark's: before scores around 11, an SD of 5, and ρ = 0.5.

Press +50. The pile in panel 3 is t-scores of mean gains.

---

# Try it: Five students' gains

.box-inv-8.small.sp-after-half[Gains: 5, 6, 4, 6, 13. So M = 6.8 and SS = 50.8]

.box-8.small.sp-after-half[H₀: μ = 0. α = 0.05, two tails. The cutoff is t(4) = ±2.776]

.box-inv-8.small[Find σ̂, σ̂<sub>M</sub> and t, then decide. Work alone, compare with a neighbor, then we vote]

???

About 6 minutes, with calculators. Same five students as before. Vote with hands on the decision. Students should notice that this is exactly Topic 8's calculation.

---

# Try it: Answer

.box-inv-8.small[σ̂<sub>M</sub> = 3.56 / √5 = **1.59**]
]

.box-inv-8.small[4.27 is beyond 2.776: **reject H₀**]
]

.box-8.small[We report: t(4) = 4.27, p = 0.013. Even five students can show a change, if they all move the same way]

???

The last line leads to the next block. Why can so few students show a change? Because each one is compared with themselves.

---

# Why measure twice?

---

---

# Strong students stay fairly strong

.box-inv-7.small[A gain takes each student's own starting level out of the picture]

???

Across all 539 students, the correlation between baseline and endline is r = 0.50. We'll meet r properly in Topic 11. For now, it says how tightly the two scores move together. The population version is ρ.

---

# Same students, two ways to test

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:center;"> Standard error </th>
   <th style="text-align:center;"> t </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Pretending they're two separate groups </td>
   <td style="text-align:center;"> 0.429 </td>
   <td style="text-align:center;"> t(530) = 18.23 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Using each student's gain </td>
   <td style="text-align:center;"> 0.305 </td>
   <td style="text-align:center;"> t(265) = 25.68 </td>
  </tr>
</tbody>
</table>

.box-inv-7.small.sp-after-half[Same 266 students, same mean gain of 7.82. Pairing cuts the standard error by 29%]

.box-7.small[The first row is wrong here: the two columns of scores aren't independent]

???

Both tests agree here, because the change is so large. With a smaller change, the wrong test could easily miss it. Pairing means fewer students are needed to detect the same change, which is a real saving for a study.

---

# ρ sets the spread of the gains

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> How tightly before and after move together </th>
   <th style="text-align:center;"> ρ </th>
   <th style="text-align:center;"> SD of gains </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Not at all </td>
   <td style="text-align:center;"> 0.0 </td>
   <td style="text-align:center;"> 7.07 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Moderately </td>
   <td style="text-align:center;"> 0.5 </td>
   <td style="text-align:center;"> 5.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Very tightly </td>
   <td style="text-align:center;"> 0.9 </td>
   <td style="text-align:center;"> 2.24 </td>
  </tr>
</tbody>
</table>

.box-inv-7.small.sp-after-half[Each score here has σ = 5. The more tightly they move, the steadier the gains]

.box-7.small[A smaller SD of gains means a smaller σ̂<sub>M</sub>, and so a bigger t]

???

If someone wants the formula, the SD of gains is σ√(2(1 − ρ)). It isn't needed for this course, because we compute the gains and take their SD directly.

With ρ = 0.5, the SD of gains equals σ. Mindspark's r is about that: gains have σ̂ = 4.97 for the treatment group.

---

# Predict it: Tighten the scores

.box-inv-8.small.sp-after-half[Keep the mean gain the same. Raise ρ from 0.5 to 0.9, so before and after move together much more tightly]

.box-8.small[What happens to the pile of t-scores in panel 3?]

.options[
.option[
**A** It moves away from 0
]
.option[
**B** It moves toward 0
]
.option[
**C** It stays where it is
]
.option[
**D** It depends on the mean before
]
]

???

About 6 minutes for this slide and the next two. Hands are fine. Neighbor first, then vote.

---

# Predict it: With ρ = 0.5

???

This is the starting point: a mean gain of 2.5, with ρ = 0.5. Read off the share beyond the cutoffs. Then go to the next slide for ρ = 0.9.

---

# Predict it: With ρ = 0.9, a bigger t

???

The answer is A. Same mean gain, but the bottom of the fraction shrank: σ<sub>M</sub> of the gains fell from 1.58 to 0.71. These are two presets, because the ρ slider is hidden on a slide. It's on the simulator page.

---

---

# Pairing has a price

.box-inv-7.small[Each person is their own comparison]

.box-inv-7.small[A smaller standard error]

.box-inv-7.small[Fewer people for the same power]
]

.box-inv-7.small[Time passes between the two scores]

.box-inv-7.small[Anything that changed gets mixed into the gain]

.box-inv-7.small[The test can't say what caused it]
]

???

The test can tell us that scores changed. It can't tell us why. The book calls these time-related factors and order effects, and it covers counterbalancing, which swaps the order of conditions. A pretest and posttest can't be swapped.

---

# Who gets the credit?

---

---

# Did Mindspark add 7.82 points?

.box-4.medium.sp-after-half[Students gained 7.82 points, t(265) = 25.68]

.box-inv-4.small.sp-after-half[The test is right: their scores really rose. Sampling error can't explain that]

.box-inv-4.small.sp-after-half[Imagine a press release: "Students gain nearly 8 points with Mindspark!"]

.box-4.small[What else was going on for these students during those months?]

???

This is what a study without a control group would report. Ask students to name what else changed. Take a few answers before the next slide.

---

# Mindspark wasn't all that happened

.box-inv-4.small[Children got older]
]

.box-inv-4.small[The endline test wasn't the same test]
]

.box-4.small[A paired test can't separate any of this from the program]

???

The last one is specific to our data. The endline tests were different from the baseline test, out of 34 or 35 by grade band, so a raw gain can't be read as pure learning.

The scaled scores in the data hint at how much that matters. In baseline SDs, control students gained 0.35, against about 1.21 in raw points. If those scales are linked across rounds, which the codebook doesn't say, the new test accounts for a good part of the raw gain.

---

# Predict it: Students with no program

.box-inv-8.small[
Control students lost the lottery, and had no Mindspark during those 4.5 months. On average, how much did their math scores rise?
]

.options[
.option[
**A** About 0 points
]
.option[
**B** About 2 points
]
.option[
**C** About 5 points
]
.option[
**D** About 8 points, like the Mindspark group
]
]

???

About 6 minutes. Vote, turn to a neighbor, vote again, then show the answer.

Andy: create this poll in Poll Everywhere, then add the `.poll-bar` button from EDUC 265's 06-01 with its presenter id.

---

# Predict it: Answer

.options[
.option.wrong[
**A** About 0 points
]
.option.wrong[
**B** About 2 points
]
.option.correct[
**C** About 5 points
]
.option.wrong[
**D** About 8 points, like the Mindspark group
]
]

.box-inv-8.small.sp-after-half[
Control students gained 4.74 points, t(272) = 15.48
]

.box-8.small[
A paired test finds a huge "effect" with no program at all
]

---

---

# Both groups rose, one rose more

???

This is the picture to remember. The control line is what would have happened anyway. Both lines rise. The program's contribution is how far the Mindspark line ends above the control line.

---

# A paired test alone over-credits

<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:left;"> What we'd conclude </th>
   <th style="text-align:center;"> Points </th>
   <th style="text-align:center;"> Where it comes from </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Paired test, Mindspark students alone </td>
   <td style="text-align:center;"> 7.82 </td>
   <td style="text-align:center;"> Gain: t(265) = 25.68 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Paired test, control students alone </td>
   <td style="text-align:center;"> 4.74 </td>
   <td style="text-align:center;"> Gain: t(272) = 15.48 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Two-sample test at endline </td>
   <td style="text-align:center;"> <strong>2.69</strong> </td>
   <td style="text-align:center;"> Gap: t(537) = 5.37 </td>
  </tr>
</tbody>
</table>

.box-4.small.sp-after-half[7.82 / 2.69 = **2.9**: nearly three times the program's effect]

.box-inv-4.small[The paired test isn't wrong. It asks if scores changed, not why]

???

Be careful with the words. Every number here is right. The mistake is reading a gain as an effect.

---

# Why doesn't 7.82 − 4.74 give 2.69?

.box-inv-4.small.sp-after-half[Subtracting gains gives 3.09, a two-sample t on gains: t(537) = 7.15]

.box-inv-4.small.sp-after-half[It differs from 2.69 by 0.40: the gap at baseline, which favored the control group by chance]

.box-4.small.sp-after-half[Both are fair estimates, because both compare with the control group]

.box-inv-4.small[The endline gap ignores baseline. Gains use it, so their standard error is smaller: 0.43, not 0.50]

???

A student will do this subtraction, so get ahead of it. By hand, the rounded numbers give 3.08. Unrounded, it's 3.09. A fair lottery doesn't make groups identical, as we saw in Topic 6. Among these 539 students, the baseline gap of 0.40 points isn't significant.

The two estimates, 2.69 and 3.09, give the same story, and this is the same two-sample test from Thursday, with gains as the scores. Regression later in the course gives a third way to use baseline.

---

# This is why control groups exist

.box-4.medium.sp-after-half[What would have happened without the program?]

.box-inv-4.small.sp-after-half[We can't watch the same students both with and without Mindspark]

.box-inv-4.small.sp-after-half[The lottery gave us the next best thing: similar students, who lived through the same months]

.box-4.small[The control group's rise is what would have happened anyway. Whatever goes beyond it is the program]

???

This is the idea behind the whole study. The word for that missing world is the counterfactual. School, growing up and the new test all hit both groups. The only thing the lottery let differ was the program.

---

# Play it: Name that test

.box-inv-8.small.sp-after-half[**1** One-sample t  ·  **2** Paired t  ·  **3** Two-sample t. Hold up your fingers]

.box-8.small[**A** 40 students take a reading test in September and again in June]

.box-8.small[**B** Tutored students and untutored students each take one final exam]

.box-8.small[**C** 25 teachers' scores are compared with a national mean of 100]

.box-8.small[**D** Each child is matched with a similar child, and one of each pair gets the program]

???

A quick game with hands, about 3 minutes. Read each clue, count to three, and look around the room. Scan the room for clue D. Ask: what could explain the gain in clue A?

---

# Play it: Answers

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Clue </th>
   <th style="text-align:left;"> Test </th>
   <th style="text-align:left;"> The giveaway </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:left;"> Paired t </td>
   <td style="text-align:left;"> Same students, two scores. But no control group </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:left;"> Two-sample t </td>
   <td style="text-align:left;"> Different students, one score each </td>
  </tr>
  <tr>
   <td style="text-align:left;"> C </td>
   <td style="text-align:left;"> One-sample t </td>
   <td style="text-align:left;"> One mean against a known value </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D </td>
   <td style="text-align:left;"> Paired t </td>
   <td style="text-align:left;"> Each pair gives one difference </td>
  </tr>
</tbody>
</table>

.box-8.small[Ask two things: How many scores per person? Is there a comparison group?]

---

# The blueprint so far

<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:left;"> Blueprint step </th>
   <th style="text-align:center;"> Statistic </th>
   <th style="text-align:center;"> Bottom of the fraction </th>
   <th style="text-align:center;"> df </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1. One score </td>
   <td style="text-align:center;"> z </td>
   <td style="text-align:center;"> σ </td>
   <td style="text-align:center;"> None </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2. One mean, σ known </td>
   <td style="text-align:center;"> z </td>
   <td style="text-align:center;"> σ<sub>M</sub> </td>
   <td style="text-align:center;"> None </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3a. One mean, σ unknown </td>
   <td style="text-align:center;"> t </td>
   <td style="text-align:center;"> σ̂<sub>M</sub> </td>
   <td style="text-align:center;"> n − 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3b. Mean gain </td>
   <td style="text-align:center;"> t </td>
   <td style="text-align:center;"> σ̂<sub>M</sub> of the gains </td>
   <td style="text-align:center;"> n − 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4. Two means </td>
   <td style="text-align:center;"> t </td>
   <td style="text-align:center;"> σ̂<sub>M₁−M₂</sub> </td>
   <td style="text-align:center;"> n₁ + n₂ − 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5. Two or more means: next </td>
   <td style="text-align:center;"> F </td>
   <td style="text-align:center;"> Topic 10 </td>
   <td style="text-align:center;">  </td>
  </tr>
</tbody>
</table>

.box-8.small[Same fraction every time: what we saw, over what sampling error alone would give]

???

On Thursday, step 5 goes past two groups, and the fraction becomes a ratio of variances.

---

# Where are we?

.box-inv-1.small[Summarize data]

.box-inv-1.small[Visualize data]
]

.box-inv-6.small[Did scores change?]
]

.box-inv-3.small[Did the program cause it?]

???

A paired test can tell us that scores rose. Only a comparison group, here one made by a lottery, lets us say why.

---

# What you can do now

.box-inv-8.small[Run a two-sample t-test: pool, find the standard error, compute t]

.box-inv-8.small[Report it with Hedges' g and a confidence interval]

.box-inv-8.small[Turn before and after scores into gains, and run a paired t-test]

.box-inv-8.small[Say why pairing shrinks the standard error]

.box-inv-8.small[Explain why a gain isn't an effect, and what a control group adds]

???

Lab 8 on Friday starts with a 15-minute individual check on two samples, then does the gains test on the full data, in a spreadsheet.

---

# Next steps

ANOVA. Read sections <span style="white-space: nowrap;">12-1 to 12-4</span> and 12-6, and the start of 12-5
]

A 15-minute individual check on two samples, then gains and ANOVA. Bring a calculator and a laptop
]

.box-3.small[Sunday, February 28, 6 PM: Chapters 11 and 12 are due in MindTap]

.picture-small img {
  display: block;
  margin: 0 auto;
  max-width: 100%;
  max-height: 230px;
}

.anteater img {
  display: block;
  margin: 0.15em auto 0;
  max-height: 105px;
}

/* This chunk sits at the end because any output before the first slide's
   "class:" line stops remark from reading that slide's properties */

/* Long titles get two smaller lines inside the 110 px title bar */
.remark-slide-content.long-title h1 {
  font-size: 1.05em;
  line-height: 1.05;
  margin-top: -100px;
  margin-bottom: 55px;
}

.source {
  position: absolute;
  bottom: 18px;
  left: 60px;
  font-size: 0.55em;
  color: #47505a;
}

.book-tag {
  margin-top: 0.7em;
  font-size: 0.65em;
  color: #47505a;
  text-align: right;
}

table.deck-table {
  width: 92%;
  margin: 0.4em auto;
  border-collapse: collapse;
  font-size: 0.72em;
}

table.deck-table.tight td,
table.deck-table.tight th {
  padding: 0.12em 0.6em;
}

table.deck-table th {
  border-bottom: 2px solid #244a70;
  padding: 0.3em 0.6em;
}

table.deck-table td {
  border-bottom: 1px solid #c9d1d9;
  padding: 0.3em 0.6em;
}

.options {
  display: grid;
  grid-template-columns: 1fr 1fr;
  gap: 0.35em 1em;
  margin-top: 0.6em;
  margin-bottom: 0.8em;
  font-size: 0.65em;
}

.option {
  background-color: #f3f3f3;
  border: 2px solid #cccccc;
  padding: 0.35em 0.6em;
}

.option.correct {
  background-color: #b18323;
  border-color: #b18323;
  color: white;
}

.option.wrong {
  opacity: 0.6;
}

.poll-respond {
  display: block;
  font-size: 0.5em;
  text-align: center;
  margin-top: 0.1em;
}

/* The printed PDF keeps the question and drops the answer slide. A slide that
   waits for a live button press would print empty, so it goes too */
@media print {
  .remark-slide-container:has(.poll-answer),
  .remark-slide-container:has(.press-slide) {
    display: none !important;
  }
}
</style>