Hypothesis testing

# Hypothesis testing

**Topic 7 · Thursday, January 28, and Tuesday, February 2**

]

---

# Where are we?

.box-inv-1.small[Summarize data]

.box-inv-1.small[Visualize data]
]

.box-inv-6.small[From a sample to a population]

???

We've moved. Until now we described what we saw. From today we use one sample to say something about a population we can't see.

This topic takes two sessions, so six blocks in all.

---

# The question we keep asking

.box-8.medium.sp-after-half[If what I think about the population is true, how weird is my sample?]

.box-inv-8.small.sp-after-half[Topic 6 gave us the tool: z = (M − μ) / σ<sub>M</sub> says how far a mean is from μ]

.box-inv-8.small[Today's new question: how weird must it be before I change my mind?]

???

A z-score for a mean measures weirdness. A hypothesis test adds a rule, written down in advance, for what to do with that number.

---

# Plan for today

.box-1.medium.sp-after-half[Two steps before any data]

.box-6.medium.sp-after-half[Observe and decide]

.box-3.medium.sp-after-half[When the decision is wrong]

???

Three blocks of about 20 minutes. Each ends with students doing something: predict, try, play.

---

# Two steps before any data

---

---

# The four steps of a hypothesis test

<table class="deck-table">
 <thead>
  <tr>
   <th> Step </th>
   <th> What we do </th>
   <th> On the blueprint </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td> 1. State a null hypothesis, H<sub>0</sub> </td>
   <td> Say what we assume about the population </td>
   <td> Panel 1 </td>
  </tr>
  <tr>
   <td> 2. Set criteria </td>
   <td> Say what would be too unexpected if H<sub>0</sub> is true </td>
   <td> Panel 3 </td>
  </tr>
  <tr>
   <td> 3. Observe a sample </td>
   <td> Ask how unexpected this one sample is </td>
   <td> Panel 2 </td>
  </tr>
  <tr>
   <td> 4. Reject or maintain H<sub>0</sub> </td>
   <td> Decide what to believe about the population </td>
   <td> Back to panel 1 </td>
  </tr>
</tbody>
</table>

.box-1.small[These same four steps return in every test this quarter: z, t, F and regression]

???

This is the thread of the course. If students remember one slide from this deck, it should be this one. The order on the blueprint is 1, 3, 2: we decide what's weird before we look at our sample.

---

# Step 1: State a null hypothesis

.box-inv-1.small.sp-after-half[The null hypothesis, H<sub>0</sub>, says that in the population there's no change, no difference or no relationship, or it names a specific value]

.box-1.small.sp-after-half[Our example: a reading test has μ = 100 and σ = 15. A school tries a new program with 10 students]

???

These are always statements about the population, so they use Greek letters. Nobody writes H<sub>0</sub>: M = 100, because we can see M.

The numbers match the simulator's defaults on purpose.

---

# The truth stays hidden

.box-inv-1.small[Maybe the program really moved μ to 113.4. We can't see it]

???

H<sub>0</sub> might be true or false. All we'll ever get is a sample. So the logic runs the other way: assume H<sub>0</sub>, then see whether the sample fits. We come back to the dashed curve on Tuesday.

---

# Step 2: Say what's too unexpected

.pull-right[
.box-inv-1.small[The **critical region** holds the 5% of means least likely if H<sub>0</sub> is true]
]

???

Alpha is also called the level of significance. ±1.96 comes from the table in Topic 5: 2.5% in each tail. We pick alpha before seeing any data.

Now students can finally stop ignoring the red lines in the simulator.

---

# Step 2 on the blueprint: press +500

???

These are the cutoffs from last class. There's no true effect here, so H<sub>0</sub> is true. **Press +500.** Read "Beyond the cutoffs". It should say about 5%. Press +500 again if you like.

---

# About 5% land beyond the cutoffs

???

The red dots are real samples from a population where H<sub>0</sub> is true. They still land in the critical region. Hold on to that for block 3.

---

# Predict it: Ten times the sample

.box-inv-8.small[
H<sub>0</sub> is still true. With n = 10, about 5% of z-scores landed beyond ±1.96. Now each sample has 100 people. What share lands beyond ±1.96?
]

.options[
.option[
**A** Far less than 5%: bigger samples are more accurate
]
.option[
**B** About 5%
]
.option[
**C** Far more than 5%
]
.option[
**D** It depends on the value of μ
]
]

???

About 6 minutes. Vote, turn to a neighbor, vote again, then show the answer.

Andy: create this poll in Poll Everywhere, then add the `.poll-bar` button from EDUC 265's 06-01 with its presenter id.

---

# Predict it: Answer

.options[
.option.wrong[
**A** Far less than 5%: bigger samples are more accurate
]
.option.correct[
**B** About 5%
]
.option.wrong[
**C** Far more than 5%
]
.option.wrong[
**D** It depends on the value of μ
]
]

.box-inv-8.small.sp-after-half[
A bigger n does make means land closer to μ. But z divides by σ<sub>M</sub>, which shrinks by just as much
]

.box-8.small[
So z-scores keep the same curve, and α stays what we set it to
]

---

# Predict it: Still about 5% at n = 100

???

Press +500 once or twice more, so the students see the share settle.

---

# Observe and decide

---

---

# Step 3: Observe one sample

???

This is the only sample a real researcher would see. Read panel 2 aloud: M = 105.5, z = 1.16, p = 0.246. The pink line in panel 3 shows where it falls. We work out the z and the p by hand next.

---

# Step 3: How far, in standard errors?

.box-inv-6.small[First, the standard error]

$$
\sigma_M = \frac{\sigma}{\sqrt{n}} = \frac{15}{\sqrt{10}} = 4.74
$$

.box-inv-6.small[Then the test statistic: our mean sits 1.16 standard errors above what H<sub>0</sub> says]

$$
z = \frac{M - \mu}{\sigma_M} = \frac{105.5 - 100}{4.74} = 1.16
$$

???

The z-score is our *test statistic*: one number that says how far the sample is from H<sub>0</sub>. Later tests swap in t and F, but they do the same job.

---

# Step 3: The p-value is an area

.box-inv-6.small[The **p-value** is the probability of a mean at least this extreme, if H<sub>0</sub> is true]

???

Both tails count, because H<sub>1</sub> says μ ≠ 100: a surprise in either direction counts. From the table, the tail beyond 1.16 is 0.1231. Double it to get 0.246.

So if H<sub>0</sub> is true, 24.6% of samples of 10 land this far out or farther. That's not weird. The simulator gives p as a number, but it doesn't shade this area, which is why this one is a plot.

---

# Step 4: p is above α, so we maintain H<sub>0</sub>

<div class="step-row"><div class="box-1 step-card"><b>1. State a hypothesis</b><br>H<sub>0</sub>: μ = 100</div><div class="box-6 step-card"><b>2. Set criteria</b><br>α = 0.05</div><div class="box-3 step-card"><b>3. Observe</b><br>−1.96 &lt; z &lt; 1.96<br>p = 0.246 &gt; 0.05</div><div class="box-8 step-card"><b>4. Decide</b><br>Here, maintain H<sub>0</sub></div></div>

???

This is the summary box for the whole test. Two checks, same answer: z is not in the critical region, and p is bigger than alpha.

The simulator stops at p. It doesn't write the decision, so that's our job.

---

# Maintaining H<sub>0</sub> doesn't prove it

.box-6.medium.sp-after-half[We looked for evidence against H<sub>0</sub> and didn't find enough]

.box-6.small[So we say "maintain H<sub>0</sub>", and never "H<sub>0</sub> is true"]

???

Think of a court. "Not guilty" doesn't mean "innocent". It means the evidence fell short.

---

# What the p-value doesn't tell you

<table class="deck-table">
 <thead>
  <tr>
   <th> A common reading </th>
   <th> Why it's wrong </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td> "There's a 24.6% chance that H<sub>0</sub> is true" </td>
   <td> p assumes H<sub>0</sub> is true, so it can't be the chance that it is </td>
  </tr>
  <tr>
   <td> "A small p means a big effect" </td>
   <td> p also depends on n. We show this on Tuesday </td>
  </tr>
  <tr>
   <td> "p > 0.05 means there's no effect" </td>
   <td> The sample may just be too small to show it </td>
  </tr>
</tbody>
</table>

.box-6.small[p answers one question: if H<sub>0</sub> is true, how unusual is a sample like mine?]

???

The first one is the most common mistake, among researchers too.

---

# The four steps, ready to use

<table class="deck-table">
 <thead>
  <tr>
   <th> Step </th>
   <th> In practice </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td> 1. State H<sub>0</sub> </td>
   <td> H<sub>0</sub>: μ = some value, in the population </td>
  </tr>
  <tr>
   <td> 2. Set criteria </td>
   <td> Pick α. Find the critical values, such as z = ±1.96 </td>
  </tr>
  <tr>
   <td> 3. Observe a sample </td>
   <td> Compute the test statistic: z = (M − μ) / σ<sub>M</sub>. Find p </td>
  </tr>
  <tr>
   <td> 4. Reject or maintain H<sub>0</sub> </td>
   <td> Reject if z is in the critical region, which is the same as p &lt; α </td>
  </tr>
</tbody>
</table>

.box-6.small[z and p always agree on the decision. Use whichever is quicker]

???

If z is past ±1.96, then p must be below 0.05, because that's how ±1.96 was chosen. This is the version to use on Friday's lab.

---

# Try it: The ninth graders

.box-inv-8.small.sp-after-half[Treat all 619 Mindspark students as a population: at baseline, μ = 11.12 and σ = 3.90 in math]

.box-inv-8.small.sp-after-half[The 84 ninth graders have M = 12.39. If they were a random draw from the 619, would that mean be too unexpected?]

.box-8.small.sp-after-half[Run all four steps, with α = 0.05]

.box-inv-8.small[Work alone for two minutes. Compare with a neighbor. Then we vote: reject or maintain?]

???

About 6 minutes, with calculators and the table. Hands for the vote.

---

# Try it: Answer

<div class="step-row"><div class="box-1 step-card"><b>1. State a hypothesis</b><br>H<sub>0</sub>: μ = 11.12</div><div class="box-6 step-card"><b>2. Set criteria</b><br>α = 0.05, so z = ±1.96</div><div class="box-3 step-card"><b>3. Observe</b><br>z = 2.99<br>p = 0.003 &lt; 0.05</div><div class="box-8 step-card"><b>4. Decide</b><br>Here, reject H<sub>0</sub></div></div>

.box-inv-8.small.sp-after-half[
σ<sub>M</sub> = 3.90 / √84 = 0.43, so z = (12.39 − 11.12) / 0.43 ≈ 2.99
]

.box-8.small[They don't look like a random draw from the 619. The test can't say why]

???

They're in a higher grade, so a gap isn't a shock. Rejecting H<sub>0</sub> tells us the gap is too big for sampling error alone. It says nothing about the cause.

By hand, rounded inputs give about 2.95. Anything near 3 gives the same decision.

For anyone who asks: the ninth graders are part of the 619, which makes the true standard error a little smaller. That doesn't change the point.

---

# When the decision is wrong

---

---

# Same population, a different sample

???

Same settings, just a different draw. M = 113.5, z = 2.85, p = 0.004. Ask what they'd conclude if this were their one sample.

---

# This sample says reject H<sub>0</sub>

<div class="step-row"><div class="box-1 step-card"><b>1. State a hypothesis</b><br>H<sub>0</sub>: μ = 100</div><div class="box-6 step-card"><b>2. Set criteria</b><br>α = 0.05</div><div class="box-3 step-card"><b>3. Observe</b><br>z &gt; 1.96<br>p = 0.004 &lt; 0.05</div><div class="box-8 step-card"><b>4. Decide</b><br>Here, reject H<sub>0</sub></div></div>

.box-inv-3.small[We'd write: the mean differs from 100, and it's **statistically significant** at the 5% level (p < .05)]

???

"Statistically significant" is a technical phrase. It means only that p fell below alpha. It doesn't mean large or important: that's Tuesday.

---

# But we know H<sub>0</sub> was true here

.box-3.medium.sp-after-half[The simulator set μ = 100, and we still rejected H<sub>0</sub>]

.box-inv-3.small.sp-after-half[We would have falsely concluded that μ ≠ 100]

.box-inv-3.small.sp-after-half[Nothing went wrong with the sample or the math. This was an unlucky draw]

.box-3.small[In real research, nobody tells you afterward]

???

This is what a simulator gives us that real data can't: we know the truth, so we can see when a test gets it wrong.

---

# Two ways to be wrong

<table class="deck-table error-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Our decision </th>
   <th style="text-align:center;"> If H<sub>0</sub> is true </th>
   <th style="text-align:center;"> If H<sub>0</sub> is false </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Reject H<sub>0</sub> </td>
   <td style="text-align:center;"> <b>Type 1 error</b>: a false alarm </td>
   <td style="text-align:center;"> Correct </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Maintain H<sub>0</sub> </td>
   <td style="text-align:center;"> Correct </td>
   <td style="text-align:center;"> <b>Type 2 error</b>: a miss </td>
  </tr>
</tbody>
</table>

???

We get to pick a row. We never get to see the column. Our last sample was the top-left cell, and the first one was the bottom-left cell.

The simulator has no live version of this table, so we build it by hand.

---

# Every red dot is a false alarm

???

So this is what alpha means. By setting α = 0.05, we accept that 5% of the time when H<sub>0</sub> is true, we'll make a Type 1 error. The chance of a Type 1 error is alpha. We chose it.

---

# α = 0.05 is a convention, not a law

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:center;"> α </th>
   <th style="text-align:center;"> Critical z </th>
   <th style="text-align:center;"> Type 1 errors, if H<sub>0</sub> is true </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:center;"> 0.10 </td>
   <td style="text-align:center;"> ±1.64 </td>
   <td style="text-align:center;"> About 1 sample in 10 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 0.05 </td>
   <td style="text-align:center;"> ±1.96 </td>
   <td style="text-align:center;"> About 1 sample in 20 </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> ±2.58 </td>
   <td style="text-align:center;"> About 1 sample in 100 </td>
  </tr>
</tbody>
</table>

.box-3.small.sp-after-half[Researchers mostly use 0.05, sometimes 0.01 or 0.10]

.box-inv-3.small[p = 0.099 and p = 0.101 are nearly the same evidence, even if α = 0.10 splits them. Report the p-value, not just the verdict]

???

"Significant" and "not significant" sound like opposites. The evidence behind them can be almost the same. There's nothing special about any of these numbers.

---

# Play it: Name that error

.box-8.small.sp-after-half[Answer with a neighbor, as a question. Nothing is scored]

???

About 6 minutes. Read one clue at a time, in order. Take answers by hands or by calling out.

---

# Play it: Answers

???

The 300 clue leads into Tuesday. We can't shrink both errors just by moving alpha. Power is about how else to reduce the misses.

---

# Before Tuesday

Sampling distributions and the four steps. Bring a calculator and a laptop
]

Read sections 8-4 to 8-6. Tuesday we ask: what if H<sub>0</sub> is false?
]

.box-3.small[Mastery and Problem Set for Chapter 7 are due Sunday, January 31 at 6 PM]

---

# Session 2: Tuesday, February 2

.box-inv-8.small.sp-after-half[Thursday in one line: four steps, and two ways to get the decision wrong]

.box-5.medium.sp-after-half[What if H<sub>0</sub> is false?]

.box-7.medium.sp-after-half[Power]

.box-4.medium.sp-after-half[Planning a study, reading a result]

???

Three blocks of about 20 minutes, each ending with something to do: predict, predict, try.

---

# What if H<sub>0</sub> is false?

---

---

# So far H<sub>0</sub> was true. What if it isn't?

.box-inv-5.small[Suppose the program really works, and the true mean is 113.4]

???

The dashed curve from Thursday. On Thursday we lived in the world where H<sub>0</sub> is true. Now let's see what the test does when it's false.

---

# Effect size: how big is the gap, in SDs?

$$
\text{Cohen's } d = \frac{\mu_1 - \mu_0}{\sigma} = \frac{113.4 - 100}{15} = 0.89
$$

.pull-right[
.box-inv-5.small[n isn't in the formula. It's about the size of the effect, not the sample]
]

.box-5.small[d is about the population, so we can only estimate it. Its estimate, Hedges' g, comes in Topic 8]

???

Cohen's d is a z-score for the effect: how far the true mean moved, in standard deviations. μ₀ is the value under H<sub>0</sub>, and μ₁ is the true value in the world where the program works.

---

# Two worlds: where sample means land

.box-inv-5.small[Critical values come from the H<sub>0</sub> world. Any mean past them gets "reject"]

???

Each curve is the sampling distribution of M, with n = 10. Both have σ<sub>M</sub> = 4.74. The critical values are 1.96 standard errors from 100: 90.7 and 109.3.

A test doesn't know which world it's in. It just checks where M landed.

---

# Each kind of outcome is an area

.box-inv-5.small[**Power** is the chance of rejecting H<sub>0</sub> when it's false. It's the opposite of a miss]

???

Red is alpha, in the H<sub>0</sub> world. Gold and blue split the H<sub>1</sub> world at the critical value: gold is a miss, blue is a catch. They add up to 1, so power is 1 − β.

There's a sliver of the H<sub>1</sub> curve below 90.7 too, which also counts as power. It's too thin to see.

Don't give the percentages yet: the next slide asks for them.

---

# Predict it: How often do we catch it?

.box-inv-8.small[
The program works: μ = 113.4, so d = 0.89. We test it with n = 10 and α = 0.05, two-tailed. In what share of samples do we reject H<sub>0</sub>?
]

.options[
.option[
**A** About 5%
]
.option[
**B** About 50%
]
.option[
**C** About 80%
]
.option[
**D** About 100%
]
]

???

About 6 minutes. Students can reason from the last picture: how much of the H<sub>1</sub> curve sits right of the critical value? Vote, turn to a neighbor, vote again, then show the answer.

Andy: create this poll in Poll Everywhere, then add the `.poll-bar` button from EDUC 265's 06-01 with its presenter id.

---

# Predict it: Answer

.options[
.option.wrong[
**A** About 5%
]
.option.wrong[
**B** About 50%
]
.option.correct[
**C** About 80%
]
.option.wrong[
**D** About 100%
]
]

.box-inv-8.small.sp-after-half[
Power is 80%. So β is 20%: one real effect in five gets missed
]

.box-8.small[
A is α: that's how often we reject when H<sub>0</sub> is true
]

---

# Predict it: About 80% get rejected

???

Now H<sub>0</sub> is false, and the population in panel 1 has mean 113.4. The dashed curve is still the H<sub>0</sub> world, and this pile is the H<sub>1</sub> world. Every red dot is a correct rejection, and every white dot is a Type 2 error. The readout gives the true power. Press +500 again to see the share settle.

The link uses d = 0.893 so that the mean shows as 113.4. At d = 0.89 it would show 113.3.

---

# Power

---

---

# Power: catching a real effect

???

This is the same picture as our two worlds, in SD units. If H<sub>1</sub> is true, with this alpha, 80% of samples correctly reject H<sub>0</sub>.

Power of 0.80 is the usual target, in the same way α = 0.05 is: a convention. Drag d to 0 and power disappears from the readout, because H<sub>0</sub> is then true.

---

# 1,000 samples fill in the table

<table class="deck-table error-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Our decision </th>
   <th style="text-align:center;"> H<sub>0</sub> is true (μ = 100) </th>
   <th style="text-align:center;"> H<sub>0</sub> is false (μ = 113.4) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Reject H<sub>0</sub> </td>
   <td style="text-align:center;"> <b>29</b> Type 1 errors </td>
   <td style="text-align:center;"> <b>412</b> correct </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Maintain H<sub>0</sub> </td>
   <td style="text-align:center;"> <b>471</b> correct </td>
   <td style="text-align:center;"> <b>88</b> Type 2 errors </td>
  </tr>
</tbody>
</table>

???

Each column is one world, and each adds up to 500. Rates are read down a column, never across. The deck computes this table, because the simulator has no live version of it.

---

# Four ways to raise power

.box-inv-7.small[**2. Use one tail:** a directional test]
]

.box-inv-7.small[**4. Find a larger effect:** a larger d]
]

.box-7.small[We start from d = 0.89, n = 10 and α = 5%, two-tailed, with power of 80%]

???

We'll change one thing at a time, and use the Power tab for each. Ask students to guess which moves the blue area the most.

---

# Way 1: Looser α lifts power to 88%

???

The critical value drops from 1.96 to 1.65, so more of H<sub>1</sub> is past it. The tool shows 1.64, because the exact value is 1.645. The cost shows in red: Type 1 errors double. Move the Alpha slider live from 0.05 to 0.10.

---

# A directional test puts all of α in one tail

.pull-left[
.box-inv-7.small[H<sub>0</sub>: μ ≤ 100 and H<sub>1</sub>: μ > 100. α stays at 5%, so the cutoff moves in]
]

.pull-right[
.box-inv-7.small[Choose the direction before seeing the data. An effect the other way can't be found]
]

???

The total risk of a Type 1 error stays at 5%. All of it now sits where we expect the effect, so the critical value falls from 1.96 to 1.65, as in the book's table. The simulator shows 1.64, because the exact value is 1.645.

Choosing a direction after looking at the data would double the real alpha.

---

# Way 2: One tail lifts power to 88%

???

Same power as way 1, and that's not a coincidence. Both put the upper cutoff at 1.65. The difference: here the total Type 1 risk stays at 5%. The cost is that an effect in the other direction can't be found at all.

---

# Way 3: Larger n lifts power to 93%

???

Remember Topic 6: a larger n makes the sampling distribution pointier. Both curves narrow and pull apart. Going from 10 to 15 raises power to 93%, with α still at 5%. The cost is real: more participants, more money.

---

# Way 4: Larger d lifts power to 92%

???

A bigger difference is easier to catch. The H<sub>1</sub> curve shifts right. We can't simply choose this one, but we can run a stronger version of a program, or use a test that's closer to what the program teaches.

---

# The four ways, side by side

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:left;"> Change </th>
   <th style="text-align:center;"> Power </th>
   <th style="text-align:left;"> What it costs </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Start: d = 0.89, n = 10, α = 5%, two-tailed </td>
   <td style="text-align:center;"> 80% </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 1. Loosen α to 10% </td>
   <td style="text-align:center;"> 88% </td>
   <td style="text-align:left;"> Twice the Type 1 errors </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2. One-tailed test </td>
   <td style="text-align:center;"> 88% </td>
   <td style="text-align:left;"> Blind to an effect in the other direction </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3. n = 15 </td>
   <td style="text-align:center;"> 93% </td>
   <td style="text-align:left;"> More students, more money </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4. d = 1.07 </td>
   <td style="text-align:center;"> 92% </td>
   <td style="text-align:left;"> Needs a stronger program or a better-matched test </td>
  </tr>
</tbody>
</table>

.box-7.small[Only ways 3 and 4 raise power without touching what we count as evidence]

???

Numbers are computed from the normal curve with pnorm and qnorm, and they match the Power tab.

---

# Predict it: Rescue a weak study

.box-inv-8.small.sp-after-half[A real but small effect: d = 0.20. With n = 10 and α = 5%, power is only 10%]

.box-8.small.sp-after-half[Which rescue helps more?]

.options[
.option[
**A** Loosen α to 10%
]
.option[
**B** Raise n from 10 to 40
]
.option[
**C** They help about the same
]
]

.box-inv-8.small[Then the Power tab decides]

???

About 6 minutes. Ask a neighbor, then hands. On the next slide, move one slider at a time.

---

# Predict it: Move the sliders

.box-8.small[α = 10% gives 17%. n = 40 gives 24%. Both are far from 80%]

???

The answer is B. Alpha can only shuffle the cutoff. Sample size pulls the curves apart. But even at n = 40, this study misses most of the time, which is the point of the next block.

---

# Planning a study, reading a result

---

---

# Two questions before any data

.box-inv-4.small.sp-after-half[The book finds the power of one test. Researchers usually fix power at 80% and turn the question around]

How small an effect can I detect with 80% power? This is the **minimal detectable effect**
]

How many people do I need for 80% power? This is the **required sample size**
]

???

Both answers rest on one fact, which is on the next slide. Funders ask these questions, and every grant proposal answers them.

---

# Where 2.80 comes from

.box-inv-4.small[For 80% power, μ must sit 1.96 + 0.84 = 2.80 standard errors from 100]

???

1.96 gets us from the H<sub>0</sub> mean to the critical value. Then 80% of the H<sub>1</sub> curve must lie past that value, and z = 0.84 is the 80th percentile of a normal curve. Add them up. This is why 2.80 shows up in power calculations everywhere.

---

# Two formulas, one idea

<div class="math-display">\[
n = \left(\frac{1.96 + 0.84}{d}\right)^2
\]</div>

.box-inv-4.small[That's why n = 10 had 80% power for d = 0.89. These use d in SD units, so σ = 1]

???

In points rather than SDs, multiply the left formula by σ: 2.80 × 15 / √10 = 13.3 points, so a true mean of about 113.3. Always round a sample size up.

These are for one mean against a known μ, as in this topic. Comparing two groups takes about four times as many people in total. We'll meet that case in Topic 9.

---

# In education, an effect of 0.5 SD is rare

<table class="deck-table tight">
 <thead>
  <tr>
   <th style="text-align:left;"> Which studies </th>
   <th style="text-align:center;"> Mean </th>
   <th style="text-align:center;"> Median </th>
   <th style="text-align:center;"> 90th percentile </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> All </td>
   <td style="text-align:center;"> 0.16 </td>
   <td style="text-align:center;"> 0.10 </td>
   <td style="text-align:center;"> 0.47 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Math tests </td>
   <td style="text-align:center;"> 0.11 </td>
   <td style="text-align:center;"> 0.07 </td>
   <td style="text-align:center;"> 0.37 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 100 students or fewer </td>
   <td style="text-align:center;"> 0.30 </td>
   <td style="text-align:center;"> 0.24 </td>
   <td style="text-align:center;"> 0.77 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> More than 2,000 students </td>
   <td style="text-align:center;"> 0.05 </td>
   <td style="text-align:center;"> 0.03 </td>
   <td style="text-align:center;"> 0.17 </td>
  </tr>
</tbody>
</table>

.box-inv-4.small.sp-after-half[Effects from randomized trials of education programs, in SD units. Bigger studies find smaller effects]

.box-4.small[Mindspark's endline gap, 2.69 points, is about 0.46 SD: very large here]

.source[Effects: Kraft (2020), *Educational Researcher*, 49(4). Mindspark: Muralidharan, Singh, and Ganimian (2019).]

???

Kraft collected 1,942 effects from 747 randomized trials, with standardized achievement tests. Cohen's labels would call most real education programs tiny. By Kraft's numbers, 0.10 is about typical, and it gets even smaller at scale.

Mindspark's number here is the plain endline gap divided by the pooled SD of raw scores. The published estimates use scaled scores and adjust for baseline, so they differ. And note it's an estimate, so in our notation it's a g.

Our running d of 0.89 would sit far above the 90th percentile of this table. It keeps the pictures readable, but it isn't realistic.

---

# Statistical significance isn't importance

<table class="deck-table">
 <thead>
  <tr>
   <th style="text-align:center;"> Sample size </th>
   <th style="text-align:center;"> Effect </th>
   <th style="text-align:center;"> z </th>
   <th style="text-align:center;"> p </th>
   <th style="text-align:center;"> Decision </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:center;"> 100 </td>
   <td style="text-align:center;"> d = 0.05 </td>
   <td style="text-align:center;"> 0.50 </td>
   <td style="text-align:center;"> 0.617 </td>
   <td style="text-align:center;"> Maintain H<sub>0</sub> </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 1,000 </td>
   <td style="text-align:center;"> d = 0.05 </td>
   <td style="text-align:center;"> 1.58 </td>
   <td style="text-align:center;"> 0.114 </td>
   <td style="text-align:center;"> Maintain H<sub>0</sub> </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 10,000 </td>
   <td style="text-align:center;"> d = 0.05 </td>
   <td style="text-align:center;"> 5.00 </td>
   <td style="text-align:center;"> &lt; 0.001 </td>
   <td style="text-align:center;"> Reject H<sub>0</sub> </td>
  </tr>
</tbody>
</table>

.box-inv-4.small.sp-after-half[Same tiny effect, opposite verdicts. With a large n, the sampling distribution gets so narrow that even small effects are significant]

.box-4.small[Significance says an effect is detectable. Its size says if it matters]

???

Each row assumes the sample shows exactly d = 0.05, so z = d × √n. The verdict changed. The effect didn't.

The opposite also happens: a large effect in a small study can be "not significant". So when you read a result, ask two questions. Is it significant? And how big is it, compared with what's realistic?

---

# Try it: Plan two studies

.box-inv-8.small.sp-after-half[Use power of 80% and α = 0.05, two-tailed. The multiplier is 2.80]

You can test 25 students. What's the smallest effect you can detect? Is it realistic, by Kraft's table?
]

You expect Kraft's median effect, d = 0.10. How many students do you need?
]

.box-inv-8.small[Work alone for two minutes. Compare with a neighbor. Then we vote]

???

About 6 minutes, with calculators.

---

# Try it: Answer

2.80 / √25 = **0.56** SD

That's above the 90th percentile of all studies in Kraft's table. This study would miss most real programs
]

(2.80 / 0.10)² = **784** students

Realistic effects need big samples
]

.box-inv-8.small[Small studies mostly find nothing, and when they do find something, it tends to be overstated]

???

Using the unrounded multiplier in Study B gives 785, so either is fine.

The last line is one reason the table's row for small studies has such a large mean: small studies only reach significance when their estimate lands far out, and journals favor those.

---

# What you can do now

.box-inv-8.small[Run the four steps: state H<sub>0</sub>, set criteria, observe a sample, reject or maintain]

.box-inv-8.small[Say what a p-value is, and what it isn't]

.box-inv-8.small[Tell a Type 1 error from a Type 2 error, and locate α, β and power on two curves]

.box-inv-8.small[Compute Cohen's d, and name four ways to raise power]

.box-inv-8.small[Find a minimal detectable effect and a required sample size]

.box-inv-8.small[Explain why statistical significance isn't importance]

---

# Next steps

Read sections 9-1, 9-2 and 9-4 on the t-test, for when σ is unknown
]

Power and the one-sample t-test. Bring a calculator and a laptop
]

.box-3.small[Mastery and Problem Set for Chapter 8 are due Sunday, February 7 at 6 PM. The midterm is Tuesday, February 16]

.picture-small img {
  display: block;
  margin: 0 auto;
  max-width: 100%;
  max-height: 230px;
}

.anteater img {
  display: block;
  margin: 0.15em auto 0;
  max-height: 105px;
}

/* This chunk sits at the end because any output before the first slide's
   "class:" line stops remark from reading that slide's properties */

/* Long titles get two smaller lines inside the 110 px title bar */
.remark-slide-content.long-title h1 {
  font-size: 1.05em;
  line-height: 1.05;
  margin-top: -100px;
  margin-bottom: 55px;
}

.source {
  position: absolute;
  bottom: 18px;
  left: 60px;
  font-size: 0.55em;
  color: #47505a;
}

.book-tag {
  margin-top: 0.5em;
  font-size: 0.65em;
  color: #47505a;
  text-align: right;
}

table.deck-table {
  width: 92%;
  margin: 0.4em auto;
  border-collapse: collapse;
  font-size: 0.72em;
}

table.deck-table.tight td,
table.deck-table.tight th {
  padding: 0.12em 0.6em;
}

table.deck-table th {
  border-bottom: 2px solid #244a70;
  padding: 0.3em 0.6em;
  text-align: left;
}

table.deck-table td {
  border-bottom: 1px solid #c9d1d9;
  padding: 0.3em 0.6em;
}

table.error-table td {
  padding: 0.6em;
}

/* Subscripts shouldn't push a title or a line of text out of place */
.remark-slide-content sub {
  line-height: 0;
}

.remark-slide-content h1 sub {
  font-size: 0.6em;
}

/* The summary of the four steps, one box per step */
.step-row {
  display: grid;
  grid-template-columns: repeat(4, 1fr);
  gap: 0.4em;
  margin-bottom: 0.4em;
}

.step-card {
  font-size: 0.62em;
  line-height: 1.25;
  padding: 0.4em 0.3em;
  margin: 0;
}

.options {
  display: grid;
  grid-template-columns: 1fr 1fr;
  gap: 0.35em 1em;
  margin-top: 0.6em;
  margin-bottom: 0.9em;
  font-size: 0.65em;
}

.option {
  background-color: #f3f3f3;
  border: 2px solid #cccccc;
  padding: 0.35em 0.6em;
}

.option.correct {
  background-color: #b18323;
  border-color: #b18323;
  color: white;
}

.option.wrong {
  opacity: 0.6;
}

.poll-respond {
  display: block;
  font-size: 0.5em;
  text-align: center;
  margin-top: 0.1em;
}

/* The printed PDF keeps the question and drops the answer slide. A slide that
   waits for a live button press would print empty, so it goes too */
@media print {
  .remark-slide-container:has(.poll-answer),
  .remark-slide-container:has(.press-slide) {
    display: none !important;
  }
}
</style>