Welcome to Chalkbeat Ideas, a new section featuring reported columns on the big ideas and debates shaping American schools. We’re fresh off our first event, which this column is based on. Thanks to those of you who joined us. Missed it? You can watch it here.
In the fall of 2020, many school systems remained closed for in-person instruction citing concerns over the still-raging COVID-19 pandemic. Some did not fully reopen until the following school year.
It was an unprecedented decision that sent political, societal, and educational shockwaves that are still rippling. This was intensely debated then and continues to be a source of scrutiny now. Studies have found that schools that stayed closed longer experienced greater drops in student test scores.
Is that still true? In short, yes, though the lasting effect is smaller than might be suspected considering the large role school closures continue to play in the American psyche and in our politics.
As of last year, school systems that were fully remote in the 2020-21 academic year remained 1 to 2 percentage points behind in reading and math than places that quickly reopened. That’s according to a new analysis of data by Brown University professor Emily Oster, presented today at a Chalkbeat Ideas webinar.
“There was the ability to recover some over the next set of years,” said Oster, who was a vocal critic of pandemic school closures. “I still think we should have opened schools.”
Let’s unpack this and other takeaways from the large tranche of state testing data compiled by Oster and colleague Clare Halloran as part of their work for the Education Data Center.
Students are still behind, but there has been some recovery
This data — on the share of third through eighth graders proficient on state exams — confirms what we already know: Most places have not fully recovered from pandemic-era learning declines. Among 27 states with results, less than half have seen math scores return to 2019 levels. In reading, the share with full recovery is even smaller.
(Why only a subset of states? Many have changed their tests, making it impossible to compare results between now and the pandemic. A few other states haven’t released scores for this year yet.)
These results do offer some encouraging news. Every single state has seen at least some gains in math. Many are close to their pre-pandemic levels. In reading, results are more equivocal. Some places, including Minnesota, Massachusetts, and Missouri have seen scores continue to fall. But more states than not have seen some progress.
A caveat to remember: These are comparisons of different groups of students over time. The third graders tested this year were not even in grade school at the height of the pandemic.
State tests and NAEP tell different stories
The federally administered NAEP exams tend to be more pessimistic than states’ scores. NAEP showed some recovery between 2022 and 2024 in fourth grade math, but not in eighth grade. Reading scores dropped even further in both grades.
Oster and Halloran show that NAEP and states’ own scores are moderately correlated with each other. But a number of states showed strong recovery on their own tests, whereas their NAEP scores were flat or even down.
In Illinois, for instance, the share of eighth graders reading proficiently jumped by 20 points on the state exam between 2022 and 2024. During the same time, NAEP eighth grade reading scores in the state were flat.
Lindsay Record, a spokesperson for the Illinois State Board of Education, said that the state’s ELA exam has significantly more content in writing than the NAEP reading test. “These are two different assessments with different purposes,” she wrote in an email. On NAEP, Illinois eighth graders outperform the national average in both reading and math, she noted.
Which results should we trust, NAEP or state tests? Each has advantages, according to Andrew Ho, a testing expert professor at Harvard. “This is apples and oranges, and they’re both fruit, and we like fruit,” he said.
State assessments are more tightly aligned with state standards, which is what students are supposed to be learning. These exams are taken by virtually every student. NAEP only tests a representative subsample, which means scores can rise or fall due to random chance.
Yet NAEP is low stakes for schools, so there is less incentive to try to inflate scores through excessive teaching to the test.
The demographic group hit hardest by learning loss: girls
The new state test data shows that girls, on average, fell particularly far behind during the pandemic and have been slower to recover. As of last year, girls’ average proficiency score is 6 points behind pre-pandemic benchmarks, while boys are just 3 points back. In contrast, differences by race have not been as large.
This striking, and too many surprising, trend has emerged across a number of exams, as well as in college achievement.
No one knows for sure why this happened. Some researchers have suggested that girls benefit more from schooling, so taking it away, as happened during the pandemic, hurt them more. It’s not clear why girls’ recovery has been slower, though.
Learning loss persists even where schools closures were brief
Oster and Halloran analyzed the effect of school closures by comparing districts within the same state that were otherwise similar but made different decisions about when to reopen. At first the learning gap was substantial, but since 2021 it has shrunk. Oster described the difference now as “largely insignificant.”
In one sense, the fact that school closures from several years ago still seem to be lingering at all is remarkable. Plus, this probably understates the closure effect. It doesn’t count students who have aged out of the testing window. And some of the harmful effects may fade away in test scores but persist in other difficult-to-measure ways.
Also keep in mind: All schools were closed in the spring of 2020, so this is a study of variation in school closure decisions, not the total effect of closures.
Still, what is striking is that even school systems that very quickly reopened continue to be behind academically. That suggests most of the persistent learning loss is not driven by the still-contentious closure decisions from the fall of 2020.
Reach me at [email protected].
Thumbnail photo by Erica Seryhm Lee for Chalkbeat
How was today’s newsletter? Take 2 minutes to let us know.
Looking for your next read? Check out these other great newsletters.
