How will we know if new school choice programs are helping students learn?

Welcome to Chalkbeat Ideas, a new section featuring reported columns on the big ideas and debates shaping American schools. Forwarded this newsletter? Sign up here.

In 2023, the research firm RAND received a large grant to study education savings accounts, which provide parents with stipends to support various forms of private education.

Susha Roy, a researcher on the project, was eager to figure out whether these rapidly growing programs were helping students learn. Roy and colleagues began looking for places to study. Perhaps Arizona, they thought, which launched the country’s first ESA with universal eligibility in 2022. This quickly proved to be a dead end, though, because the state doesn’t require ESA students to take any sort of standardized test.

They also looked at Florida. The state does mandate an assessment for ESA students, but it doesn’t have to be the same test that public school students take. And anyway RAND didn’t have any luck getting the state to share the private school testing data, Roy says. (A spokesperson for the state Department of Education didn’t respond to a request for comment)

In the end, the RAND researchers did not publish a study on ESA achievement outcomes, as they originally intended. Instead they wrote a report describing how difficult it was to do such a study.

“Many more states are implementing these programs at a pretty fast clip,” says Roy. “The need for evaluation of how they’re affecting students is more important than ever.” 

The RAND researchers’ experience brings into sharp relief the challenge faced by both parents and policymakers. So far little good evidence has emerged about this new generation of choice programs. Parents also have limited objective data for judging individual private schools in an increasingly crowded education marketplace.

Part of this speaks to the inherent challenges of evaluating educational programs, especially new ones. But the obstacles also reflect policy choices and a philosophical shift among choice advocates. Many have increasingly emphasized choice as a form of parental empowerment — and a way to escape public schools — rather than a mechanism for boosting academic achievement. This philosophical shift could affect public schools too.

The number of students getting a public subsidy for private education has more than doubled in just a few years, reaching 1.5 million, according to a recent estimate. That figure is still just a fraction of the nearly 50 million students in public schools, but it will surely continue to grow. Both Texas and the federal government will soon launch new stipends that can pay for private school tuition.

Public schools and their programs are most commonly judged by state math and reading tests, which are required by federal law in grades 3-8. Private schools are typically not required to give state tests. They also don’t have to take NAEP, the federal monitoring exam.

The new federal choice law does not include testing requirements and the Trump administration has indicated it will try to limit states from imposing their own rules.

Most new state-created ESA and voucher programs do require private schools to administer a nationally benchmarked test, typically of the schools’ choosing. Because the exams are different, comparisons to public schools are difficult, even assuming researchers can get this data. (A few states, including Indiana and Iowa, do use the same test as public schools.)

“We won't be able to evaluate this program in anything like the way that we have evaluated education policies in the past,” says Doug Harris, a Tulane professor who has researched and criticized school voucher-style programs. An analysis by FutureEd noted that it is rare for state laws to include funding for evaluations of academic outcomes. 

From a research perspective, we’re not likely to be totally in the dark, though. Researchers are more optimistic that they’ll be able to study the longer-run effects beyond test scores, like college-going. They can also examine the effect of competition on public schools.

Patrick Wolf, a professor at the University of Arkansas and a supporter of choice programs, acknowledges that there will be challenges in evaluation, but is hopeful to produce some meaningful results. He says that even if schools take different tests there may be ways to roughly equate them. “I’m a ‘don’t make the perfect the enemy of the good’ guy,” he says. 

Still, a lack of comparable testing makes it difficult for parents to evaluate schools across sectors. Third-party websites, designed to help parents choose among schools, often don’t rate private schools. Most states “lack a framework to help parents decide what is working for their students — there’s no clear benchmark against which to measure student success,” notes the FutureEd report.

To be clear, I don’t want to overly romanticize how test scores are used in public schools. Many education programs and policies are never evaluated. Researchers often can’t get the data they need, sometimes because public officials don’t want the scrutiny. Although test scores do seem to impart some useful information, ratings for individual schools are incomplete at best and can be deeply flawed and biased at worst.

Private school officials and their advocates say that they shouldn’t have to use public school testing because they don’t want to follow the same standards as public schools. To many, that’s part of the point of private education. 

A recent study found that even though students who participated in Ohio’s voucher program experienced declines in test scores, their chances of attending and completing college increased. This bolsters school choice advocates who contend that parents can see what test scores miss. “Accountability is parental choice. If the school doesn’t serve, parents leave,” says a website from ACE Scholarships, which provides stipends for private school tuition.

Such an argument has far-reaching implications in this new era. When educational options are plentiful, as is the case in many places, public schools also become schools of choice — after all, unhappy families can look elsewhere. If parental satisfaction suffices to judge a school, then it’s hard to see why this logic would not apply to public schools too. Many public school leaders also say test scores are poor measures of quality.

For better or for worse, over the last few decades testing has been policymakers’ main way of judging individual schools, major school reforms, and even the country’s education system as a whole. As more students receive public funding for their education without comparable testing, this arrangement could be increasingly challenged.

Reach me at [email protected].