Since rejoining Chalkbeat nearly a year ago, it’s been energizing to start a new section and newsletter focused on the big ideas and questions facing American education. Thank you for coming along for that ride. If you’ve found value in this work, I hope you’ll consider supporting it by becoming a Chalkbeat member. I know you’re inundated with these sorts of requests, so your support would genuinely mean a lot. ~Matt
If you build it, they may not use it.
This is the emerging problem with artificial intelligence tools designed to help students learn. The latest evidence is a new study finding that middle school students rarely used an AI-powered tutor when given the option to do so.
The study examined Khanmigo, an AI chatbot associated with the popular Khan Academy program. The findings confirm the sentiment of founder Sal Khan, who previously told me that the AI tutor was a “non-event” for most students.
“If a keen, engaged student wants to use the technology as a good tutor, it's ready to go,” said University of Toronto researcher Philip Oreopoulos. “The issue is that most students don't use it that way.”
In the study, Oreopoulos worked with a Tennessee school district to randomly assign low-performing students from 18 middle schools to use Khan Academy during the school day. Khan Academy provides online lessons and practice that students work through at their own speed. Khanmigo is programmed to respond to questions without giving away the answers, though students had to actively seek its help.
The study compared Khan students with another group of students who instead used a combination of other digital math programs and small-group teacher-led instruction.
By year two of the study, the researchers found some promising results. Students using Khan Academy made faster math gains than the comparison group. This suggests that Khan Academy has value — a notable finding at a moment of intense backlash to ed-tech.
But the researchers say the benefits of Khan didn’t seem to come from AI. Most of the Khan Academy middle schoolers simply weren’t engaging with Khanmigo productively or at all.
Students only used Khanmigo about a third of the days they were working in Khan Academy. When interactions did occur, many students just sent off-topic messages or tried to convince Khanmigo to provide the correct answers. “Access was nearly universal but engagement was thin,” the researchers wrote. “It may be that human attention is the key ingredient for realizing the full benefits of personalized learning.”
Here are some examples of students’ interactions with Khanmigo from the new study:
Paraphrased examples of students' conversations with Khanmigo from the study "One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment"
“What this has put in huge spotlight — not that it’s a shock to educators — is how much motivation matters,” said Sal Khan in an interview.
The more AI can mirror a human tutor, the better it might become, Oreopoulos says. Right now, it’s not there. Maybe that’s a good thing: It’s not clear that we would want children to experience an AI chatbot as too human-like.
Since the study period, Khan Academy has redesigned its interface to better integrate Khanmigo and encourage more students to use it. “The AI could not just sit next to the content. It had to be woven into it. We had to make productive struggle harder to sidestep,” Khan wrote recently of the changes. The goal, he says, is not AI for its own sake, but to hopefully drive larger learning gains.
Overall, research on AI in education is nascent, but an early theme is that students often won’t use it for learning. “When given AI tutors, many students do not engage,” concluded a recent summary.
In another study released last week, Oreopoulos and colleagues sought to test a solution to this problem. In the same Tennessee school district, the researchers designed their own online math program. Some students were given access to an AI chatbot to help them work through problems. Students could only move on if they answered the questions correctly. In contrast to Khanmigo, the AI chatbot popped up without student prompting.
The students with access to AI moved less quickly through problems, but were more accurate in their answers. A week later when they were quizzed again on similar problems without access to AI, they were slightly more likely to respond correctly compared with a control group. This is preliminary evidence that in tightly controlled settings students can be coaxed into using AI well.
Of course, the challenge is that students spend most of their time outside of tightly controlled settings, and adolescents by their nature often seek the path of least resistance.
“The same technology that can be used as a good tutor, can also be used to make your life easier. But making your life easier equals less effort, less learning,” said Oreopoulos.
Reach me at [email protected]. Forwarded this email? Sign up here to receive the Chalkbeat Ideas newsletter.
Looking for your next read? Check out these other great newsletters.
