Before you say yes to an AI tutoring pilot: six questions for school leaders

On September 14, the Gates Foundation committed $1 billion to what it calls equitable AI. Forty percent of it, about $400 million over two years, goes to education: AI tutoring to individualize learning, and teaching tools for classrooms in the United States and abroad.

Money at that scale turns into pilot invitations. Some will reach your district. This post is about what to ask before you say yes.

What we know about the pilots

Fortune reported on September 19 that the tools will be tested in schools serving high shares of low-income students, paired with teacher training, and evaluated by independent researchers before they are scaled. A foundation spokesperson told the magazine that outside researchers, including a Johns Hopkins study of one tool, will evaluate the pilots.

The most useful detail for school leaders is the success measure. According to Fortune, success will be judged by whether students who have fallen behind actually catch up, not by raising scores overall.

What the skeptics said

The same coverage carried real doubts. Fortune’s headline says teachers warn the money could “end up widening” the divide it is meant to close. In the article, that wording is Dora Palfi’s, an education-technology executive. Amy McBride, a teacher, told Fortune that the students who need the most help may also need the most adult guidance in learning how to use AI.

Two days earlier, The Chronicle of Philanthropy gathered experts on the pledge. One of them, Sarah Reckhow, pointed to the foundation’s earlier K-12 efforts, including its small-schools program and its teacher-effectiveness work, which did not work out as hoped. At the time, one expert noted that the foundation had not spelled out who would evaluate the tools; the Fortune report later named outside researchers. The piece described a pilot tool used in 21 middle schools, with one teacher’s classroom as the example, which is promising but not yet a study.

Both things can be true. The design, with teacher training and outside evaluation, is sensible. The evidence that AI tutoring works at scale, for the students it targets, is still ahead.

Six questions before you sign

1. Who owns the grade? The tutor can explain, prompt and practice. A teacher should still decide what a student has learned. Put that in writing.
2. Which students, and who decides? If the pilot targets students who are behind, how are they chosen, and what do the other students in the room do?
3. Where does the time come from? Training is part of the design. Ask how many hours teachers get, and whether it happens inside the working day.
4. What data leaves the building? List every field the tool collects about a student, where it is stored, and who can see it. California’s September package regulates the use of K-12 pupil data in AI systems. Review the contract with counsel. This is general education, not legal advice.
5. What is the comparison? Catching up compared with what: last year’s cohort, a similar class without the tool, or the same students before the pilot? Agree on this before day one.
6. What happens when it ends? If the pilot funding stops after two years, who pays, and what happens to the data?

A worked example

Here is a hypothetical plan for one middle school, to show the shape. The numbers are illustrations, not results.

• Question: Do seventh graders who scored below grade level in math last spring close part of that gap by March?
• Baseline: Spring scores and a September diagnostic, recorded before the tool is switched on.
• Comparison: Two seventh-grade math sections use the tutor three times a week; two similar sections continue as before. Assignment is decided before anyone sees the diagnostic results.
• Teacher role: Teachers review the tutor’s weekly summary and decide which students get small-group time.
• Measure: The share of below-grade students who move up at least one band on the March diagnostic, in each group.
• Stop rule: If a teacher reports that the tool is giving students wrong explanations, pause and review.

A plan like this is small enough to run and clear enough that an outside evaluator, a board or a funder can read it.

What to do this week

• Write down who in your district would receive a pilot invitation, and who would decide.
• Draft your answer to question 4 now. It takes longest.
• Ask your math and literacy leads which students they would most want a tutor to reach, and why.
• Set a baseline date before any pilot starts, even if none is confirmed.

New money is a chance to learn something real. The schools that learn the most will be the ones that decided what to measure before the tool arrived.