Two questions come up whenever a school considers an AI tool: what happens to student data, and who is responsible for it? The honest answer to the second is simple. Schools remain responsible for how student data is shared, whichever vendor holds it. The risk is not new. A Chalkbeat report quotes a privacy counsel saying that many of the risks with AI are "similar to the ones other ed-tech tools already presented, but on a much larger scale." What has changed is how easy it is for a teacher to sign up for a tool in a minute and paste in something they should not.
This is a practical guide, not legal advice. Privacy law varies by country and state, so check with your district's legal or privacy lead. In short, know what counts as student data, give teachers a simple rule for what never to paste into a tool, ask vendors for written answers to a short list of questions, train staff, and have a plan for when something goes wrong.
What counts as student data?
More than most people assume. The same Chalkbeat report lists names and family information, attendance and behaviour records, disabilities and disciplinary history as examples covered by laws that protect student data. In practice, treat as sensitive anything that identifies a student or reveals something about them: names, photos and contact details, grades, test scores and attendance, behaviour notes, discipline records, IEP, 504 and other support information, and parents' names, addresses and identification numbers. Student writing counts too, if it can be linked to a person.
De-identified data is safer, but be careful. A detailed description of a student in a small class can identify them even without a name, so removing the name is a start and not a guarantee.
What do FERPA and COPPA say, in one paragraph?
Two laws come up most in the United States. FERPA protects education records, and lets schools share them with vendors under certain conditions, including that the vendor is under the school's direct control with respect to the use and maintenance of those records. COPPA covers online services that collect personal information from children under 13, and generally requires parental consent, with some flexibility for schools consenting on parents' behalf for educational purposes, and not for commercial ones. Other countries and many US states have their own rules, and some are considering new AI-specific bills. Ask your district which apply.
What is a simple rule for teachers?
If you would not put it on a poster in the corridor, do not paste it into an AI tool. That single test catches most problems, and a few habits back it up. Remove names and identifiers before pasting anything, including from student work, and describe instead of identifying, for example "a grade 7 student who struggles with organisation" instead of a name and a diagnosis. Use school-approved accounts and not a personal email. Even education-focused platforms often let teachers sign up without approval, and a general chatbot may not feel like "outside technology" to a busy teacher.
Do not paste IEPs, behaviour reports or health information into a general tool. Read the pop-ups too. Some education tools warn you not to enter personal data, and some try to detect and delete it, but they still rely on you being careful. Finally, assume that what you enter may be kept unless you know otherwise. In many cases, the information users provide can be incorporated into a model, according to a privacy counsel quoted by Chalkbeat, which means someone else might later retrieve it.
What should you ask a vendor?
Before a tool is used with students, get written answers to a short list of questions, and read the answers against the contract, not the marketing page. The questions fall into five groups.
The first is data collected. Ask what personal data the tool collects and why, whether any of it is not needed for the tool to work, and whether students can use it without giving personal details. The second is use of data. Ask whether student data is used to train models, and ask for the contract wording and not a slogan. Ask whether it is used for advertising or sold, and which third parties or subprocessors receive it.
The third group is storage and control: where the data is stored and for how long, whether the school can delete it on request and how quickly, what happens to the data if the vendor is sold or shuts down, and whether teachers and administrators can see what students entered. The fourth is security and incidents, meaning what security standards the vendor meets and can show evidence for, and how and how fast you will be told about a breach. The fifth is contract and compliance. Ask whether the agreement says the vendor acts under the school's control for education records, how the vendor handles COPPA and any state or national rules that apply, and what age the product allows and how that is enforced.
If a vendor cannot answer these plainly, that is an answer. Any "no" on the use of data for training or advertising should be a stop until it is resolved.
What can the news teach us?
Chalkbeat reported that Los Angeles Unified rolled out an AI assistant for students called Ed, which was quickly discontinued after the company that built it ran into financial trouble. The abrupt shutdown left parents and advocates without answers about what had been done with the student data the platform held. The takeaway is the "what happens if the vendor disappears" question. Ask for the answer in writing, and ask for an export and deletion process before you sign.
Why is training part of the answer?
Rules do little if teachers do not know them. An Education Week survey of 1,135 educators, reported by Chalkbeat in December 2024, found 58% had received no training on AI. The American Federation of Teachers has argued that school and district technology departments should lead in vetting tools, not leave it to individual teachers.
One district described in the same report tells staff plainly to omit personally identifiable information, and runs regular sessions on the risks. For example, principals were shown how to use an AI tool to help write behaviour reports by describing the facts without real names, and then checking the output before moving it into the district template. Useful staff training covers what counts as student data, with examples, which tools are approved and how to get a new one reviewed, how to anonymise information before using any tool, and what to do if you paste something by mistake. The answer to that last one is to tell your privacy lead straight away.
What should you do if something goes wrong?
Have a simple plan before you need it. Stop using the tool for that task and report to the privacy lead or administrator the same day. Record what was shared, when and with which tool, and ask the vendor to delete the data and to confirm that it has. Tell affected families as your policy and the law require, and then learn from it by updating the training or the approved list. Speed matters more than perfection here, so people should feel safe reporting a mistake quickly.
Where does this fit with other decisions?
Privacy is one of six areas in our procurement scorecard for AI tutors, and a section of any school AI policy. Run the questions above before a pilot, not after it.
The short version
Schools remain responsible for how student data is shared. Teachers should anonymise everything and use approved accounts. Ask vendors for written answers on training, retention, deletion and shutdown, train staff, and have an incident plan ready.
Frequently asked questions
Is it safe to paste student work into ChatGPT if I remove the name? Removing the name reduces the risk but does not remove it, because writing can be identifying and the tool may keep it. Use approved tools where possible, and check your school's rules.
Do free tools need the same checks? Yes. Free tools still handle student data, and you often pay in data instead of money.
Who should approve new tools? Your school or district technology and privacy leads. Teachers should know how to request a review, and reviews should be quick enough that people do not work around them.
What about students using AI at home? You cannot control home use, but you can teach students not to enter personal information, and tell families which tools your school has approved.