Guilty Until Proved Human

AI detection tools do not work. They discriminate against non-native English speakers. And universities are still paying millions for them.

A student submits her essay. An algorithm flags it. She is now guilty until proved human.

This is not hypothetical. This is happening in universities right now. Students are being accused of cheating by software that does not work, punished for writing in their own voice, and forced to prove their innocence against a tool that cannot tell the difference between a Nigerian English speaker and a chatbot.

In this post we will:

• Look at the evidence that AI detection tools are technically unreliable.

• Show who is being harmed, with named cases.

• Follow the money behind the detection industry.

• Ask what the alternative looks like.

---

The evidence

I keep saying AI detection software does not work. This is not provocation. It is peer-reviewed fact.

Weber-Wulff et al. (2023) tested fourteen AI detection tools. Not one scored above 80% accuracy. The best-performing detectors are wrong at least one in five times. Most are worse.

For non-native English speakers, the numbers are catastrophic. Liang et al. (2023), published by Stanford researchers in Patterns (Cell Press), found that seven widely used GPT detectors flagged 61.3% of genuine human-written essays by non-native speakers as AI-generated. One detector flagged 97.8% of TOEFL (Test of English as a Foreign Language) essays. All seven unanimously misidentified nearly one in five.

These are not edge cases. These are the tools universities are paying for.

OpenAI built its own detector. It correctly identified just 26% of AI-written text while falsely flagging 9% of human writing. OpenAI shut it down in July 2023, citing “low rate of accuracy”. The company that built the thing that universities are trying to detect could not build a reliable detector. That should have been the end of the conversation.

It was not.

---

Who is being harmed

Brittany Carr was a student at Liberty University. Her assignments were flagged by AI detectors. She received failing grades. She had written about her personal cancer diagnosis. She provided her handwritten revision history. The university required her to take a ‘writing with integrity’ class and sign an apology. She left.

Thierry Rignol, a student at Yale School of Management, was suspended for a year after GPTZero flagged his exam. He is suing the university, alleging discrimination as a non-native English speaker and denial of due process. A judge denied his preliminary injunction.

Ailsa Ostovitz, seventeen, had her grade docked at Eleanor Roosevelt High School in Maryland after one of her assignments was flagged at 30.76% probability. After her mother met with the teacher, the teacher acknowledged they no longer believed she used AI. The grade had already been docked.

Zi Shi, a Mandarin-speaking student in Ohio, had his English assignment flagged by GPTZero. He had used Grammarly to check his grammar. Not to generate text. To clean it up.

Madeleine, a nursing student at Australian Catholic University, waited six months for charges to be dropped. During the investigation her transcript was marked ‘results withheld’. She could not secure a graduate nursing position.

Many students are now deliberately introducing misspellings and improper grammar to avoid being flagged. They have learned that writing well is a liability.

Read that again. Students are learning to write worse to survive a system that claims to protect academic integrity.

---

The discrimination problem

There is a reason these tools fail for non-native speakers. AI detectors measure text perplexity: how predictable the word sequences are. Non-native English writing tends to use simpler, more formulaic sentence structures. The detectors interpret low perplexity as a hallmark of machine generation.

The result: writing that reflects a second language education, writing that reflects years of effort to learn English, gets flagged as artificial. The student’s hard-won fluency becomes evidence against them.

Then there is the word ‘delve’. It has become shorthand for AI slop. Educators see it and assume the text was generated. But the reason ‘delve’ appears so frequently in ChatGPT’s output is because of the language of the people who trained it. OpenAI hired underpaid workers in Nigeria, where ‘delve’ is common in business English, to provide early training data. Meanwhile, in Kenya, workers were paid less than $2 an hour to label traumatic content for safety filters.

The language came from one group in the Global South. The labour came from another. The profit went elsewhere entirely. And now detection tools that flag the word ‘delve’ penalise the linguistic patterns of the same marginalised communities whose work built the models in the first place.

A YouGov/Studiosity survey of 2,373 students, published last month in Inside Higher Ed, found that 75% of AI-using students reported significant stress over being wrongly flagged. International students were twice as likely to say they experienced “a lot” of stress.

This is not a technical problem. It is a structural one.

---

Follow the money

Turnitin was acquired for $1.75 billion in 2019 by Advance Publications, the company that owns Condé Nast. It serves 17,000 institutions and 71 million students. Revenue reached approximately $203 million in 2024. The California State University system alone has paid over $6 million across seven years for Turnitin.

Turnitin’s own disclaimer states the tool ‘should not be used as the sole basis for adverse actions against a student’. The company knows. The institutions keep buying.

GPTZero, built by Princeton undergraduates, raised $13.5 million and reached profitability within eighteen months. Annual recurring revenue: $16 million by 2025.

There is money in suspicion. And the product does not need to work for the money to flow.

Meanwhile, the arms race has started. NBC News reported that 43 ‘AI humaniser’ tools accumulated 33.9 million website visits in a single month. Students who never used AI for their assignments are now running their work through humaniser tools to avoid false positives. The detection tools are pushing students toward the very technology they claim to discourage.

---

The alternative

A growing number of institutions have worked this out. The University of Waterloo discontinued Turnitin’s AI detection in September 2025 after internal testing found it flagging human-written text as “100% generated by AI”. Curtin University disabled it in January 2026. MIT published guidance titled ‘AI Detectors Don’t Work. Here’s What to Do Instead.’ Yale, Johns Hopkins, Northwestern, Boston University, UC Berkeley, Georgetown, NYU, and dozens more have followed.

Yet over 40% of US teachers still use detection tools.

The question was never ‘did a machine write this?’ The question was always ‘what did the student learn?’ Detection answers neither. It produces a probability score and hands the consequences to a human who was not trained to interpret it.

The alternative is not complicated. It is assessment that makes detection unnecessary. Tasks that require personal experience, contextual judgement, and evidence of process. Portfolios. Oral examinations. Work that shows the thinking, not just the product. If AI can pass your assessment, your assessment is not testing a human.

That is why I built the Integrity Debt Audit, a free tool to help educators identify if their assessments can be easily automated by AI. Taking suspicion away from the student and using it to make more resilient and authentic assessments in the first place.

This is not about being soft on cheating. It is about being honest about what detection actually does: it weaponises a faulty algorithm against the students with the least institutional power to fight back.

Detection starts from suspicion. Education starts from curiosity. You cannot build both in the same system.

If you are a student who has been falsely accused, I want to hear from you. If you are an educator who has stopped using detection tools, I want to know what you did instead.

I read and respond to all your comments.

Go slow.

---

The Slow AI Curriculum is a 12-month programme for critical AI literacy. 171 members across education, policy, healthcare, and tech. Each month we take one theme, go deeper with theory, critical prompts, and a live seminar where we actually talk to each other.

This month we explored synthetic empathy: what happens when AI simulates care, who benefits, and what gets lost. Next month is security and surveillance.

The conversations in these seminars are some of the sharpest I have been part of in fifteen years of academia. If you want to build the judgement that no detection tool can measure then