Before you start
An AI detector is a program that scores how likely a text is to be AI-written. There are two kinds. A classifier learns patterns from examples of human and AI text, then guesses which one a new text resembles. A watermark checker looks for a hidden statistical pattern that a model adds while it writes.
Two terms matter throughout. A false positive is human writing that the detector labels as AI. A true positive is AI writing that it labels correctly. Catching more AI text by flagging more human text is a trade and not an improvement.
This lesson reads OpenAI's own notes on two detectors, three published studies and a post from Google DeepMind and Hugging Face. It does not teach how to beat a detector. The aim of this course is text that is specific and sounds like you, and the signs in the previous lesson are about readers and not scores.
OpenAI built a classifier, then withdrew it
In February 2023 OpenAI announced a free classifier trained to tell AI-written from human-written text. OpenAI's own announcement page could not be opened for this lesson, so the figures here are quoted as a reader posted them in OpenAI's developer forum on 26 July 2023, copying OpenAI's wording.
OpenAI called the classifier "not fully reliable". On a challenge set of English texts it labelled 26% of AI-written text as likely AI-written, and it wrongly labelled 9% of human-written text as AI-written. It added that reliability typically improves as the input gets longer.
Within six months it was gone. The notice, as the reader quoted it, says that as of 20 July 2023 the classifier was no longer available due to its low rate of accuracy. It adds that OpenAI was researching more effective provenance techniques for text, which means ways to show where a text came from.
26%
of AI-written English text labelled "likely AI-written"
Source: OpenAI developer forum: New AI classifier for indicating AI-written text (read November 6, 2026)
9%
of human-written text wrongly labelled as AI-written
Source: OpenAI developer forum: New AI classifier for indicating AI-written text (read November 6, 2026)
Launch announced
A post by Logan Kilpatrick says "We're launching a classifier trained to distinguish between AI-written and human-written text."
No longer available
The notice quoted in the forum says the classifier is withdrawn "due to its low rate of accuracy".
Source: OpenAI developer forum: New AI classifier for indicating AI-written text (read November 6, 2026)
OpenAI's earlier detector said the same thing
OpenAI had published a detector before, for text from GPT-2, an older model. It sits on Hugging Face as roberta-base-openai-detector. Its model card quotes OpenAI's research as reaching detection rates of about 95% for text from the 1.5 billion parameter GPT-2, and says that classifying text from larger models is more difficult.
For newer tools the card is blunt. It strongly suggests not using the model as a ChatGPT detector for making grave allegations of academic misconduct against undergraduates and others, because it might give inaccurate results on ChatGPT text.
We believe this is not high enough accuracy for standalone detection and needs to be paired with metadata-based approaches, human judgment, and public education to be more effective.
Who gets flagged: non-native English writers
A 2023 Stanford study, GPT detectors are biased against non-native English writers, ran seven detectors over 91 TOEFL essays, which are essays written by non-native English speakers, and 88 essays by US eighth graders. The detectors consistently flagged the non-native essays as AI-generated while accurately identifying the native ones.
On the TOEFL essays the average false positive rate was 61.22%. All seven detectors flagged 18 of the 91 essays, and at least one flagged 89 of them. When the researchers asked ChatGPT to enhance the word choice of the same essays, the average rate fell to 11.77%, and only 1 essay was flagged by all seven.
The authors read that as a sign that detectors may penalise writers with limited linguistic range, since simple prompting both reduced the bias and bypassed the detectors. They caution against using these detectors in evaluative or educational settings.
Source: Hugging Face Papers: GPT detectors are biased against non-native English writers (read November 6, 2026)
A benchmark of twelve detectors
RAID, a benchmark published in 2024, set out to test detectors on more than the easy case. The authors write that many commercial and open-source detectors claim accuracy of 99% or more, while few are checked on shared benchmarks. RAID's page reports what it found.
| What RAID varied or measured | What it found |
|---|---|
| Scale | Over 6 million generations from 11 models across 8 domains, with 11 adversarial attacks and 4 decoding strategies, scored against 12 detectors: 8 open-source and 4 closed-source. |
| Open-source detectors at default thresholds | Their false positive rates are "dangerously high", say the authors. |
| Closed-source detectors | None above 1.7% false positives at the thresholds tested. The authors recommend calibrating a detector on in-domain data before use. |
| A repetition penalty | It reduced accuracy for all detectors, by up to 32 points, whatever the decoding strategy. |
| A model the detector was not trained on | A RoBERTa detector trained on GPT-2 text scored 95% or more on GPT-2 output in five domains, and rarely beat 60% on other models' text in the same domains. |
Watermarks are a different idea with their own limits
A watermark is added while the model writes, not guessed afterwards. In one method, the previous word seeds a random split of the vocabulary into a green list and a red list, and the model is nudged toward the green words. A checker then counts how many green words a text holds. Natural text lands near the chance level, and watermarked text lands well above it.
Both methods below depend on the model's maker. A watermark exists only if the tool added one, and the checker has to match that tool's setup, so a watermark check says nothing about text from a tool that added none.
| Method | What the source reports | Limits it states |
|---|---|---|
| A Watermark for Large Language Models, University of Maryland, 2023 | A z score above 4 gives a false positive rate of about 3 in 100,000. In the paper's own test, 98.4% of about 500 generations were detected at that threshold, about 128 tokens, with multinomial sampling. The model was OPT-1.3B, each text was 200 tokens from news-like prompts, and the watermark was the soft one with two set parameters. The abstract says detection can work from as few as 25 tokens. | Removing the watermark from a long text takes changing roughly a quarter of the tokens or more, in the paper's analysis of its hard rule. That analysis assumes the attacker knows the watermark and picks each changed token to do the most damage. |
| SynthID Text, Google DeepMind and Hugging Face, 23 October 2024 | Survives cropping, modifying a few words and mild paraphrasing. Every watermark setup needs its own trained detector, with at least 10,000 examples recommended. | Less effective on factual responses. Detection confidence can drop when text is thoroughly rewritten or translated. Not built to directly stop motivated adversaries. |
A detector vendor on false positives
Turnitin's YouTube channel has a video titled "Understanding false positives within Turnitin's AI writing detection capabilities". The title shows that false positives are a topic detector vendors address themselves. This lesson does not describe what the video says, and it takes no figure from it.
What to ask for instead of a score
Wikipedia's Signs of AI writing tells its editors not to rely solely on detection tools such as GPTZero and Pangram. It says the tools perform better than random chance and still have non-trivial error rates, and that paraphrasing, markup and spacing changes, and models unseen in training, can throw them off.
- Ask for the working: notes, outlines and the draft history.
- Ask the writer to explain a choice, such as why a source is cited or why a sentence says what it says. The guide lists the ability to explain one's own editorial choices among its signs of human writing.
- Check the names, figures and links in the text. For a link that looks invented, the guide suggests asking how the mix-up happened before jumping to conclusions.
- If you still run a detector, test it first on a paragraph you know is yours, and read the score as one signal. RAID's authors recommend calibrating a detector on in-domain data before use.
If you are the writer
Anthropic's AI Fluency cheat sheet defines Transparency Diligence as "Being honest about AI's role in your work with everyone who needs to know".
- Keep your notes, outlines and drafts. They show how the text grew, which a detector score cannot.
- Tell your reader where AI helped whenever they need to know, and learn your school's or employer's rule before you write.
- If a score is used against you, ask which tool produced it, at what threshold, and what the tool's own documentation says. Offer your drafts and notes.
Common mistakes
- Treating a score as proof. A notice quoted in OpenAI's forum says it withdrew its own classifier for low accuracy. The card on its GPT-2 detector quotes OpenAI saying its own detection rate was not high enough for standalone detection.
- Trusting one detector. In the Stanford study the detectors disagreed widely: about a fifth of the TOEFL essays were flagged by all seven, while nearly all were flagged by at least one.
- Using an old detector on new models. In RAID, the RoBERTa detector trained on GPT-2 text rarely beat 60% on text from other models.
- Judging a non-native writer's essay by a score. The Stanford study found those essays were the ones the detectors flagged.
- Rewriting only to move a score. The fixes in the previous lesson aim at readers, and a score can move without the text getting more specific.
Try it yourself
- Pick a paragraph you wrote before 30 November 2022, the day ChatGPT launched. Wikipedia's guide says that if an edit was made before that date, AI use can be safely ruled out for the text.
- Run it through a free detector you can find, and write down the score and the tool's name. If the tool calls your own old paragraph AI-written, you have met a false positive first-hand.
- Run a paragraph from your own AI-assisted draft through the same tool, and compare the two scores.
- Write down three things you would ask a writer for if a score were all you had.
Next course: AI connectors
You now have a full writing routine: draft from your own notes, edit without losing your voice, fix the signs, and answer questions about detectors. The next course, AI connectors, starts with ChatGPT apps and connectors, and how to turn one on.
Check yourself
Why did OpenAI withdraw its AI text classifier in July 2023?
Show answer
A. A notice quoted in OpenAI's forum said it was no longer available due to its low rate of accuracy The notice, as a reader quoted it in OpenAI's forum, gives its low rate of accuracy as the reason.
In the Stanford study, what was the average false positive rate on essays by non-native English writers?
Show answer
D. 61.22% The seven detectors averaged 61.22% on the 91 TOEFL essays. The 19.78% is the share flagged by all seven, and the 97.80% is the share flagged by at least one.
What does a watermark check need, according to the SynthID Text post?
Show answer
B. A trained detector for each watermark setup The post says every watermarking configuration needs its own trained detector.
A detector flags a student's essay as AI. What does Wikipedia's guide advise?
Show answer
C. Do not rely on the tool alone, because it has non-trivial error rates, and ask the writer to explain their choices The guide tells its editors not to rely solely on detectors and lists explaining one's own choices among the signs of human writing.
Frequently asked questions
Do AI detectors work?
They give a score, and the sources in this lesson say not to rely on it alone: Wikipedia's guide says detection tools have non-trivial error rates. The sections above on false positives and on the RAID benchmark give the details.
Why did OpenAI shut down its AI detector?
Because of its low rate of accuracy, according to a notice a reader quoted in OpenAI's developer forum. The classifier had launched on 1 February 2023.
Can AI detectors flag human writing as AI?
Yes, and that is called a false positive. On the challenge set OpenAI described, as quoted in its forum, its classifier called 9% of human text AI-written. In the Stanford study the detectors flagged more than half of the non-native English essays.
What is an AI watermark?
A signal built into the words as the model writes, which only a matching checker can read. Two methods are a 2023 University of Maryland paper's green list scheme and Google DeepMind's SynthID Text. A tool carries one only if its maker added it.
Can a teacher prove I used ChatGPT?
Not from a score alone. Wikipedia's guide says detectors have non-trivial error rates and tells editors not to rely on them alone. Drafts, notes and an explanation of your choices are better evidence.
Should I run my own essay through an AI detector first?
It shows how one tool reacts, and the score is not a verdict. Detectors disagree with each other: in the Stanford study, about a fifth of the essays were flagged by all seven and nearly all by at least one. Keep your drafts and notes instead.
Sources
- OpenAI developer forum: New AI classifier for indicating AI-written text https://community.openai.com/t/new-ai-classifier-for-indicating-ai-written-text/46532
- Hugging Face: roberta-base-openai-detector (OpenAI's GPT-2 output detector) https://huggingface.co/openai-community/roberta-base-openai-detector
- Hugging Face Papers: GPT detectors are biased against non-native English writers https://huggingface.co/papers/2304.02819
- Hugging Face Papers: RAID benchmark of machine-generated text detectors https://huggingface.co/papers/2405.07940
- Hugging Face Papers: A Watermark for Large Language Models https://huggingface.co/papers/2301.10226
- Hugging Face blog: Introducing SynthID Text https://huggingface.co/blog/synthid-text
- Wikipedia: Signs of AI writing https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
- Anthropic: AI Fluency key terminology cheat sheet https://www-cdn.anthropic.com/4396730ed190e691a3712cf2fd6bfe35509deca2.pdf
- Turnitin on YouTube: Understanding false positives within Turnitin's AI writing detection capabilities https://www.youtube.com/watch?v=4e9zM2MZvRQ