How AI Detectors Are Trained (and Why They Drift)

Two scans, one detector, same afternoon. A 1,200-word essay drafted straight out of a chatbot comes back flagged hard, and so does a 1,200-word essay written by a student who learned English in a Turkish high school. Only one of those answers is right, and the tool isn't malfunctioning in the second case: it's doing precisely what its training taught it to do.

AI detectors are trained on paired corpora of human and machine text, learning a statistical boundary between the two rather than any rule about authorship. That boundary is only as good as the corpus behind it, and every new generator model moves the machine side of the line, which is why accuracy decays quietly between retrains.

How AI detectors are trained, step by step

Strip away the marketing and the pipeline is boringly conventional machine learning. It runs in five stages.

First, someone assembles a human corpus: essays, articles, forum posts, product reviews, scientific abstracts, whatever the tool is meant to police. Second, they generate a machine counterpart, prompting a spread of models to produce comparable text in comparable genres at comparable lengths. Third, every sample gets a label. Fourth, a model learns the boundary, typically a transformer fine-tuned for binary classification, sometimes a lighter feature model built on predictability statistics. Fifth, and most consequentially, somebody picks the cutoff score at which the tool says "AI."

That last step is a policy decision wearing a lab coat, and it belongs to product managers as much as to researchers. The complete walkthrough of detector mechanics covers the inference side. This piece is about everything that happens before the model ever sees your paragraph.

Worth noting: not every detector is trained at all. Zero-shot methods score text against a language model's own probability surface without a labeled corpus, which is a different set of trade-offs entirely, and the perplexity and burstiness explainer unpacks the statistics underneath both approaches.

The corpus is the product

Everything a detector believes comes from its corpus, so the sourcing decisions are the whole ballgame.

On the human side, the obvious move is to scrape the web. The obvious move is also contaminated: anything published after roughly late 2022 may contain AI writing that nobody labeled. Clean pre-2022 archives exist, but they skew toward published, edited, professional prose, which is exactly the register that a nervous undergraduate does not write in. Train on polished journalism and you build a model that quietly treats "polished" as "human," or, depending on which side of the boundary the prose lands, as "machine."

On the machine side the question is coverage. Which generators? At what temperature? With what prompts? A corpus built from a hundred variations of "write a five-paragraph essay about photosynthesis" teaches a detector to recognize one narrow slice of default chatbot voice. Real submissions come from custom instructions, multi-turn edits, and models nobody in the lab tested.

Label noise, and why it costs more than it looks

A label is a claim that a specific document was written by a human or a machine. Every corpus contains some claims that are false.

Consider the scale of ambient AI writing. Turnitin ran more than 200 million papers through its AI detector in its first year, from April 2023 to April 2024, and reported that 11% came back at 20% or more likely AI writing, with 3% at 80% or more. Whatever the true error rate inside those figures, they describe an environment where a nontrivial share of ordinary student text has machine involvement. Scrape a corpus from that environment and label it "human" by default, and you have taught your model that some AI writing is human writing.

The reverse contamination happens too. Text generated by a model and then heavily edited by a person sits in the machine pile carrying human fingerprints. Both errors blunt the boundary in the same direction: toward mush.

Hand-labeling helps, and humans are surprisingly good at this. A 2025 ACL study by Russell, Karpinska and Iyyer found that expert annotators who use ChatGPT frequently identified AI-generated articles with 99.3% accuracy, holding up against evasion tactics that defeated automated tools. But expert annotation does not scale to millions of documents, and the people who are that good at it are not cheap.

Domain imbalance is how TOEFL essays end up flagged

The most cited failure in this field is a training-data story, not a model-architecture story.

In the 2023 Stanford study published in Patterns, Liang, Zou and colleagues ran seven detectors over TOEFL essays written by non-native English speakers. The detectors flagged an average of 61.3% of them as AI-generated. One flagged 97.8%. The same seven tools were nearly flawless on essays by US eighth-graders writing in their first language.

Nothing in those detectors was designed to penalize second-language writers. The corpus did it. Native-speaker student writing was abundant in training; second-language writing with its narrower vocabulary and more regular sentence rhythms was not, and the model had no reason to learn that this pattern also belongs to humans. We walk through the downstream consequences in our piece on why human writing gets flagged.

Apply that to a real classroom. In a 200-student first-year composition course where 40 students are international, a detector carrying the average bias from that study would flag roughly 24 of those 40 essays before anyone reads a word. Even at a fraction of that rate, the burden falls entirely on one group.

Drift: why a detector that was sharp in March struggles in September

Training freezes a snapshot of how machines write. Machines then keep changing.

Each frontier release shifts the machine-side distribution. Newer models produce longer-range coherence, more sentence-length variation, fewer of the tics that made 2023 output easy to spot. A classifier tuned on the old distribution starts missing the new one, and the first symptom is a quiet rise in false negatives that nobody notices because nobody complains about a passing score. Our analysis of whether detectors catch GPT-5-class writing digs into how much of that gap current tools have actually closed.

Adversarial pressure runs on its own clock. The RAID benchmark from Dugan and colleagues at UPenn (ACL 2024) tested detectors against more than 10 million documents and 12 attack types, and commercial tools degraded sharply under paraphrase and homoglyph attacks. Those attacks get repackaged into consumer tools within weeks of publication.

The pattern to notice: three of those five rows push toward missing AI, and two push toward flagging humans. A vendor can fix one column and let the other quietly worsen, then report a single "accuracy" number that hides the trade.

Retraining cadence, and what vendors will not tell you

Here is the honest part. Almost nothing about retraining is publicly verifiable.

No major text-detection vendor publishes its corpus composition, its retraining calendar, its per-language false-positive rates, or a changelog you could audit. When a detector's behavior shifts, users find out by noticing that last month's score no longer reproduces. Anyone who tells you they know a commercial detector's exact training recipe is guessing.

What we do have is independent measurement. A 2025 NBER working paper by Jabarian and Imas at the University of Chicago tested commercial detectors against a strict policy cap of 0.5% false positives and found exactly one that met it, at per-detection costs of two to six cents. That cost figure matters more than it looks: continuous retraining and multi-engine inference are expensive, and price pressure is one reason cadence slips.

The market is also consolidating. GPTZero was acquired by Superhuman, Grammarly's parent, in June 2026, with roughly 19 million registered users and about $30 million in ARR at acquisition according to TechCrunch. Consolidation usually means more retraining budget and less independent comparison.

Copy this into an email before you buy anything:

When was your current detection model last retrained, and against which generator models was it evaluated? What is your measured false-positive rate on non-native English writing specifically? What score threshold triggers a flag in our account, and can we change it?

If the answers are vague, that is your answer. AI Detector 360 publishes ours on the methodology page, including how we set confidence bands, why short samples get refused instead of scored, and which formats each engine covers across text, PDF, DOCX, image and video.

The strongest objection: just train on more data

The reasonable counter to all of this is that these are early-days problems. Scale the corpus, scale the compute, and the boundary sharpens. It's a fair argument, and it isn't entirely wrong: detection has genuinely improved since 2023, when OpenAI retired its own classifier for catching only 26% of AI text while false-flagging 9% of human text. Nobody ships numbers that bad now.

But scale does not fix three things. It does not fix label noise, because more scraped text means more contamination. It does not fix the moving target, because generator capability scales at least as fast as detector capability. And it does not fix the fundamental overlap: some human writing is genuinely, legitimately predictable, and no amount of data separates two distributions that physically intersect.

More data makes a better detector. It does not make a certain one.

What this means for the score in front of you

Practical translation of everything above:

• If the text is in a genre or language the tool probably saw little of (second-language academic writing, technical boilerplate, translated text), weight the score down and look for corroborating evidence.
• If the generator is newer than the detector's last known update, a clean pass means less than it appears to.
• If a single engine produced the score, you are seeing one corpus's opinion. Agreement across independently trained engines is a meaningfully stronger signal, which is why AI Detector 360 runs multiple engines and reports an explicit confidence level rather than a lone percentage.
• If the sample is under a couple of hundred words, no training pipeline saves you. There isn't enough signal to classify.

Run anything you're unsure about through our free AI detector for a sentence-level heatmap and a downloadable PDF report, then treat the output the way you'd treat a lab result from an instrument whose last calibration date you don't know. Useful. Not final.