Open-Source LLMs Are Harder to Detect. Here's Why

OpenAI retired its own AI text classifier in July 2023 for low accuracy. It caught 26% of AI text and false-flagged 9% of human text, which is usually filed away as a story about an early, weak tool. The more useful reading is that the company holding the weights, the training data and the sampling code still could not build a dependable detector for its own model's output.

Open source LLM detection is harder still, because there is no fixed target. Fine-tuning, quantization and sampling settings each shift a model's statistical fingerprint, and local inference leaves no provider log to cross-check. Detectors trained on a handful of closed models generalize poorly to thousands of community variants nobody catalogued.

Why open source LLM detection is a different problem

Every text detector is, underneath, a comparison. It measures properties of your text and asks how well they match what it has learned about machine-generated writing. That learning came from somewhere, and where it came from decides what the tool is good at.

Train on GPT-family and Claude-family output and you get a tool tuned to those families' habits. Point it at output from a community fine-tune of an open-weight base, trained on a few thousand roleplay transcripts and running at temperature 1.1, and you have handed it a genre it has never read. Sometimes it still fires, because a lot of statistical regularity is shared across all transformer output. Sometimes it shrugs.

That variance is the whole story. It is not that open models are magically invisible. It is that your uncertainty about any single score goes up, and uncertainty is the thing most detector interfaces are worst at communicating.

What a model fingerprint actually is

Strip away the marketing and a text detector is measuring how surprising your word choices are to a language model, plus how that surprise is distributed across the piece. Low average surprise and unusually even distribution reads as machine. High surprise with lumpy variation reads as human. Our explainer on perplexity and burstiness walks through the mechanics properly.

The fingerprint is not one number. It is a shape: a distribution over token probabilities, sentence lengths, punctuation habits, transition-word frequency, and the degree to which the text keeps returning to the most probable next word instead of the third or fourth. Closed models each carve out a recognizable region of that space and stay there, because the vendor controls the decoding defaults for everyone.

Open weights hand that control to the user. Which brings us to the knobs.

Four knobs that move the fingerprint

Knob	What it changes	Effect on detection
Fine-tuning	The model's learned preferences, on new data	Largest effect; can create a style the detector never trained on
Sampling settings	Temperature, top-p, repetition penalty	Directly reshapes the probability curve detectors measure
Quantization	Weight precision, from 16-bit down to 4-bit	Subtle drift in output distribution; small but real
System prompt	Voice, structure, register before a word is generated	Cheap, instant, and frequently underestimated

Temperature deserves a note. Turn it down and output becomes more predictable, which makes it easier to flag. Turn it up and prose gets erratic, which reads as more human to a perplexity-based tool while quietly getting worse at the task it was asked to do. That trade-off is real, and it is why "just raise the temperature" is not the reliable evasion people assume.

Fine-tuning is the heavyweight. A model trained further on a specific corpus starts producing that corpus's habits, and if those habits resemble ordinary human prose in a specific domain, detectors have less to grab. This is also why the same mechanism that produces false negatives on fine-tunes produces false positives on formulaic human writing. Both are cases of the fingerprint failing to separate two populations.

Why closed models are the easy case

There is a comfortable assumption that closed models are easier to detect because they are somehow more distinctive. That is not it. They are easier for four boring operational reasons.

There are few of them, they are versioned, they are centrally hosted, and their decoding defaults are set by the vendor rather than the user. A detector vendor can build a training set against maybe a dozen widely used endpoints and cover most of what real people actually produce. Add the fact that a provider holds server-side logs, and closed-model use leaves corroborating evidence that open-weight use simply does not.

Open weights break all four properties at once. Nobody knows how many derivative checkpoints exist. Nobody versions them consistently. They run on hardware you cannot subpoena. And the person running the model chose the settings. Our piece on detecting DeepSeek and other open-weight models works through what that means for one specific family.

There is a second-order effect worth naming. Detector vendors train on what they can collect cheaply, and what they can collect cheaply is output from popular hosted endpoints. So every improvement cycle re-centers the training distribution on closed models, making tools a bit better at the case that was already easiest and no better at the case that was already hardest. Nobody designed that dynamic. It falls out of where the data lives, and it will keep operating until someone funds the unglamorous work of collecting output from thousands of community checkpoints.

What the field is actually trying

Four approaches, honestly rated.

1. Generation-time watermarking. Statistically bias token selection so the output carries a recoverable signal. Elegant, and useless for open weights, because anyone can delete the watermarking code before running the model.
2. Training-free curvature methods. Zero-shot approaches measure how a text's probability behaves under small perturbations, which generalizes across models better than a trained classifier. They need model access or a proxy model, they are slow, and they degrade under paraphrase.
3. Cross-model ensembles. Score against several reference models and combine. This is the pragmatic mainstream, and it is why AI Detector 360 runs multiple engines, publishes an explicit confidence level, and shows a sentence-level heatmap instead of one number pretending to certainty. The weighting is documented on our methodology page.
4. Process evidence instead of text evidence. Version history, drafts, and in-person work. Unfashionable, unglamorous, and by far the most robust option when the model is running on someone's laptop.

The RAID benchmark (Dugan et al., ACL 2024) is the sobering backdrop for the first three: across more than 10 million documents and 12 adversarial attacks, commercial detectors degraded sharply under paraphrase and homoglyph attacks. Open-weight variation is a related pressure applied earlier in the pipeline.

A framework for people who have to decide anyway

Screening still has value. It just needs thresholds matched to consequences.

Take a concrete case. Priya runs contributor operations at a trade publication and screens a few hundred pitches a month. She is not trying to catch cheaters; she is trying to protect a rate card. Reject on a score alone and she will eventually turn away a careful, structured writer whose prose happens to read machine-like, and that writer will tell everyone she knows. Ignore scores entirely and Priya pays professional rates for something a laptop produced in ninety seconds. The way out is procedural rather than technical: the score decides who gets asked for drafts, and the drafts decide who gets paid.

• If the decision is editorial triage (which of 400 submissions gets a human read first), a detector is excellent. False positives cost you a few minutes of reading. Use it freely.
• If the decision is payment or publication, require a second signal: drafts, a source conversation, a live edit. A score alone should never move money.
• If the decision is disciplinary or contractual, the score is an exhibit, not a finding. Anything less is the Texas A&M mistake in a new costume.

Run the numbers on the middle case. A content team screening 5,000 submissions a month pays roughly $100 to $300 in per-detection costs at the $0.02 to $0.06 range measured in the 2025 NBER working paper by Jabarian and Imas. Cheap. Now apply Turnitin's disclosed sentence-level false-positive rate of about 4% to a 40-sentence article and you should expect roughly 1.6 sentences wrongly highlighted in a perfectly human piece. If your editors treat every highlighted sentence as an accusation, that cheap screening just became expensive.

A policy sentence worth stealing, for a contributor agreement or syllabus:

We screen submissions with automated AI detection as a triage signal only. A flag prompts a conversation and a request for drafts or version history; it is never treated on its own as a finding of undisclosed AI use.

That single sentence prevents more damage than any tool upgrade.

What nobody can verify yet

We cannot tell you a reliable accuracy figure for detection of a specific open-weight fine-tune, and neither can anyone else, because the population of fine-tunes is unbounded and unlisted. Published benchmarks cover a sample; your suspect text came from outside it.

We also cannot tell you that scores from any tool, ours included, mean the same thing across model families. They do not. That is precisely why a sentence-level heatmap and a stated confidence level matter more than a headline percentage: you can see whether the tool found a concentrated pattern or spread a thin suspicion across everything.

Two things do hold up. Human experts remain the strongest detector on record, at 99.3% accuracy in the 2025 ACL study by Russell, Karpinska and Iyyer, robust to evasion that beats automated tools. And evidence about how a document was written beats evidence about how it reads. You can test both today with our free AI detector, which gives you 5,000 characters with no sign-up, or read how AI detectors work first. Volume users will find the credit math on the pricing page, where 1 credit covers 100 words.

Detection of open-weight output is not hopeless. It is just not a percentage you get to hide behind.