Benchmark

How often our AI detector is wrong about a human

Every AI detector returns a confident percentage. None of them tells you how often that percentage is wrong about someone who wrote their own work. That second number is the one that matters if you are the student being accused, so it is the one we publish.

Last measured 24 August 2026. Every figure below comes from a script in the repository that can be re-run against the same labelled corpora.

Measured false-positive rates

The rate at which verified human-written academic text receives our strongest verdict. Accuracy is not one number: it changes with language and with document length, so we report four profiles rather than an average that would hide the weak ones. Every rate is a 95% upper bound rather than the count we observed — the bound is what we are entitled to claim, and it is always the worse of the two. The thresholds in these tables are on our internal measurement scales; the score a customer sees is the same information on one fixed display axis - amber begins at 20, red at 50 - with each measured threshold mapped exactly onto those anchors.

The weaker verdicts, which are noisier

Most readers never see the strongest verdict. They see one of these, and the error rates here are the ones that will actually apply to them. The English lower amber band (“some AI fingerprints”) is the worst of them by a wide margin.

What these numbers are not

• Zero observed is not zero. Neither long-form profile produced a single false positive at the strongest threshold — 0 of 1,412 English papers, 0 of 866 German theses. We publish 0.21% and 0.35% anyway, because those are the 95% upper bounds at those sample sizes. Publishing “0%” would be the overclaiming this page exists to oppose.
• The English corpus is the wrong genre. Those 1,412 documents are journal articles: edited, professional, written by teams. A student thesis is none of those. The rate is honestly measured on what it says it was measured on, and the English thresholds are deliberately left where they are until enough second-language English student theses exist to justify moving them. We hold 20.
• Short text has no published rate at all. The short profile governs a 300-400 word window, and no human corpus we hold is drawn from that window. The rates that used to appear here for it were removed on 21 August 2026 rather than left standing, because a plausible number with no population behind it is still an unfounded claim.
• The AI corpora are small — 20 to 25 documents per language, from two model families. A model we have not tested may behave differently.
• French and Spanish are uncalibrated. They return a score and no accuracy claim, and they never issue a “strong AI fingerprints” verdict.

What we measured against

Every human document below was published before ChatGPT was available, so its authorship is not a matter of judgement.

We once shipped a detector that was worse than a coin flip

In an early version, measurement against 236 pre-2022 arXiv abstracts returned an AUC of 0.184. A coin flip is 0.50. The detector was ranking machine-written text as more human than human-written text; inverting its output would have improved it. It had been returning confident percentages the entire time.

The cause was a family of lexical-richness features — Yule’s K, MTLD, the Zipf exponent, hapax legomena — all running backwards with large effect sizes. They score dense human academic prose as machine-like and lexically varied machine prose as human. That is the same mechanism behind the published finding that detectors falsely flag a majority of non-native English writers.

Those features now carry zero weight. They are still computed and still shown as evidence; they simply do not move the score. We publish this because it is the strongest argument we have for measuring: the failure was invisible from the output and obvious from the corpus, and any detector that has never run this test has no idea whether it has the same problem.

What no detector can do

No AI detector, including this one, is proof of authorship. These tools measure statistical properties of text, and human writing that is formal, structured, or written by a non-native speaker shares many of those properties. A score is a reason to ask a question, never a reason to conclude an answer. If a detector has been used against your work, our guide on what to do when a detector flags your thesis sets out how to respond.

Researchers and journalists: the corpora, the scoring code and the calibration harnesses are all in the repository behind this site. If you want to reproduce or challenge any figure here, write to contact@thesisdraft.com and we will help.