Benchmark

How often our AI detector is wrong about a human

Every AI detector returns a confident percentage. None of them tells you how often that percentage is wrong about someone who wrote their own work. That second number is the one that matters if you are the student being accused, so it is the one we publish.

Last measured 9 August 2026. Every figure below comes from a script in the repository that can be re-run against the same labelled corpora.

Measured false-positive rates

The rate at which verified human-written academic text receives our strongest verdict. Accuracy is not one number: it changes with language and with document length, so we report four profiles rather than an average that would hide the weak ones.

What these numbers are not

• Long-form results are in-sample. The long-form weights were fitted on the same corpus they were then measured against, so the separation is optimistic. The human/AI gap is wide enough that the direction is not in doubt, but treat the figure as a working estimate rather than a benchmark.
• Zero observed is not zero.Long-form measurement saw no false positives in 40 human documents. We publish 3 in 100 anyway, because the rule of three puts the 95% upper bound near 7% at that sample size. Publishing “0%” would be the overclaiming this page exists to oppose.
• The AI corpora are small — 20 to 25 documents per language, from two model families. A model we have not tested may behave differently.
• Wikipedia is not thesis prose. The German long-form corpus is expository writing. It is formal, German, long and verifiably human, but it is not the register a student writes in.

What we measured against

Every human document below was published before ChatGPT was available, so its authorship is not a matter of judgement.

We once shipped a detector that was worse than a coin flip

In an early version, measurement against 236 pre-2022 arXiv abstracts returned an AUC of 0.184. A coin flip is 0.50. The detector was ranking machine-written text as more human than human-written text; inverting its output would have improved it. It had been returning confident percentages the entire time.

The cause was a family of lexical-richness features — Yule’s K, MTLD, the Zipf exponent, hapax legomena — all running backwards with large effect sizes. They score dense human academic prose as machine-like and lexically varied machine prose as human. That is the same mechanism behind the published finding that detectors falsely flag a majority of non-native English writers.

Those features now carry zero weight. They are still computed and still shown as evidence; they simply do not move the score. We publish this because it is the strongest argument we have for measuring: the failure was invisible from the output and obvious from the corpus, and any detector that has never run this test has no idea whether it has the same problem.

What no detector can do

No AI detector, including this one, is proof of authorship. These tools measure statistical properties of text, and human writing that is formal, structured, or written by a non-native speaker shares many of those properties. A score is a reason to ask a question, never a reason to conclude an answer. If a detector has been used against your work, our guide on what to do when a detector flags your thesis sets out how to respond.

Researchers and journalists: the corpora, the scoring code and the calibration harnesses are all in the repository behind this site. If you want to reproduce or challenge any figure here, write to contact@morphica.de and we will help.