SiddharthThe Invisible Signature: How AI Text Watermarking Works, Where It Breaks, and What It Can Never Prove
Siddharth
August 11, 202617 min read
Every word an AI model generates can carry an invisible statistical signature — one that survives copy-paste, survives editing, and can be verified without the model itself. But the signature answers a subtler question than "who wrote this," and the math of how it works reveals exactly where it fails.
Here's the setup. A language model is about to generate the next word in a sentence. Before it samples from its probability distribution, a cryptographic key silently biases which acceptable word gets chosen. Not enough to change the meaning — just enough that, over hundreds of words, a statistical pattern emerges. A detector holding the same key can recompute that pattern and ask: did this text's word choices align with our secret bias more often than chance would allow?
That's it. That's the entire field. Everything else — the red/green lists, the Gumbel tricks, the tournament samplers, the alignment searches — is a variation on how you apply the bias and how you detect it. The implementations are genuinely elegant. The failure modes are genuinely instructive. And the gap between what people think watermarks prove and what they actually prove is where the real story lives.
A text watermark is a shared source of pseudorandomness between the generator and the detector, derived from a secret key, that biases which of the many acceptable next tokens gets emitted. The generator and detector never exchange the text's provenance. They both recompute the same pseudorandom values from (secret key, local context), and the detector asks a statistical question.
Think of it like invisible ink and a UV light. The writer and the reader both know the formula for the ink. The writer embeds it in every sentence; the reader sweeps the UV light across the page and checks whether the ink shows up more than it would in a random document. The text itself hasn't changed — it reads normally to anyone without the light. But the pattern is there, statistically, waiting to be found.
Three properties are in perpetual tension, and you cannot maximize all three:
Detectability is statistical power per token — how strongly the signal shows up. Quality is how little the output diverges from what the model would normally produce. Robustness is survival under edits, paraphrase, and translation. You can have strong detection and good quality, but the mark will be fragile. You can have robustness, but you'll pay in detection power or quality or both. Every paper in this space is a different point on this triangle.
And then there's one hard resource constraint that no scheme escapes — the single most important fact in the entire field:
Text length is a proxy. The real budget is total entropy across the generated tokens. When the model is highly uncertain about what comes next — an essay, a story, an open-ended answer — there are many plausible tokens to choose from, and the watermark can bias that choice. When the model is certain — the answer to a math problem, the syntax of a code block, a factual lookup — there's only one right answer, and biasing it produces garbage.
This is not an engineering problem you can throw resources at. It's the second law of thermodynamics applied to language. If there's no noise, you can't hide a signal in it.
Every practical scheme — all four families — is an instance of the same loop. The only thing that differs is one line: how WATERMARKED_SAMPLE works.
Two design choices dominate everything downstream.
Context width h — how many previous tokens seed the pseudorandom function. This is the single most consequential parameter, and the one most often stated wrong. With h = 0, the pseudorandom bias is fixed for the entire text — maximally robust to edits (changing one token doesn't desynchronize anything), but maximally vulnerable to stealing: query the model enough times and you can recover the global bias. With h small (1–4), each edit corrupts at most h+1 scoring positions, bounding the damage. This is the standard production regime. With h large, a single edited token desynchronizes every subsequent seed, and the mark evaporates. Near-unbreakable to reconstruct, but useless.
Where the randomness enters. Either you modify the distribution (Family A) or you keep the distribution intact and fix the sampling rule (Families B and C). This distinction is not cosmetic — it determines whether the watermark degrades quality at all.
Kirchenbauer, Geiping, Wen, Katz, Miers, and Goldstein published A Watermark for Large Language Models at ICML 2023 . It became the reference design — the one everyone reimplements and benchmarks against.
The mechanism is beautifully simple. At each generation step, the previous h tokens are hashed with the secret key to produce a pseudorandom seed. That seed shuffles the entire vocabulary and splits it into two lists: a green list (typically 25–50% of the vocabulary) and a red list (the rest). Then, before sampling, every green-list token gets a small boost to its logit score.
Here's the subtle part that most explainers get wrong. The logit boost δ doesn't force green tokens — it nudges them. At a high-entropy position (the model is torn between several words), a boost of 2.0 substantially reorders the candidates. At a low-entropy spike (the model is 99.9% sure the next token is a closing parenthesis), the boost is a rounding error — the correct token still wins. The watermark automatically applies itself where there's room and backs off where there isn't.
Detection is a binomial test. Under the null hypothesis (text independent of the key), each token lands in its green list with probability γ. The green count over T scored tokens follows Binomial(T, γ), and the one-sided z-statistic is:
z = (|s|G − γT) / √(T · γ · (1 − γ))
This is not "more than 50% green." With γ = 0.5, 50% green is the null expectation — it's what unwatermarked human text produces on average. The threshold is set by your target false positive rate, derived from the consequence of being wrong. For z = 4.0, you get roughly p ≈ 3×10⁻⁵. For accusatory use, you want z ≥ 5 or better.
There's a well-known bug here: the binomial null assumes independent draws, but real text repeats n-grams — names, technical terms, boilerplate. If the same h-gram context recurs, you re-score the same pseudorandom draw multiple times, inflating the z-statistic and producing false positives on repetitive human text. The fix is deduplication: score each distinct (context, token) pair once. Every serious implementation does this. Skipping it is the most common way a homegrown detector ends up accusing people who write with a lot of repeated phrasing.
The quality cost is real. The honest framing: δ buys detectability and pays in KL divergence from the model's distribution. It's often imperceptible to readers, but it is a genuine distributional shift. You cannot claim distortion-freeness for KGW. Later schemes like DiPmark modify the reweighting so the average over keys preserves the distribution, recovering most of the quality while keeping a green-list-shaped detector.
This family does something radically different: it doesn't touch the distribution at all. It replaces the random in "random sampling" with pseudorandom derived from the key.
The mathematical heart is the Gumbel-max trick, due to Aaronson and Kirchner (2022). At each position, you generate one uniform random number per vocabulary item — but the random numbers are pseudorandom, seeded by (key, context). Then instead of sampling normally, you pick the token that maximizes ri1/pi. The key mathematical fact: over the randomness of r, this argmax is distributed exactly as p. The output is a perfect sample from the model's distribution.
The watermark is free in quality terms. There is no quality/detectability tradeoff knob, because there's no distortion to trade. The signal is that the specific token chosen correlates with the specific r values — a correlation only the key holder can verify.
Detection scores each token by how large its r value was. Under the null, each r value is uniform, so the score follows a Gamma distribution. Under the watermark, the sampler systematically favors tokens with large r, pushing the score above the null expectation.
This is reportedly the basis of at least one major lab's internal watermarking effort. It's mathematically beautiful. But the determinism is a genuine product liability: "regenerate" gives you the identical answer. Mitigations add a nonce or rotate keys per request — which reintroduces the multiple-testing problem at detection time.
Dathathri et al. published Scalable watermarking for identifying large language model outputs in Nature in 2024 . This is the only scheme with published evidence from real production deployment — validated against nearly 20 million responses from live interactions with a frontier model.
Instead of one pseudorandom object per position, the scheme derives m binary scoring functions and runs a knockout tournament:
Because the candidates are independent draws from the true distribution and ties are broken uniformly, the single-token marginal is preserved — the non-distortionary property. A distortionary variant breaks ties by preference instead, buying detection power at a quality cost.
Detection averages the g-values across all tokens and all layers. Under the null, each g-value is a coin flip (Bernoulli ½). Under the watermark, the tournament winner has systematically elevated g-values, pushing the mean above 0.5. Critically, the detector computes these keyed scores without needing the language model at detection time — detection is cheap, parallelizable, and doesn't require serving the generator.
There's a genuinely interesting vulnerability here. A 2026 analysis proved that the mean score is inherently vulnerable to increased tournament layers, constructing a "layer inflation" attack. The scheme violates self-robustness: stacking additional watermark layers progressively decreases the statistical separation between watermarked and unwatermarked text, weakening detectability rather than strengthening it. The authors argue this generalizes — any scheme whose detector relies on aggregated mean statistics is potentially susceptible.
Kuditipudi et al. (TMLR 2024) solve a problem the others don't: survival under insertions and deletions, which desynchronize every context-hashed scheme. Instead of deriving randomness from local context, they fix a key sequence of pseudorandom values, pick a random starting offset per generation, and walk through it. Detection doesn't recompute a hash — it searches for the best alignment between the text and the key sequence under an edit-distance cost, then computes a p-value by permutation test.
The wins: survives structural edits that would destroy any context-hashed scheme. The loses: detection is O(T · n) alignment work, far more expensive, and the permutation test needs many resamples for small p-values. This is the design point you reach for when robustness to structural edits matters more than detection cost.
| Family | Quality cost | Detection | Edit robustness | Production-proven |
|---|---|---|---|---|
| KGW (green/red lists) | Moderate (δ-dependent) | Binomial z-test | Moderate (h-dependent) | Widely benchmarked |
| Gumbel-max (Aaronson) | Zero (distortion-free) | Gamma distribution | Low (sync-sensitive) | Reportedly internal |
| SynthID-Text (tournament) | Zero (configurable) | Mean g-score | Moderate | ~20M live responses |
| Kuditipudi (alignment) | Zero | Permutation test | High (ins/del safe) | Research |
This is the section that matters most in practice. A watermarking scheme can be mathematically sound and still produce catastrophic false accusations — because the detection statistics are where deployments actually fail.
The false positive rate you target should be a function of what happens to someone who's wrongly flagged:
p < 10⁻³ is fine — errors wash out in aggregate.p < 10⁻⁶.p < 10⁻⁹, and you should still not treat it as dispositive.
Every one of these multiplies your effective false positive rate:
k keys means k hypotheses. Apply Bonferroni correction (p_threshold / k) or you've silently inflated your false positive rate by k times.W windows is W hypotheses. Correct for it.A detector that tests 3 keys × 4 model generations × 200 windows is running 2,400 tests. An uncorrected p < 10⁻⁴ threshold produces roughly a 20% false positive rate per document. This is not a hypothetical — it's the arithmetic of every real deployment.
Detection requires retokenizing candidate text, and retokenization frequently differs from what the model actually emitted. Copy-paste through a rich text editor, smart quotes, non-breaking spaces, normalized whitespace, Markdown round-trips — all silently shift token boundaries and destroy scoring positions. Normalize aggressively (NFKC, quote and dash canonicalization, whitespace collapse) before scoring, and expect measurable loss regardless.
Output a p-value and the scored-token count T. A z = 3.1 over 40 tokens and a z = 3.1 over 4,000 tokens mean very different things about how much you should trust the estimate. Surfacing a binary "AI: yes/no" throws away the only information that lets a human calibrate.
Two of these deserve special attention.
Watermark stealing (Jovanović et al., ICML 2024) matters because it enables spoofing, not just removal. For roughly $50 in API queries, an attacker can recover enough of the green-list bias to both remove watermarks and forge them — writing defamatory text that a detector attributes to a specific model. That's a worse failure than a missed detection. It's a direct argument for small γ, rotated keys, and never exposing an unrestricted public detection API.
"Watermarks in the Sand" (Zhang et al., 2024) is frequently cited as proving watermarks are hopeless. State its assumption precisely: the attack requires access to a quality oracle roughly as capable as the watermarked model itself, plus a perturbation oracle. That assumption is the entire result. If you had a model that good and unwatermarked, you'd just use it. The theorem is real and important; it is not "paraphrasing defeats watermarks."
At least one major frontier lab has begun embedding invisible text watermarks in its models from August 2026, alongside signed C2PA provenance metadata on generated files — applied at the model level across API access, coding tools, and conversational interfaces, worldwide rather than EU-only. The watermark is designed to travel with text through copy-paste and some editing. Verification tools for third parties are planned but undated.
The announcement is admirably clear about two caveats that belong in any honest discussion. First: a detected mark does not prove the model wrote the content, since people use models to edit or translate their own writing. Second: heavy editing or format changes can strip the marking entirely.
For contrast: the SynthID-Text implementation was open-sourced in a major ML library. The method is public; the production key is not. That's the right shape — it lets researchers attack the construction while keeping the deployed instance secure. "Detector is public vs. private" is the wrong axis; the real axis is "is the key public, and is there an unrestricted query oracle."
The most important caveat is also the most underrated. The watermark answers "did this text pass through the model?" — not "who wrote this?"
The field is at an inflection point. The math is mature — the four families cover the design space comprehensively, and the detection statistics are well understood. Production deployment has begun. What's missing is the social and legal infrastructure: who holds the keys, who gets to verify, what counts as evidence, and what happens to the falsely accused. The cryptography is ready. The governance is not.
If you're building a system that uses these techniques, the most important thing you can do is internalize the gap between what the statistics prove and what people will claim they prove. A watermark is a probability, not a verdict. Treat it that way.
Thoughts and essays, published with Yokush. See more posts
Siddharth7 min
Siddharth11 min
Siddharth15 min
Comments 0