Why Tonic Textual beats LLMs at PII detection and synthesis: the PrivacyBench results

PrivacyBench is an open benchmark from Tonic.ai that measures how well a system de-identifies workplace data exports (email and Slack messages full of names, email addresses, and slack handles) by scoring both halves of the job: detecting the PII and replacing it with coherent synthetic values. The headline result is that using Tonic Textual for the detection stage, with Opus 4.8 handling synthesis in both cases, beats letting Opus do the whole job itself, lifting end-to-end de-identification 4.3 points, from 87.7% to 92.0%, and catching more of the PII to begin with (95% recall versus 88.8%). The 21 test datasets are generated by Tonic Fabricate, so the ground-truth labels come with no human annotation.

What PrivacyBench measures (and how we built it)

Teams de-identify their email and Slack exports so they can safely put that data to work training and fine-tuning models, building agent and reinforcement-learning environments, and sharing data with partners, without leaking the PII buried in the free text. PrivacyBench measures how well a de-identification system actually does that. It's one of Tonic.ai's AI model benchmarks, and it treats de-identification as a two-stage task, scoring both: detecting the sensitive entities (named entity recognition, or NER) and replacing them with coherent synthetic values. Those two stages map to the two things any de-identified dataset has to get right at once: privacy (the real PII is gone) and utility (what replaces it stays coherent enough to use).

The second stage is the hard, under-measured part. In the original real data, one person can turn up as a formal name, a nickname, an email address, and a Slack handle across dozens of messages. A good synthesis system has to collapse all of those surface forms onto a single, consistent synthetic identity. Existing NER benchmarks, like CoNLL, OntoNotes, and TAB, score detection well, but none score the quality of the replacement. That gap is what this de-identification benchmark fills.

A strong replacement has to hold three things together:

• Cross-message consistency: every mention of one person maps to the same synthetic identity across every message.
• Within-identity derivation: the synthetic email and handle plausibly derive from the synthetic name, and a nickname maps to a nickname form of it, not an unrelated name.
• Organization coherence: every surface form of one employer maps to forms of a single synthetic organization, with dependent values like email domains kept in step.

Tonic.ai built PrivacyBench to score exactly this. Tonic Fabricate generated the labeled datasets, and the benchmark evaluates six de-identification pipelines that each pair a detection backend (an LLM or Tonic Textual) with LLM-based synthesis.

How PrivacyBench scores it: Three metrics

PrivacyBench reports three scores, one score for NER, one score for choosing the synthetic replacements, and one score that combines both measuring the entire de-identification pipeline. Each is measured against ground-truth spans fixed when the data was generated.

Privacy-bench metrics: what each one measures

Metric	What it measures
NER recall	Of the PII known to be in the data, how much did the pipeline catch and actually replace? A miss here is a privacy leak.
Synthesis accuracy	Of the entities it did replace, how many replacements are coherent? An LLM judge scores each one, per character, against the synthetic identity it should belong to. No prior benchmark scores this.
NER + synthesis accuracy	The bottom-line score. It folds detection misses into the denominator, so a pipeline can’t inflate its synthesis score by detecting only the easy entities.

Three scores exist because each of the first two, alone, hides something. Recall is really a privacy measure: it captures how much PII you caught and replaced, but says nothing about whether the de-identified data is still usable. Synthesis accuracy is really a utility measure. It asks whether the replacements are coherent enough to keep the data usable, but on its own it's gameable: detect a handful of easy entities, synthesize them perfectly, and post a high number while missing the hard PII. The combined score is the honest one because it captures privacy and utility together: it counts a missed entity as a failure right alongside an incoherent replacement, the way an F-score refuses to let precision and recall be traded off against each other. Getting both right at once is the whole goal of synthetic data. The code that computes all three is open source.

How the benchmark data was built

Every PrivacyBench dataset is fully synthetic, generated by Tonic Fabricate from a seed that declares each character's PII before any text exists, which makes the ground truth a byproduct of generation rather than a human-annotation project. That dissolves the two problems that make PII benchmarks expensive: a public benchmark can't contain real PII, and trustworthy labels are slow and costly to hand-tag. Because the cast of characters and their identifiers are fixed up front, the corpus is written to use exactly those values, and the labels are known without anyone tagging a single message.

The v1 release spans 21 email and Slack datasets across industries including pharma, retail, finance, airlines, tech, and manufacturing. The seeds deliberately engineer the ambiguity that makes attribution hard by

• having multiple characters with the same name;
• having many characters with nicknames; and
• in messages sometimes you can only tell which character is being referred to by reading contextual messages before and after a given message.

For example, one set uses two different people who both go by "Chris" in the same workspace, separated only by context, one Chris’ full first name is Christopher and the other’s full first name is Christine. Synthetic data lets the authors produce labeled, trustworthy benchmark material with no privacy exposure and difficulty they dial in on purpose. The full corpus is public on Hugging Face.

What the benchmark found

Across all six pipelines, using Tonic Textual for detection produced the best end-to-end de-identification: Textual paired with Opus 4.8 scored 92.0% on the combined NER + synthesis metric, ahead of every LLM-only pipeline. De-identification through Textual also costs over 60% less than using Opus (see Fig. 2 below), and the benchmark reports it runs faster too. The six pipelines pair two detection backends with three Anthropic synthesis models, which lets the results separate the contribution of detection from the contribution of synthesis.

Privacy-bench results by NER backend and synthesis model: NER recall, synthesis accuracy, and combined synthesis + NER accuracy

NER backend	Synthesis model	NER recall	Synthesis accuracy	Synthesis + NER accuracy
Textual	Opus 4.8	95.0%	97.0%	92.0%
Opus 4.8	Opus 4.8	88.8%	99.0%	87.7%
Textual	Sonnet 4.6	95.0%	90.5%	85.9%
Textual	Haiku 4.5	95.0%	87.3%	82.8%
Sonnet 4.6	Sonnet 4.6	83.8%	90.5%	76.2%
Haiku 4.5	Haiku 4.5	79.9%	88.5%	71.1%

Three things stand out. First, on Textual vs. LLM PII detection, Textual's recall stays flat at 95% no matter which model handles synthesis, because detection is decoupled from the synthesis model. LLM-only detection doesn't hold up the same way: its PII detection accuracy falls from 88.8% with Opus to 83.8% with Sonnet to 79.9% with Haiku as the model shrinks. That decoupling is also a cost lever: you can run synthesis on a smaller, cheaper model like Haiku and still detect at 95%, where an LLM detecting on its own drops to 79.9%.

Second, one number looks like a win for LLM PII detection and isn't. The Opus-only pipeline posts the single highest synthesis accuracy of any pipeline, 99.0%, but only because it had fewer, easier spans to work with. Its lower recall (88.8%) means it never detected the hard PII, so it was graded on the easy entities it did catch. Fold those detection misses back into the denominator and its bottom-line score drops to 87.7%, behind Textual + Opus's 92.0%. That gap is exactly the gaming the combined metric is built to neutralize.

Third, the benchmark sorts the models cleanly by capability (Opus > Sonnet > Haiku), and does so across both detection backends. Recovering that known ordering on a task none of the models was tuned for is the real validation that PrivacyBench measures capability, not noise.

The case for Textual over raw LLMs

If you're de-identifying unstructured data for AI, the benchmark points one way: use Tonic Textual for the detection stage instead of asking an LLM to do the whole job itself. In every pairing, swapping Textual in for detection raised recall to a flat 95%, lifted the end-to-end de-identification score, and did it faster and more cheaply than LLM-based detection. And because Textual's recall doesn't degrade when you pair it with a smaller synthesis model, you can run detection through Textual and synthesis through a cheaper LLM without giving up detection quality.

Beyond the benchmark, Textual uses proprietary NER to detect, redact, and coherently synthesize sensitive unstructured text for model training and fine-tuning, RAG pipelines, LLM privacy layers, secure partner data sharing, and data monetization. Textual is a practical foundation for secure AI model training where the data has to be safe before it's useful. For teams working under regulatory obligations, the same de-identification keeps sensitive data compliant as it moves into AI workflows. Tonic Fabricate is the complementary piece: it generated the benchmark's trustworthy labeled data, and it produces labeled training and evaluation data the same way. In legal tech, Ontra uses Textual to consistently de-identify and synthesize the unstructured data behind its AI work, which is how its team generates ground-truth datasets faster.

The strongest case is the one you run yourself. PrivacyBench is open, so you can put your own de-identification workflow to the same test: point your pipeline at the 21 public datasets and the shared scoring code, see where it scores, and check whether using Textual for detection moves your combined number the way it did across these six pipelines. To try Textual on your own data, explore the product or spin up a free Textual account and scan your first documents in minutes.