PrivacyBench V2.5
Open Benchmark

PrivacyBench, the open benchmark for end-to-end de-identification

PrivacyBench scores de-identification end to end: raw email, Slack, and document exports in, a coherent synthetic export out. See how Tonic's data de-identification pipeline compares with six frontier LLM pipelines. The data, answer key, and scoring code are all public.

In Tonic.ai's PrivacyBench v2.5 benchmark
Tonic's Identity Synthesis pipeline
95.3%
GPT-6 Astra
90.1%
Claude Opus 5
89.9%
GPT-5.6 Sol
86.7%
Claude Sonnet 5
84.7%
GPT-5.6 Terra
82.2%
GPT-5.6 Luna
65.4%
Results

PrivacyBench scores sensitive data detection and coherent synthesis, end-to-end

Detection

Find every piece of PII in raw email, Slack, and document exports.

Tonic's Identity Synthesis pipeline
92.5% F1

PII detection against human annotations. Highest of any engine tested.

Coherent synthesis

Every mention of a person, in every file, has to map to one synthetic identity.

Tonic's Identity Synthesis pipeline
94.3%

Synthesis accuracy. Coherent replacements, against 87.5% for the next best.

End-to-end de-identification

End-to-end pipeline score (NER accuracy + synthesis)

Tonic's Identity Synthesis pipeline
95.3%

End-to-end score. Highest of 7 pipelines, 5.2 points ahead of the best frontier LLM.

At a glance
21
workspace exports
18,040
messages
357
documents
130,370
labeled PII spans
7
pipelines

PrivacyBench is 21 workspace exports of email, Slack, and documents, with every PII span labeled by human annotators. Each of the 7 pipelines tested has to find that PII and replace it with consistent synthetic identities, and is scored against the public answer key.

Why Tonic beats frontier LLMs at end-to-end de-identification: PrivacyBench v2.5 results

By Joe Ferrara, PhD, Head of AI at Tonic.ai

PrivacyBench is Tonic.ai's open benchmark for de-identifying workspace data, part of the Tonic.ai AI model benchmark library. Since version 2, it has scored the whole job end to end. A pipeline receives raw .eml emails, a Slack workspace export, and PDF, Word, Excel, and CSV documents, then has to detect the PII, replace it with coherent synthetic values, and write the export back out in its original file formats. In the latest release, PrivacyBench v2.5, Tonic's data de-identification pipeline, which pairs Tonic Textual's PII detection capabilities with a proprietary approach to entity resolution and coherent synthesis, scored highest on every metric across seven pipelines and reached 95.3% on the combined detection-and-synthesis score, 5.2 points ahead of the best pure-LLM pipeline. Tonic also lifted the LLM it ran with, GPT-5.6 Sol, from 86.7% on its own to 95.3%, an 8.6-point gain.

21 workspace exports · 18,040 messages · 357 documents (PDF, DOCX, XLSX, CSV) · 10 entity types · 130,370 labeled PII spans · 7 pipelines tested · human-validated answer key

If you've seen earlier PrivacyBench results, v2.5 uses the same data as v2 with a corrected answer key and a rescaled combined score, so compare pipelines within v2.5 rather than against the v2 numbers. v1 is a different, easier test, and its scores aren't comparable with either. See what's new in PrivacyBench v2.5, what changed from PrivacyBench v1, and the v1 results further down this page.

What PrivacyBench measures, and why it goes end to end

Teams de-identify their email, Slack, and shared-drive exports so they can put that data to work: training and fine-tuning models, building agent reinforcement learning environments (RLE), licensing operational data to AI labs, and sharing data with partners. None of that works if real PII survives in the output, and none of it works if the replacements break the data.

That data never arrives as a tidy list of sentences. It arrives as an export: a mix of semi-structured files and free text. One person shows up as a display name in an email header, a handle in a Slack message, a name in the signature block of a Word document, and a row in a spreadsheet. A de-identification pipeline has to find every one of those mentions and map all of them onto a single synthetic identity, in every file. PrivacyBench scores the whole export together, so a pipeline that renames someone one way in the emails and another way in the documents is scored as incoherent.

Every pipeline has to complete four stages:

  1. Extract the text from raw export files, including base64-encoded email attachments.
  2. Detect the PII with named entity recognition (NER).
  3. Synthesize a coherent replacement for every detected entity.
  4. Render the synthetic export back out in the same file formats as the input.

A strong replacement has to hold four things together:

  • Cross-message consistency: every mention of one person maps to the same synthetic identity across every message.
  • Cross-file consistency: that same identity holds in the PDFs, Word documents, spreadsheets, and CSVs, not just in the messages.
  • Within-identity derivation: the synthetic email address, handle, phone number, and employee ID plausibly belong to the synthetic name, and a nickname maps to a nickname form of it.
  • Organization coherence: every surface form of one employer maps to a single synthetic organization, with dependent values like email domains, addresses, and URLs kept in step.

Existing NER benchmarks, like CoNLL, OntoNotes, and TAB, score detection well. None of them score the quality of the replacement, and none of them score it across a real export. That gap is what PrivacyBench fills.

Tonic Textual detects entities, links every mention to one individual, synthesizes consistently, and preserves each entity's type and format across email, Slack, and documents.
Tonic detects entities, links every mention to one individual, and replaces them consistently across every source, the same coherence PrivacyBench scores.

What makes a de-identification benchmark worth trusting

A benchmark result only helps you if the test resembles the job you need done and you can check the result yourself. PrivacyBench is built on both.

  • Real export formats, not flattened tables. Inputs are RFC 5322 .eml files with base64 attachments, a Slack workspace export, and a Google Drive-style file listing. Every labeled span is located in its file's native coordinates: byte offsets in an email, JSON pointers in a Slack day file, per-character boxes on a PDF page, cells in a spreadsheet.
  • Realistic narrative, not template fill. Tonic Fabricate generated a month of email and Slack activity for each of the 21 fictional protagonists, keeping one coherent storyline across the whole period. The supporting documents were generated from those storylines and are referenced in the conversations the way real work files are.
  • Linked and realistic at once. Every identity is linked across files for scoring, the unstructured-data equivalent of referential integrity, and the data still reads like a company doing its work: the kind of messy, cross-referenced text your own exports contain.
  • Deliberately hard attribution. The seeds include characters who share a name, characters with nicknames, and messages where only the surrounding conversation tells you which person is meant.
  • Scores that separate systems. Each of the three metrics spreads the pipelines tested: NER recall ranges from 55.5% to 96.4%, and synthesis accuracy from 77.5% to 94.3%. No metric is one that every system passes by construction.
  • An answer key humans checked. Annotators reviewed the generated labels for 6 of the 21 datasets and confirmed 95.8% of them exactly, then added the spans the generator missed so detection could be scored on precision, recall, and F1.
  • Current competition. The end-to-end comparison covers six agent pipelines built on frontier models from OpenAI and Anthropic, and the detection comparison adds two widely used open-source NER models, Microsoft Presidio and GLiNER2.
  • Open data, open answer key, open scoring code. The full dataset and ground truth are public on Hugging Face, the scoring code is open source on GitHub, and the Tonic Textual NER scores are reproducible with the Textual SDK.

Tonic.ai built this benchmark and the system that scores highest on it. That's the reason every piece of it is public: so you can score your own pipeline against the same data and compare directly with the results here. None of Tonic.ai’s models were trained on any of the data in PrivacyBench.

How the benchmark data was built

Every PrivacyBench dataset is fully synthetic. Tonic Fabricate generated each one from a seed that declares every character's PII before any text exists, which makes the ground truth a byproduct of generation rather than a hand-labeling project. That solves the two problems that make PII benchmarks expensive: a public benchmark can't contain real PII, and trustworthy labels are slow and costly to tag by hand.

Each of the 21 datasets is one fictional protagonist's workspace, set in a different industry, including pharma, retail, finance, airlines, tech, and manufacturing. Together they hold 18,040 messages (2,540 emails and 15,500 Slack messages) and 357 supporting documents, 17 per dataset: 10 PDFs, 3 Word documents, 2 Excel workbooks, and 2 CSVs. In 11 datasets the documents travel as email attachments and Slack file shares; in the other 10 they live in a Drive folder.

The answer key covers 130,370 PII spans across ten entity types: given name, family name, email address, username, organization, phone number, street address, employee ID, account number, and URL. Each span is tied to the character or organization it belongs to, which is what lets the benchmark check that one real identity maps to one synthetic identity.

How PrivacyBench scores de-identification: three metrics

PrivacyBench reports three scores: one for detection, one for the quality of the replacements, and one that combines both to score the entire pipeline. NER recall and synthesis accuracy are defined as they were in v1. v2.5 changes how the combined score is reported, so it sits on the same scale as the other two.

Privacy-bench metrics: what each one measures
Metric What it measures
NER recall Of the PII known to be in the data, how much the pipeline detected. A miss here is a privacy leak: real PII left in the output.
Synthesis accuracy Of the entities the pipeline detected, how many it replaced coherently. An LLM judge scores each replacement, per character, against the synthetic identity it should belong to.
Synthesis + NER accuracy The bottom-line score. Of the PII known to be in the data how much of it was synthesized coherently, then take the square root of that ratio. It counts detection misses and synthesis errors. The square root is a normalizer factor that makes this score the geometric mean of NER recall and synthesis accuracy.

Each of the first two scores hides something on its own. Recall is a privacy measure: it tells you how much PII was caught, but nothing about whether the de-identified data is still usable. Synthesis accuracy is a utility measure, and on its own it can be gamed: detect a handful of easy entities, replace them perfectly, and post a high number while the hard PII slips through. The combined score captures privacy and utility together, which is why it's the number to rank by.

Earlier versions reported the combined score as the product of NER recall and synthesis accuracy, which always landed below both. v2.5 reports the geometric mean, the square root of that product. The ranking and what the score means don't change, but the combined score now sits in the same range as the two scores it combines, so all three are easier to read side by side.

One message scored end to end. Detection misses and incoherent replacements both pull the combined score down, and it lands between the two stage scores. PrivacyBench computes the same metrics across entire exports.
One message scored end to end. Detection misses and incoherent replacements both pull the combined score down, and it lands between the two stage scores. PrivacyBench computes the same metrics across entire exports.

What the benchmark found

Tonic's data de-identification pipeline led all seven pipelines on every metric. It pairs Tonic Textual's NER with the pipeline's Identity Synthesis, a proprietary approach to entity resolution and coherent synthesis that links every mention of a person or organization across the export and replaces them consistently. The six other pipelines are pure-LLM agents: each model wrote its own extraction and rendering code in a sandbox, then performed NER and synthesis itself.

PrivacyBench v2.5 results: macro average across 21 datasets, ordered by the combined score

Privacy-bench results by pipeline: NER recall, synthesis accuracy, and combined synthesis + NER accuracy
Pipeline NER recall Synthesis accuracy Synthesis + NER accuracy
Textual Graph Synthesis 96.4% 94.3% 95.3%
GPT-6 Astra 92.9% 87.5% 90.1%
Claude Opus 5 95.1% 85.1% 89.9%
GPT-5.6 Sol 91.2% 82.4% 86.7%
Claude Sonnet 5 90.3% 79.6% 84.7%
GPT-5.6 Terra 86.1% 78.7% 82.2%
GPT-5.6 Luna 55.5% 77.5% 65.4%

Two results explain Tonic's lead.

Tonic's NER model is a small, efficient encoder-based model, not an LLM. That makes it much faster and cheaper to run than asking an LLM to find the PII, and on PrivacyBench it still detects PII better than the frontier models do. End to end, it caught slightly more of the known PII than any pure-LLM pipeline, and against the human annotations, Tonic posted the two highest F1 scores of any engine tested, as the detection results below show.

The even more definitive lead is in synthesis. Finding a person's name is one job. Giving that person the same new identity in every email, Slack message, and document they appear in is a harder one, and it's where the pure-LLM pipelines lose ground. Tonic's Identity Synthesis links every mention of a person or organization across the export before it replaces anything, so its replacements stayed coherent more often than any other pipeline's: 94.3% of the time, against 87.5% for the next best.

Put those two together and you get the clearest result in the benchmark. Tonic's de-identification pipeline used GPT-5.6 Sol for its LLM steps, which makes the GPT-5.6 Sol row a direct comparison: the same model, with and without Tonic. Running the whole job on its own, Sol scored 86.7% and finished third among the pure-LLM pipelines, behind GPT-6 Astra and Claude Opus 5. Inside Tonic's pipeline, the same model scored 95.3%, an 8.6-point gain that took it from third place to first. Nothing about the model changed; what changed was the system around it.

Fig. 1: Combined synthesis + NER accuracy across the seven pipelines, macro average over 21 datasets.
Fig. 1: Combined synthesis + NER accuracy across the seven pipelines, macro average over 21 datasets.

How Tonic's detection compares on human-annotated data

The end-to-end results measure recall against the generated answer key, which covers the seed characters and their organizations but not every name that appears in the data. To score detection completely, human annotators labeled every PII span in the emails, Slack messages, PDFs, and Word documents of 6 of the 21 datasets: 25,855 spans across 5,198 messages and 656 document pages. Against that human gold, precision and F1 become meaningful.

NER scores against the human annotations (overlap matching)

Privacy-bench NER detection quality by engine: precision, recall, and F1
NER engine Precision Recall F1
Textual Graph Synthesis 90.9% 94.3% 92.5%
Textual via SDK 91.3% 92.9% 92.1%
Claude Opus 5 92.1% 84.9% 88.3%
Claude Sonnet 5 92.1% 82.4% 87.0%
GPT-6 Astra 88.6% 85.0% 86.8%
GPT-5.6 Sol 90.2% 82.3% 86.1%
GPT-5.6 Terra 92.4% 78.4% 84.8%
Presidio 79.0% 87.1% 82.8%
GLiNER2 83.7% 68.0% 75.1%

Tonic's two engines posted the highest F1 scores of any tested, 92.5% for Tonic's de-identification pipeline and 92.1% for Tonic Textual via SDK, and they're the only engines above 90% on precision, recall, and F1 at once. Some LLMs scored up to 1.5 points higher on precision, but trailed both Tonic engines by 8 points or more on recall. For privacy, recall is the number that matters: a precision miss over-redacts something that wasn't PII, while a recall miss leaves real PII in your data. The two open-source engines trailed further, with Presidio at 82.8% F1 and GLiNER2 at 75.1%.

The Tonic NER models were not trained on any PrivacyBench data, and the Textual via SDK row is reproducible with a Textual API key and the open scoring code.

Fig. 2: NER F1 (overlap matching) against the human annotations, by engine.
Fig. 2: NER F1 (overlap matching) against the human annotations, by engine.

PrivacyBench v2.5 uses the same workspace exports as v2. What changed is how they're scored:

  • A corrected answer key. The generated labels were audited and improved, for a net 2,539 fewer spans (132,909 to 130,370). Almost all of the removed spans were organization labels that didn't belong. Location addresses are now labeled at the street level only, and a handful of missing given names and usernames were added.
  • A rescaled combined score. Synthesis + NER accuracy is now the geometric mean of NER recall and synthesis accuracy rather than their product, which puts it on the same scale as the other two scores.
  • A newer LLM judge. Synthesis accuracy is now judged by Claude Opus 5.5, which none of the tested pipelines used.
  • An improved Identity Synthesis. Improvements to the pipeline since v2 raised its NER recall and synthesis accuracy.
  • Every pipeline rescored. The LLM pipelines' outputs are unchanged from v2. All seven pipelines were rescored against the corrected answer key, with the new judge and the new combined score.

Because the answer key, the judge, and the combined score all changed, v2.5 scores aren't directly comparable with v2 scores. The v2 data and results remain available at the v2 revision of the dataset.

PrivacyBench v1 scored email and Slack messages supplied as pre-extracted text, across five entity types, with two-stage pipelines that ran NER and then synthesis. v2 keeps the v1 email and Slack messages, now in their raw export formats, along with the same detection and synthesis metrics, and turns the task into a full de-identification job:

  • Raw exports instead of extracted text, so pipelines do their own text extraction and rendering.
  • 357 documents in PDF, Word, Excel, and CSV formats, with ground truth located in each file's native coordinates.
  • Ten entity types instead of five, adding phone numbers, location addresses, employee IDs, account numbers, and URLs.
  • Human-annotated ground truth for six datasets, which makes full precision, recall, and F1 scoring possible for detection.
  • New pipelines: Tonic's data de-identification pipeline and six agent pipelines built on current frontier models.

Because the task is harder, v2 and v2.5 aren't comparable with v1. Comparing them is like comparing a 5K time to a trail-race time: both measure how fast you are, on very different ground. v1 also reports the combined score as a product, so its combined scores sit on a different scale from v2.5's. The v1 data and scoring code remain available at the v1 revision of the dataset and the v1 tag of the metrics repo. We've also re-run v1 on the current version of Tonic, so the v1 results below reflect how Tonic performs today.

We've re-run PrivacyBench v1 using Tonic's Identity Synthesis pipeline, which is used for PrivacyBench v2 and v2.5. At the time of publishing v1, Identity Synthesis was still in development.

On v1, Tonic reached 95.4% on the combined synthesis + NER score, with 98.2% NER recall. The original v1 pipelines appear below as first published, alongside the new run. v1 results use the original definition of the combined score, the product of NER recall and synthesis accuracy, so they aren't on the same scale as the v2.5 combined scores above.

PrivacyBench v1 results: the current Tonic run and the original v1 pipelines

Privacy-bench results by pipeline and run: NER recall, synthesis accuracy, and combined synthesis + NER accuracy
Pipeline Run NER recall Synthesis accuracy Synthesis + NER accuracy
Textual Graph Synthesis Textual Graph Synthesis, September 2026 98.2% 97.2% 95.4%
Textual NER + Opus 4.8 synthesis Original v1, July 2026 95.0% 97.0% 92.0%
Opus 4.8 only Original v1, July 2026 88.8% 99.0% 87.7%
Textual NER + Sonnet 4.6 synthesis Original v1, July 2026 95.0% 90.5% 85.9%
Textual NER + Haiku 4.5 synthesis Original v1, July 2026 95.0% 87.3% 82.8%
Sonnet 4.6 only Original v1, July 2026 83.8% 90.5% 76.2%
Haiku 4.5 only Original v1, July 2026 79.9% 88.5% 71.1%

The Opus-only pipeline still posts the highest raw synthesis accuracy on v1, 99.0%, but only because its lower recall left it replacing the easier entities it caught. With detection misses counted, its combined score is 87.7%.

PrivacyBench v1: combined synthesis + NER accuracy for the current Textual run and the original v1 pipelines.
PrivacyBench v1: combined accuracy for the current Tonic run and the original v1 pipelines.

Detection on v1's human-annotated messages

Human annotators fully labeled the email and Slack messages in 6 of the 21 datasets. That makes it possible to score v1 detection on precision, recall, and F1, rather than recall alone. These are the same 6 datasets used in v2. 

NER scores against the v1 human annotations for overlap metric

Privacy-bench NER detection quality by engine: precision, recall, and F1
NER engine Precision Recall F1
Textual Graph Synthesis 95.8% 96.3% 96.0%
Textual via SDK 96.5% 92.6% 94.5%
Opus 4.8 (LLM NER) 95.4% 88.2% 91.7%
Sonnet 4.6 (LLM NER) 97.0% 85.0% 90.6%
Presidio 87.3% 89.0% 88.1%
Haiku 4.5 (LLM NER) 97.6% 79.9% 87.8%
GLiNER2 85.2% 88.6% 86.9%

Tonic is the clear winner for overall NER when precision, recall, and F1 are all considered across the six datasets. For both scoring methods, Tonic's precision, recall, and F1 are all over 90%. It is the only model that achieves over 90% for all three scores, and it's the only model that achieves over 90% recall, which it does for both scoring methods. There are other models that score similarly to Tonic on precision, but Tonic wins by a lot on recall in those cases. Recall is the more important metric when considering privacy as it measures privacy leaks.

Put your own de-identification pipeline to the test

If you're de-identifying workspace data for AI, PrivacyBench v2.5 points one way: frontier LLMs working alone trail on keeping identities coherent across a full export, and Tonic's Identity Synthesis pipeline closes that gap. The combined score, the same-model gain, and the detection results against human annotations all point the same direction.

The strongest case is the one you run yourself. Point your pipeline at the 21 public workspace exports, score it with the open metrics code, and compare your numbers with the ones here.

Tonic's Identity Synthesis pipeline is the engine behind Tonic.ai's company data de-identification work, which prepares operational data like email, chat, tickets, and documents for licensing and model training. For the principles behind that work, read the core tenets of de-identification for AI data licensing. To scope out a workspace data de-identification project with our team, book a demo.

Frequently asked questions

PrivacyBench is Tonic.ai's open benchmark for de-identifying workspace data exports: email, Slack, and shared-drive documents. It scores both halves of de-identification, detecting the PII and replacing it with coherent synthetic values, and combines them into a single end-to-end score. The datasets are public on Hugging Face and the scoring code is open source on GitHub.

v2 turns PrivacyBench into a full de-identification job. Pipelines now receive raw export files instead of extracted text, handle PDF, Word, Excel, and CSV documents alongside email and Slack messages, detect ten entity types instead of five, and write the synthetic export back out in its original formats. v2 also adds human-annotated ground truth for six datasets.

v2.5 keeps the v2 data and changes how it's scored. The answer key was audited and corrected, the combined score is now the geometric mean of NER recall and synthesis accuracy rather than their product, and a newer LLM judge scores synthesis. Every pipeline was rescored under the new rules, and Tonic's Identity Synthesis pipeline was re-run with recent improvements.

No. v2 is a harder task with different data, and v2.5 also calculates the combined score differently, so the versions aren't comparable. Compare pipelines within a version, not across versions.

Yes. Against the human annotations, Tonic posted the highest F1 of any engine tested, 92.5%, ahead of every frontier LLM and both open-source engines. Some LLMs scored slightly higher on precision but 8 or more points lower on recall, and recall is what determines whether real PII stays in your data.

Synthesis accuracy measures replacement quality: of the PII a pipeline detected and replaced, how much is coherent? An LLM judge checks each replacement, per character, against the synthetic identity it should belong to, so a nickname has to map to a nickname form of the same synthetic name, and a synthetic email address has to belong to the synthetic person.

Yes. The 21 workspace exports and their answer keys are public on Hugging Face, and the scoring code is on GitHub. Run your pipeline on the raw exports, write its replacements in the benchmark's output format, and score them against the same ground truth and metrics used here.

Joe Ferrara is Head of AI at Tonic.ai, where he uses the latest developments in artificial intelligence to improve named entity recognition and synthetic data generation in Tonic Textual. He holds a Ph.D. in Mathematics from the University of California, Santa Cruz, and a B.A. in Mathematics from the University of California, Berkeley. Prior to his role at Tonic.ai, Joe served as a Data Scientist at ICW Group, where he gained experience applying traditional data science techniques in the context of the insurance industry. His background in theoretical mathematics research gives him a unique perspective on his work in artificial intelligence and machine learning.