Sensitive text identification: Industry landscape and performance benchmarking
A data-driven benchmark of leading text de-identification approaches across legal, EHR, and call-transcript data, reporting precision, recall, and F1 under apples-to-apples conditions.
Modern AI runs on text, but sensitive data in unstructured content is messy, contextualized, and prone to error when manually de-identified, which can turn privacy into a primary bottleneck for AI adoption. This whitepaper quantifies the problem and the opportunity: we benchmark Tonic Textual against widely used open-source frameworks and cloud services across three real-world domains (Legal, EHR, Call Transcripts), report Precision/Recall/F1 with apples-to-apples adjusted comparisons, and show how pressure-tested de-identification both reduces regulatory risk and unlocks high-value datasets for model training and downstream use.
Executive summary
Tonic Textual delivers compliance-grade sensitive text identification across industries. In October 2025 benchmarks against AWS Comprehend, Azure AI Language, Google Cloud Sensitive Data Protection (DLP), Microsoft Presidio, GLiNER, and spaCy, Textual achieved near-perfect F1 0.96 / Precision 0.97 / Recall 0.96 on aggregate and maintained top performance across Legal, EHR (medical), and Call Transcript datasets—even when its entity set was constrained to match each competitor's. Alternatives showed domain variability, lower precision (over-redaction), and/or low recall (missed identifiers), reinforcing that general-purpose NER and rule systems are ill-suited for compliance-driven de-identification at scale.
.png)
The challenge: Identifying sensitive data in text
In the current era of AI, the task of identifying sensitive or private data within text may sound relatively straightforward; however unstructured text persists as a challenging domain for analysts and AI/ML developers alike due to the infinite nuances that can exist within this medium. Unlike structured data, where fields are explicitly labeled, unstructured text embeds personal information within natural language, making it highly contextual and ambiguous.
Adding to the complexity is the reality that a name, date, or location might appear in a dozen different formats, or even hold multiple meanings depending on the surrounding text. For example, the word "Washington" could refer to a person, a place, or an institution. Traditional approaches like pattern matching or regular expressions can't reliably capture these nuances, while machine learning models require careful tuning, continuous retraining, and high-quality labeled data to avoid false positives or missed detections. Complicating matters further are quasi-identifiers: data points like age, ZIP code, or occupation which, when combined, can reveal identity even if direct identifiers are removed. These subtleties make text-based identification an inherently hard problem to automate or scale.
'BYO' Data identification
Building an in-house solution to this challenge places a heavy burden on engineering and compliance teams.
The initial setup alone requires a manual audit across every data source, cataloging potential identifiers, and designing a detection framework capable of parsing multiple data types and formats. This initial stage is not just a technical exercise — it's a form of data discovery and taxonomy building that establishes the foundation for all subsequent automation. It's time-consuming, requires cross-functional collaboration, and demands both domain expertise and contextual understanding of how sensitive data manifests within an organization's unique workflows.
From there, teams must constantly update detection rules to account for schema drift, new document types, and evolving regulatory interpretations. False negatives risk exposure and legal liability; false positives degrade data utility, reducing the quality of downstream analytics, testing, and AI model training. Achieving an effective balance between data privacy and usability is rarely static and requires ongoing iteration, validation, and documentation. Meanwhile, the compute costs of scanning and processing large text corpora grow quickly, often demanding specialized infrastructure and expertise.
Identification and de-identification at scale
These challenges compound as organizations scale their data operations and enter regulated domains. What appears initially as a contained effort to build a text-detection solution can quickly expand into a sprawling compliance and engineering effort. As datasets grow in size and variance — spanning different document types like chat logs, emails, clinical notes, contracts, and customer interactions — the number of edge cases multiplies. Even if each document is individually de-identified, the potential for re-identification through data linkage or inference rises dramatically. Large enterprises also face the added burden of proving compliance to regulators under frameworks like HIPAA and GDPR, which require not only the protection of personal data but also demonstrable auditability and risk assessment. Maintaining this standard across petabytes of text and multiple jurisdictions becomes a full-time endeavor, one that most internal teams are not equipped to sustain long-term.
Ultimately, identifying sensitive information within text is not just a technical challenge; it's also a challenge with respect to operations and compliance. The intersection of linguistic variability, regulatory scrutiny, and scale means that ad hoc or homegrown systems rarely keep pace.
Quantifying de-identification
Measuring performance
In text de-identification solutions, performance is primarily defined by how accurately and consistently the system can identify and redact sensitive entities without over- or under-masking the underlying text. The three most common metrics used to measure this performance are Precision, Recall, and F1 Score, which together provide a balanced view of both the accuracy and completeness of detection. These metrics provide a clear, measurable way to evaluate how well a de-identification system can find and protect the sensitive information that's hidden within everyday text.
Precision
Precision measures the proportion of detected entities that were actually sensitive. A high precision score indicates that when the model flags a token or phrase as sensitive, it's usually correct—meaning the system generates few false positives. This is especially important in domains like healthcare and legal tech, where over-redaction can remove useful context, degrade data utility, and hinder downstream analytics. However, a solution that focuses solely on maximizing precision might miss subtle or ambiguous identifiers, introducing compliance risk.
Recall
Recall, on the other hand, measures the proportion of all true sensitive entities that the model successfully identified. High recall means the system is comprehensive—it captures the majority of sensitive instances, minimizing false negatives. In regulatory settings such as HIPAA or GDPR compliance, strong recall is critical, as failing to detect even a small number of identifiers can lead to data breaches or legal exposure. Yet, recall and precision often exist in tension: as recall improves, precision can drop, and vice versa. Effective de-identification systems must therefore optimize this balance.
F1 score
The F1 Score combines both precision and recall into a single harmonic mean, offering a unified measure of model performance. It's particularly useful when the cost of false positives and false negatives are both significant—such as in privacy-preserving data pipelines where utility and compliance must coexist. In practice, evaluating F1 scores across diverse document types, entity classes, and real-world datasets gives a more realistic picture of how a de-identification solution will perform in production. Ultimately, performance in text de-identification isn't just about hitting statistical benchmarks—it's about achieving the right equilibrium between protection and usability, ensuring sensitive data is safeguarded while retaining the context and fidelity that make it valuable for analysis, testing, or AI model training.
Putting Tonic Textual to the test
We benchmarked Tonic Textual against AWS Comprehend, Azure AI Language, Google Cloud Sensitive Data Protection (DLP), Microsoft Presidio, GLiNER, and spaCy. Tests spanned Legal, EHR, and Call Transcripts to capture domain-specific nuances—formal legal phrasing, clinical shorthand, and conversational irregularities. For each system and dataset, we computed Precision, Recall, F1. We then ran adjusted comparisons, restricting Textual's entities to match each competitor. Across all evaluations, Textual led on every metric and domain, preserving context while maximizing coverage—evidence of domain adaptability rather than taxonomy advantage.
AWS Comprehend
Amazon's NLP service offering entity recognition, sentiment analysis, and key phrase extraction, including limited support for identifying PII through its "Detect PII Entities" API.
Azure AI Language
A Microsoft cloud service that provides text analytics, entity recognition, and PII detection as part of its general language understanding suite.
Google Cloud Sensitive Data Protection (DLP)
A GCP service that scans for, classifies, and masks sensitive information across text and structured data, leveraging prebuilt detectors.
Microsoft Presidio
An open-source framework for detecting and anonymizing PII in text and structured data, using a combination of regex rules, NLP models, and customizable recognizers.
GLiNER
An open-source lightweight NER model that uses transformer-based architectures for general entity recognition but lacks domain-specific tuning for privacy compliance tasks.
spaCy
A popular open-source NLP library offering entity recognition and linguistic processing tools; not designed specifically for compliance workflows or privacy-driven use cases.
To evaluate the performance of text de-identification systems across real-world use cases, we conducted a comprehensive benchmarking experiment comparing Tonic Textual against both open-source frameworks and commercially available APIs for named entity recognition (NER) and data redaction. Internal or "build-your-own" (BYO) de-identification solutions often rely on these same models—many of which were designed for general-purpose language tasks rather than compliance-grade privacy workflows.
Each system was tested under consistent experimental conditions to ensure objective, reproducible comparison. The evaluation was conducted across three domain-specific datasets representing key regulated industries—Electronic Health Records (EHR), legal documentation, and customer service call transcripts—each chosen for its unique linguistic complexity and contextual variability.
For every system tested, we computed Precision, Recall, and F1 Score independently for each dataset. These metrics provided a clear, quantitative measure of how accurately and completely each model could identify and redact sensitive entities. In addition, we ran adjusted comparisons, aligning Tonic Textual's entity set to match the limited scope of each competitor, ensuring that performance differences reflected genuine model capability rather than differences in entity coverage.
Across all evaluations, Tonic Textual consistently achieved the highest scores in every metric and dataset, demonstrating exceptional precision, recall, and overall balance. This consistency underscores Textual's ability to generalize across unstructured, domain-specific text while maintaining compliance-grade accuracy and preserving downstream data utility.
Results: Tonic vs. Popular NER Models
The results presented in this section summarize the findings of our benchmarking experiment conducted in October 2025, evaluating the performance of leading text de-identification models against Tonic Textual, including open source options and commercial APIs. Each system was tested across a series of standardized datasets representing healthcare, legal, and customer service domains to assess real-world applicability and robustness. The outcomes reflect the capabilities of these models as of this date, recognizing that model architectures and APIs may evolve over time. Our goal is to provide an objective, time-bound snapshot of how widely used NER frameworks—such as AWS Comprehend, Microsoft Presidio, and GLiNER—compare to a purpose-built commercial solution in terms of accuracy, coverage, and consistency when applied to sensitive, unstructured text.
When reviewing the results that follow, readers should interpret each model's Precision, Recall, and F1 scores as quantitative indicators of de-identification performance. These scores are computed at the level of words, where we consider a word to be correctly predicted if it is identified as PII with the correct type. All metrics are reported as decimal values rounded to the nearest hundredth (e.g., 0.92), where higher scores indicate stronger performance. A higher Precision score means the model more accurately identifies only true sensitive entities, minimizing false positives and preserving data utility. A higher Recall score reflects the model's ability to capture a greater proportion of all true sensitive entities, reducing the likelihood of missed detections or potential privacy leaks. The F1 score, representing the harmonic mean of Precision and Recall, provides a balanced measure of overall accuracy—capturing both correctness and completeness in a single value. Together, these metrics offer a comprehensive view of how effectively each system identifies and redacts sensitive information across varying text types and domains.
Tonic Textual vs. AWS Comprehend
In this comparison between Tonic Textual and AWS Comprehend, Tonic Textual once again demonstrates a clear advantage across all key performance metrics. With an F1 score of 0.96, it achieves a stronger overall balance between Precision and Recall than AWS Comprehend, which recorded 0.88. Tonic Textual's Precision score of 0.97 indicates that nearly every entity it flags as sensitive truly is—resulting in fewer false positives and more trustworthy anonymization. Its Recall score of 0.96 shows that it successfully detects nearly all true sensitive entities, minimizing the risk of unredacted PII or PHI remaining in the data.
By contrast, AWS Comprehend trails in both dimensions, with a Precision of 0.87 and Recall of 0.90, reflecting a less consistent ability to identify sensitive information without over- or under-redacting. While Comprehend performs reasonably well in general entity recognition, its lower scores suggest it may struggle with domain-specific or contextually nuanced identifiers—particularly in unstructured or industry-regulated text. Overall, these results indicate that Tonic Textual not only provides higher accuracy but also maintains stronger reliability across diverse datasets, making it a more effective and compliance-ready solution for enterprise-scale de-identification.
Tonic Textual vs. GLiNER
GLiNER is an impressive model, bringing the 0-shot capabilities of LLMs to the highly performant BERT architecture. However, because it is trained on synthetic labels on webcrawl data, it does not perform nearly as well as models trained on carefully annotated, task specific data. In the aggregate evaluation across all datasets, Tonic Textual significantly outperformed GLiNER, underscoring the gap between purpose-built de-identification models and general-purpose open-source NER frameworks. Tonic Textual achieved an F1 score of 0.96, with Precision 0.97 and Recall 0.96, demonstrating both exceptional accuracy and comprehensive coverage in detecting sensitive entities. This balance reflects a system finely tuned to capture nearly all relevant identifiers while minimizing false positives, preserving the contextual integrity of text for downstream use.
By contrast, GLiNER's performance fell sharply short, with an F1 of 0.32, Precision 0.68, and Recall 0.20. While it occasionally identifies true positives correctly, its low Recall indicates that the majority of sensitive entities went undetected—posing clear compliance and privacy risks in real-world scenarios. The disparity between GLiNER's modest precision and extremely low recall highlights its lack of robustness when applied to domain-specific, unstructured text.
Overall, these results emphasize that effective text de-identification requires more than general entity recognition. Tonic Textual's domain-aware architecture and privacy-focused training enable it to consistently achieve compliance-grade detection accuracy where open-source NER tools like GLiNER struggle to scale beyond basic recognition tasks.
Tonic Textual vs. Google Cloud: Sensitive Data Protection
When evaluated against Google Cloud's Sensitive Data Protection (GCP_SDP) service, Tonic Textual again demonstrated a decisive performance advantage across all key metrics. With an F1 score of 0.96, Precision of 0.97, and Recall of 0.96, Textual achieved near-perfect accuracy and balance, effectively identifying and redacting sensitive entities while maintaining data integrity. These results highlight Textual's ability to generalize across heterogeneous text sources and industries without compromising compliance or usability—delivering precision-grade detection alongside comprehensive coverage.
By contrast, GCP_SDP achieved an F1 score of 0.61, with Precision at 0.58 and Recall at 0.64. While this performance indicates some capability in recognizing sensitive information, it falls short in both accuracy and consistency, with a tendency toward false positives and incomplete detection. The lower precision suggests that GCP_SDP frequently misclassifies non-sensitive tokens as sensitive, which can erode data utility and hinder downstream analytics. Similarly, its modest recall reveals that a significant portion of sensitive entities remains undetected, posing privacy and compliance risks in regulated contexts.
These results underscore the limitations of multi-purpose cloud services that include de-identification as a secondary feature rather than a core function. In contrast, Tonic Textual's specialized model architecture and domain-tuned training deliver a consistent, compliance-grade standard of accuracy that far exceeds what general-purpose detection tools can provide.
Tonic Textual vs. Microsoft Presidio
Presidio is an open source framework for developing detection and anonymization tools, with many built-in detection types. It allows users to easily extend built-in detection, but in this evaluation we consider only the built-in types offered by Presidio. In general, the regex-based detection patterns will fall short in contexts where PII is not cleanly identifiable in text, like ASR transcripts, or where context is determinative of entity type. For example, a four digit number could represent the last four digits of a social security number, or it could represent an innocuous quantity. In these situations, which is typical of most privacy applications, one should likely rely on more powerful, contextually aware NER models. In the comparison between Tonic Textual and Microsoft Presidio, the results show a substantial performance advantage for Tonic Textual across all measured dimensions. With an F1 score of 0.96, Tonic Textual achieves near-perfect balance between accuracy and completeness in sensitive-entity detection, while Presidio trails significantly at 0.57. Tonic Textual's Precision of 0.97 demonstrates a strong ability to correctly identify true sensitive entities while minimizing false positives—ensuring data remains useful after de-identification. Its Recall of 0.96 further reflects consistent coverage, successfully identifying nearly all sensitive elements across the evaluated datasets.
Microsoft Presidio's performance, by contrast, reveals key trade-offs inherent in more generalized open-source systems. While its Recall score of 0.73 indicates that it detects a reasonable portion of sensitive data, the much lower Precision score of 0.47 suggests a high rate of over-redaction—flagging many non-sensitive tokens as sensitive. This imbalance diminishes data quality and usability, especially in scenarios requiring contextual preservation such as AI training or analytics.
Overall, the results emphasize that Tonic Textual offers not just greater detection accuracy, but greater reliability across real-world use cases, maintaining both compliance-grade privacy and practical data fidelity. Presidio's lower precision and moderate recall highlight the challenges of relying on generalized, rule-based frameworks in environments where linguistic variability and regulatory standards demand far more nuanced performance.
Tonic Textual vs. Azure AI Language
Across the aggregate evaluation, Tonic Textual maintained a commanding lead over Azure AI Language, reaffirming its position as a purpose-built solution for privacy-preserving text de-identification. Textual delivered an F1 of 0.96, with Precision 0.97 and Recall 0.96, demonstrating near-perfect balance: it captures almost all true identifiers while minimizing false positives—critical for preserving downstream data utility in analytics and AI development.
By contrast, Azure AI Language reached an F1 of 0.61, with Precision 0.52 and Recall 0.75. The relatively higher recall indicates Azure finds a fair share of sensitive entities, but its low precision signals frequent over-redaction—misclassifying non-sensitive text as sensitive—eroding context and reducing dataset usefulness. This precision–recall imbalance highlights the difficulty general-purpose language services face in compliance-grade de-identification, where nuanced domain adaptation and tight false-positive control are essential.
Overall, the results underscore the gap between multipurpose NLP services and specialized de-identification platforms. Where Azure provides broad language capabilities, Tonic Textual is tuned for compliance-grade accuracy across unstructured data, consistently balancing privacy protection with the preservation of analytical value—making it the more reliable, scalable, and audit-ready choice for enterprise deployments.
Tonic Textual vs. spaCy
spaCy is one of the most popular open source NLP solutions, and for good reason. It offers easy-to-use text parsing and NER with competitive performance on NER benchmarks like Ontonotes. Unlike GLiNER, the training data for spaCy consists of the high quality, human annotated Ontonotes corpus. Unfortunately, this means spaCy is limited in both the types that it can detect (e.g. payment information) and the contexts in which it can detect. For example, spaCy performs poorly at detection in ASR transcripts or messily formatted EHR records.
Across all datasets evaluated, Tonic Textual significantly outperformed spaCy, demonstrating both higher accuracy and greater reliability in detecting sensitive information. Textual achieved an F1 score of 0.96, reflecting a nearly perfect balance between Precision (0.97) and Recall (0.96)—indicating that it successfully identified almost all true sensitive entities while minimizing false positives. In contrast, spaCy recorded a considerably lower F1 score of 0.64, with Precision at 0.54 and Recall at 0.77, suggesting that while it was able to capture a moderate portion of sensitive entities, it frequently misclassified non-sensitive text as sensitive or failed to generalize across different data domains.
These findings underscore the advantage of domain-tuned, privacy-specific models over general-purpose NER frameworks. While spaCy provides a flexible foundation for entity recognition, it was not designed for compliance-driven de-identification tasks that demand both contextual understanding and fine-grained control. The performance gap—most evident in spaCy's low Precision—illustrates how off-the-shelf NER systems tend to over-redact, leading to degraded data utility and loss of valuable context.
Tonic Textual's consistency across healthcare, legal, and conversational datasets highlights its ability to accurately isolate personal information while preserving the integrity of the surrounding data—an essential capability for organizations working to maintain privacy compliance without sacrificing analytical value.
Benchmark conclusion
Across all benchmark comparisons, Tonic Textual consistently outperformed the leading text de-identification and sensitive data detection frameworks—AWS Comprehend, Azure AI Language, Google Cloud Sensitive Data Protection (GCP_SDP), GLiNER, Microsoft Presidio, and spaCy—in both accuracy and reliability. Achieving near-perfect scores in F1 (0.96), Precision (0.97), and Recall (0.96). Azure AI Language showed imbalance between recall (0.75) and precision (0.52), indicating that while it was able to detect a fair portion of sensitive entities, it frequently over-redacted or misclassified non-sensitive information; compromising both data utility and consistency across domains. Tonic Textual demonstrated an unmatched balance between comprehensive detection and minimal false positives, ensuring sensitive entities are accurately identified without degrading analytical value or downstream usability.
In contrast, alternative models showed substantial performance gaps. GLiNER (F1 0.32) exhibited poor recall and inconsistent accuracy, missing most sensitive entities. GCP_SDP, though stronger than Azure and GLiNER, reached only an F1 of 0.61 with moderate recall (0.64) and low precision (0.58), underscoring the challenges of adapting general-purpose cloud tools to compliance-grade de-identification. Microsoft Presidio and spaCy likewise struggled to balance accuracy and context preservation, over-redacting non-sensitive information while failing to capture all true identifiers.
Taken together, these findings reaffirm Tonic Textual's position as the industry benchmark for text de-identification performance. Its consistent results across multiple domains—including healthcare, legal, and customer communications—demonstrate that Textual uniquely combines precision, recall, and scalability to deliver production-ready, audit-defensible privacy protection. Built expressly for enterprise environments, it enables organizations to unlock sensitive data safely for AI, analytics, and innovation—achieving compliance without compromise.
Datasets
Industry-specific datasets
While aggregate benchmarks provide a useful measure of overall performance, real-world de-identification demands precision across vastly different data domains—each with its own linguistic patterns, formatting quirks, and regulatory requirements. To capture these nuances, we conducted targeted evaluations of Tonic Textual and competing frameworks across three distinct vertical datasets: legal documents, medical records (EHRs), and customer service call transcripts. Each of these domains presents unique challenges for entity recognition and redaction, from the formal and context-dependent phrasing of legal language, to the dense abbreviations and inconsistent formatting of clinical text, to the informal and fragmented nature of conversational dialogue.
The goal of these tests was to demonstrate not only how performance metrics vary depending on data type, but also how consistently each system can adapt to the contextual differences that define each industry. A model that performs well in structured or formulaic data may falter when confronted with colloquial speech or domain-specific terminology. By evaluating each framework within these vertical contexts, we sought to highlight the importance of domain adaptability—showing where general-purpose tools struggle and where Tonic Textual's domain-aware approach maintains high precision, recall, and overall F1 performance across use cases.
Legal Documents
The legal document dataset included a diverse mix of court filings, legal notices, and transcript excerpts—texts rich with personally identifiable information (PII) such as names, case numbers, dates, and contact details, often embedded within complex and formal sentence structures. This dataset was chosen to test each model's ability to detect sensitive data within long-form, domain-specific language that presents overlapping entities and nuanced phrasing. The goal was to assess not only overall accuracy but also how well each system balanced comprehensive coverage (recall) with precision, ensuring redaction quality without over-sanitizing valuable legal context.
Across every metric—F1, Precision, and Recall—Tonic Textual delivered best-in-class results, achieving near-perfect scores of 0.97 across the board. Even when adjusted to match the supported entity sets of other models, Textual's performance remained consistently high, occasionally improving further (e.g., 0.98 F1 when aligned to Presidio's narrower taxonomy). This stability underscores Textual's ability to maintain compliance-grade accuracy regardless of domain scope or entity definitions.
In contrast, other models displayed significant variability: AWS Comprehend performed reliably (F1: 0.90, Precision: 0.86, Recall: 0.94), but struggled with contextually complex identifiers; GCP_DLP performed moderately well (F1: 0.79, Precision: 0.87, Recall: 0.73), showing decent coverage but more frequent false positives; and Azure AI Language (F1: 0.84, Precision: 0.78, Recall: 0.91) exhibited strong recall but lower precision, leading to over-redaction.
Open-source frameworks—GLiNER, Microsoft Presidio, and spaCy—lagged considerably behind. GLiNER (F1: 0.44) and Presidio (F1: 0.63) both demonstrated inconsistent detection, either missing a large share of sensitive entities or over-redacting non-sensitive terms, while spaCy performed poorest overall (F1: 0.52, Precision: 0.38, Recall: 0.84), confirming that general-purpose NER models are ill-suited for compliance-driven tasks. Even when Tonic Textual was adjusted to match the reduced entity support of these frameworks, it retained superior performance (F1 ≥ 0.97), reinforcing that its precision and coverage stem from deeper domain adaptability rather than entity overlap alone. Taken together, these findings establish Tonic Textual as a robust, high-fidelity solution for de-identifying legal text—capable of maintaining near-perfect balance between privacy protection and data utility where competing solutions falter.
| System | F1 | Precision | Recall |
|---|---|---|---|
| Tonic Textual | .97 | .97 | .97 |
| AWS Comprehend | .90 | .86 | .94 |
| Textual (adjusted to match AWS Comprehend) | .97 | .97 | .97 |
| GLiNER | .44 | .42 | .47 |
| Textual (adjusted to match GLiNER) | .97 | .97 | .97 |
| GCP DLP | .79 | .87 | .73 |
| Textual (adjusted to match DLP) | .97 | .97 | .97 |
| Microsoft Presidio | .63 | .57 | .72 |
| Textual (adjusted to match Presidio) | .98 | .98 | .98 |
| Azure | .84 | .78 | .91 |
| Textual (adjusted to match Azure) | .97 | .97 | .97 |
| spaCy | .52 | .38 | .84 |
| Textual (adjusted to match spaCy) | .97 | .97 | .97 |
Medical Records (EHR)
The medical records dataset contained a challenging mix of clinical notes, discharge summaries, and physician correspondence—text types characterized by dense jargon, inconsistent formatting, and embedded identifiers such as patient names, medical record numbers, and provider details. This dataset was chosen to evaluate how well each system could navigate the unique linguistic patterns and abbreviations found in electronic health records (EHRs), while balancing privacy protection with the preservation of analytical and clinical value.
Tonic Textual achieved best-in-class performance across all key metrics, with an F1 of 0.97, Precision of 0.98, and Recall of 0.96, demonstrating its ability to accurately identify nearly all sensitive entities without excessive over-redaction. Even when its entity scope was adjusted to match those of competing models, Textual maintained near-identical results—occasionally improving further (e.g., F1 of 0.98 when matched to Presidio). This stability across entity definitions underscores Textual's precision-tuned architecture and domain adaptability, allowing it to operate with compliance-grade accuracy across highly variable medical text.
By contrast, other systems exhibited wide variability in performance. AWS Comprehend performed relatively well (F1: 0.89), but with lower precision and recall than Textual, indicating occasional missed identifiers and over-redactions. GCP_DLP (F1: 0.61) and Microsoft Presidio (F1: 0.60) struggled with incomplete detection and high false-positive rates, while Azure AI Language improved slightly over its earlier baseline but still lagged substantially behind, reaching F1: 0.62, Precision: 0.53, and Recall: 0.77—capturing more sensitive entities but at the expense of accuracy and consistency. GLiNER remained the weakest performer (F1: 0.30), while spaCy, though more stable than GLiNER or Azure, achieved only F1: 0.69, reflecting its limitations as a general-purpose NER model not tailored for clinical data.
In all adjusted tests, Tonic Textual's performance remained steady or improved, confirming that its strength lies not merely in entity breadth but in its contextual precision—delivering the rare balance of privacy, accuracy, and data utility that medical applications demand. Whether applied to complex diagnostic narratives or shorthand physician notes, Textual consistently demonstrated the robustness and reliability required for privacy-preserving analytics and AI in healthcare.
| System | F1 | Precision | Recall |
|---|---|---|---|
| Tonic Textual | .97 | .98 | .96 |
| AWS Comprehend | .89 | .88 | .90 |
| Textual (adjusted to match AWS Comprehend) | .97 | .98 | .96 |
| GLiNER | .30 | .80 | .19 |
| Textual (adjusted to match GLiNER) | .97 | .98 | .96 |
| GCP DLP | .61 | .57 | .66 |
| Textual (adjusted to match DLP) | .97 | .98 | .96 |
| Microsoft Presidio | .60 | .48 | .80 |
| Textual (adjusted to match Presidio) | .98 | .98 | .98 |
| Azure | .62 | .53 | .77 |
| Textual (adjusted to match Azure) | .97 | .98 | .96 |
| spaCy | .69 | .60 | .82 |
| Textual (adjusted to match spaCy) | .97 | .99 | .96 |
Customer Service Call Transcripts
The customer service call transcript dataset presented one of the most demanding environments for sensitive data detection. Unlike structured or formal text, conversational data is highly variable—featuring informal language, speech disfluencies, and inconsistent formatting. This domain tests a model's ability to interpret and redact sensitive information in natural, free-flowing dialogue, such as names, phone numbers, or email addresses interwoven into casual exchanges.
Tonic Textual once again achieved the highest performance across all key metrics, with an F1 of 0.93, Precision of 0.91, and Recall of 0.94. These results show that Textual effectively identifies sensitive entities in noisy conversational data while maintaining the context and readability needed for downstream analytics and AI training. Even when adjusted to match the narrower entity sets supported by competing systems, Textual's results remained consistent (F1 scores between 0.92 and 0.94), underscoring its ability to adapt to variable data schemas without loss of accuracy.
In contrast, the competing models demonstrated significant limitations in this complex domain. AWS Comprehend ranked second (F1 0.86 / P 0.83 / R 0.90), performing reasonably well but less consistently on informal phrasing. GCP DLP (F1 0.54 / P 0.57 / R 0.52) and Azure AI Language (F1 0.52 / P 0.43 / R 0.65) struggled to maintain balance—either missing sensitive entities or over-redacting large portions of text. Microsoft Presidio (F1 0.43) and spaCy (F1 0.48) both showed low precision and recall, while GLiNER (F1 0.31 / R 0.21) failed to capture the majority of true identifiers. These gaps illustrate how general-purpose or rule-based models often falter when faced with the unpredictability of real-world dialogue.
Taken together, these results highlight Tonic Textual's superior ability to handle the linguistic irregularities of human speech. Its balanced precision and recall enable enterprises to safely analyze or train models on customer interaction data—without sacrificing compliance or context fidelity. Textual's consistency across adjusted and full-entity tests confirms its position as a production-ready solution for de-identifying conversational data at enterprise scale.
| System | F1 | Precision | Recall |
|---|---|---|---|
| Tonic Textual | .93 | .91 | .94 |
| AWS Comprehend | .86 | .83 | .90 |
| Textual (adjusted to match AWS Comprehend) | .92 | .90 | .95 |
| GLiNER | .31 | .55 | .21 |
| Textual (adjusted to match GLiNER) | .93 | .91 | .95 |
| GCP DLP | .54 | .57 | .52 |
| Textual (adjusted to match DLP) | .92 | .90 | .94 |
| Microsoft Presidio | .43 | .40 | .46 |
| Textual (adjusted to match Presidio) | .94 | .94 | .94 |
| Azure | .52 | .43 | .65 |
| Textual (adjusted to match Azure) | .92 | .90 | .94 |
| spaCy | .48 | .41 | .58 |
| Textual (adjusted to match spaCy) | .93 | .91 | .95 |
Datasets conclusion
The customer service call transcript dataset represented a uniquely challenging environment for entity detection and de-identification. Unlike structured text, these conversational transcripts feature informal speech patterns, inconsistent grammar, and overlapping dialogue, where identifiers such as names, phone numbers, or account details appear irregularly and without standard formatting. This dataset tested each system's ability to interpret sensitive information within real-world, free-form language—an essential capability for organizations using AI to analyze or train models on customer interactions while maintaining privacy compliance.
Tonic Textual achieved the strongest overall performance, recording an F1 score of 0.93, with Precision of 0.91 and Recall of 0.94, indicating its ability to comprehensively detect sensitive entities while minimizing false positives. Even when adjusted to match the limited entity coverage of competing systems, Textual maintained exceptional balance, with F1 scores ranging from 0.92–0.94. This consistency demonstrates its adaptability to varied data schemas and conversational structures without loss of accuracy. AWS Comprehend was the next best performer (F1: 0.86), showing solid recall (0.90) but reduced precision (0.83), suggesting a tendency toward over-redaction.
In contrast, the remaining models struggled significantly with this type of unstructured data. GCP_DLP (F1: 0.54), Microsoft Presidio (F1: 0.43), Azure AI Language (F1: 0.46), and spaCy (F1: 0.48) all exhibited weak performance, with low precision and recall values indicating inconsistent or incomplete identification of sensitive elements. GLiNER, with an F1 of just 0.31, performed particularly poorly, missing most true positives entirely. Even under adjusted conditions where Tonic Textual was limited to the same entity definitions as these frameworks, it maintained near-perfect precision and recall—further proving that its advantage stems from deeper contextual understanding and model optimization, not broader entity support. Overall, Tonic Textual's performance across conversational datasets highlights its ability to handle the messiness of real-world communication, making it uniquely capable of supporting privacy-preserving AI and analytics in customer experience workflows.
The costs and opportunities
The cost of getting it wrong
When sensitive data slips into downstream model training, the consequences can be both immediate and far-reaching. Personally identifiable information (PII) or protected health information (PHI) that enters a model's training corpus doesn't just violate compliance standards—it can become permanently encoded into the model's parameters. Once this occurs, there's no simple rollback or patch. Sensitive details such as names, addresses, or medical record numbers can resurface in model outputs through memorization or inference, creating the risk of unintentional re-identification even when the original dataset is no longer accessible.
Beyond the technical risks, data leakage carries significant regulatory and reputational impact. Under frameworks such as HIPAA, GDPR, and emerging AI governance laws, organizations are accountable for how sensitive data is processed, even when exposure results from automated systems. A single instance of private information generated by an AI model can trigger compliance investigations, lead to regulatory fines, or cause lasting damage to public trust. These breaches can also undermine strategic partnerships, slow down product approvals, or halt AI deployments entirely until a remediation plan is in place.
From a business standpoint, the operational fallout can be equally severe. Teams may be forced to discard or retrain expensive models, restart data collection efforts, or pause innovation cycles—all of which translate to wasted compute, delayed time-to-market, and lost competitive advantage. The reputational cost is harder to quantify but often longer-lasting: when customers or partners lose confidence in a company's ability to protect sensitive data, rebuilding that trust can take years.
Ultimately, the cost of getting it wrong underscores why de-identification cannot be treated as a box-checking exercise. It must be pressure-tested, measurable, and repeatable—a core part of the data lifecycle rather than an afterthought. A reliable de-identification strategy doesn't just prevent leaks; it preserves the integrity of downstream AI systems, ensuring that innovation, compliance, and trust move in lockstep.
The opportunity of getting it right
When sensitive data can be safely and responsibly incorporated into AI model training, the potential for innovation expands dramatically. Privacy-preserving techniques such as de-identification and synthetic data generation make it possible to unlock datasets that were previously off-limits, allowing organizations to extract deeper insights without compromising trust or compliance. In healthcare, this means that models can be trained on authentic clinical narratives, enabling the discovery of new treatment pathways, early-detection biomarkers, and personalized care recommendations that would be impossible to derive from sanitized or artificially limited data alone.
The same principle applies across industries. In financial services, responsibly de-identified transaction data can power fraud-detection systems that identify subtle behavioral patterns long before losses occur. In the legal and public-sector domains, anonymized document corpora allow for the training of retrieval and summarization models that accelerate case research, improve transparency, and reduce administrative backlog—without risking exposure of privileged or personal information. By ensuring privacy at the data layer, organizations can confidently extend AI into workflows that were once considered too risky to automate.
Ultimately, a well-engineered de-identification strategy doesn't just mitigate risk—it creates opportunity. It transforms sensitive data from a liability into an engine for discovery, allowing teams to explore, innovate, and build AI responsibly on the datasets that matter most. When privacy protection is built into model training from the start, organizations gain the freedom to pursue breakthroughs that are both ethically sound and economically transformative. For a companion look at how Textual holds up against LLMs on PII detection and synthesis, see our PrivacyBench benchmark.
Ander Steele, Head of AI at Tonic.ai, is at the forefront of building privacy-preserving data and machine learning pipelines. He tackles complex AI challenges such as synthetic-data generation, evaluation automation, and scalable inference, with a particular focus on making sensitive data safely useful for real-world AI systems. A mathematician by training, Ander holds a Ph.D. in Mathematics and Statistics from Boston University and a B.S. in Applied Mathematics from Georgia Institute of Technology. His diverse background includes academic positions as a Visiting Assistant Professor at UC Santa Cruz and a Postdoctoral Fellow at the University of Calgary, as well as industry experience as a Senior Data Scientist at Fullpower Technologies, where he applied deep learning to analyze sensor data related to sleep and health disorders. At Tonic.ai, he is responsible for the development of the company’s AI models, ensuring secure and compliant data use for AI training. Off the clock, you’ll find him mountain biking or running the redwoods around Santa Cruz.
