De-identifying and synthesizing healthcare PDFs of patient lab reports for model training and Expert Determination

Tonic Textual helps healthcare customers obtain expert determinations on unstructured data like lab reports, discharge notes, and EMR records. This de-identified data is often used for data commercializing or model training, like fine-tuning. Unstructured data can be difficult in general to work with because of its messiness and high occurrence of PHI. PDFs, however, are typically the most difficult data modality to work with. The reasons are several fold.

Preparing PDFs for an expert determination is a two-step process. First, all PHI must be identified and second, it must then be replaced with realistic fake data. The realism of the fake data is paramount to ensure that the data maintains a high utility, which is often a requirement when training models and reselling data. Each step has its own set of challenges that are unique to PDFs.

Before we hop into those challenges, let’s motivate this blog post by showing some common PDF outputs from Tonic Textual. Below is a side-by-side comparison of a Labcorp lab report. On the left side is the original PDF, and on the right is a redacted version of the same PDF.

Redacting PDFs is useful for some data commercialization engagements, however, synthesizing the PDF (replacing real PHI with realistic fake PHI) is far more common and is used even more often for model training. Below, the same original image is on the left next to a synthetic version of the document on the right-hand side. As you can see, the relevant PHI has been synthesized now, leading to a new PDF document that visually looks correct and whose PHI is now fake.

Now that you have seen what is possible, let's discuss the challenges that had to be overcome to arrive at these great results.

Identifying PHI in PDFs

To identify PHI in PDFs first you must convert the PDF to text. This is typically done via an OCR engine like Tesseract (free and open source) or a cloud solution like Azure Document Intelligence or AWS Textract. The problems you can expect are:

Visual vs Logical disconnects

PDFs don’t store their text content in continuous sentences or paragraphs. Each character has an X and Y coordinate on the page. When we extract text it is not given that the order in which text is extracted is the same as the way a human would read text. This leads to PHI identification issues as the text can appear garbled or nonsensical. Typically, the more structure the PDF has (tables and forms) and the more complex the layout the worse this problem becomes. One particularly tough type of PDF to handle is lab reports.

Destruction of table and form structures

Healthcare PDFs typically contain a lot of form and tabular data. For example, at the top of many PDF documents is form-like data identifying the patient and provider. It is also common to see tables of medications, lab results, measurements, and diagnostics. Some OCR engines are able to successfully preserve this type of layout information while others like Tesseract cannot. Regardless, the process is imperfect, and there is always some loss of information except for on the simplest of PDF documents (and nothing in healthcare is simple!)

Character recognition mistakes

A massive percentage of healthcare documents are not digital native PDFs. They are typically scanned documents, photocopies, faxes, etc. This can lead to actual mistakes in the core OCR process where characters are misidentified. Misidentified characters make the job of identifying PHI harder.

Visual artifacts

Headers, footers, and page numbers are common in healthcare documents. They are extracted from PDFs like all other text and are inserted at seemingly random places in the text fed into the model for identifying PHI. This can cause issues when these artifacts are dropped into the middle of paragraphs that span two pages.

Variability in layout

Even for a given type of document, like a lab report generated during an annual checkup, the layout and structure of the PDFs can vary wildly from provider to provider. Techniques which make assumptions on this layout in general do not work.

Synthesizing replacement PHI

As a core piece of functionality, Textual synthesizes replacement PHI when processing data. This is true for PDFs as well. However, with PDFs specifically, there are some very unique challenges. As a motivating example, replacing text in a JSON, HTML, TXT, or Markdown file is relatively trivial. You know the start/end position of the span of text to replace, you know the replacement span you wish, and you can insert it. With PDFs, it is not so simple and there are a whole host of reasons why.

Font embedding and Proprietary fonts and missing fonts

PDFs typically don’t store entire font tables, but instead subsets. These subsets include font information only for the characters included in the PDF and not for the entire alphabet. This means the PDF writer might lack the instructions to write specific characters. In an even worse case, the medical PDF could use a proprietary font. In cases where the system lacks the necessary font information it must fall back to the ‘next closest’ font available. Determining the correct font to use can be a challenge.

Spatial Constraints

These typically come in two forms. First, if the replacement text wraps a line it can turn into a tricky optimization problem, especially if the replacement text has a different number of words or is much different in width. This leads to the second form where the replacement text is too long or too short compared to the original. This can require shrinking or expanding the text, which causes the font size to jump around as you read the document.

Micro-typography

High-quality PDFs place words accurately at a very fine resolution. Kerning and Justification done improperly can stick out visually like a sore thumb.

Building a system that can take all of the above into consideration to ensure PDFs are accurately sanitized is something Textual has been working on for several years now. Our PDF synthesis capabilities are unique in the industry and have helped our customers achieve expert determinations on their healthcare PDFs (and other data) in as little as 5 business days. Besides the ease of obtaining the expert determination, our customers have access to the highest quality sanitized healthcare data available leading to easier data commercialization conversations and better results when training models.