For CSV files in a dataset, from the file preview, you can now configure entity type handling for entire columns. A column might be structured and sensitive, structured but not sensitive, or unstructured. Structured columns contain a single consistent type of value, such as a name, date, or number. For unstructured columns, such as a description or notes, the values can contain multiple entities of different types, and must be scanned individually. Note that this only applies to CSV files that are added to a dataset after this release. Existing CSV files are not affected.
Fixed HTML replacements so that inline breaks such as `
` do not pull adjacent text into phone and email replacement values.
Fixed HTML report mappings so that spanning HTML replacements keep their surrounding markup in report values.
Added an optional phone metadata flag to preserve US prefixes such as `(855)` or `1 (234)` during synthesis.
Added a new endpoint to retrieve dataset entity mappings grouped by file. From the Python SDK, to retrieve the entity mappings, use Dataset.get_entity_mappings(). The results include original, redacted, synthetic, and final output values. When available, the results also include tabular row, column, and score metadata. The results respect dataset entity type handling configuration. When none of the entity types are active, it can return empty per-file entity lists.
Redesigned navigation for Textual - The top navigation bar for Textual is replaced by a left-hand navigation menu. The navigation menu by default displays icons, but can be expanded to display text labels. The expanded menu includes the usage information.
Fixed an issue where the panel to add or block entity values displayed the wrong entity type.
On a dataset PDF or image file preview, the details popup for an entity includes the confidence score.
Fixed an issue where the dataset permissions page did not update immediately when you changed the assigned permission set for a user or group.
Fixed an issue where uploading a very large number of files could cause the application to hang and not display the files.
The Datasets page now displays the count of files in each dataset.
Fixed the user search for global permission set assignment.
On the dataset details page, the Analytics tab under Entities Analysis can now display more than 10 entity types.
Dataset text search - On the dataset details page, under Entities analysis, the new Dataset search page allows you to search for a word or phrase in the dataset files. For each instance found in the dataset, the results indicate the file and whether the text was detected and assigned an entity type. You can filter the results to only include matches in specific files. On a self-hosted instance, to enable the text search, you must configure a connection to either Apache Lucene or OpenSearch.
Manual redactions in datasets - From the dataset file preview, you can now manually add redactions to files. To add a redaction to a text file, double-click or select the text, then select the entity type to assign. To add redactions to PDF and image files, click Add Redaction to change to manual redaction mode. For each redaction, draw a box over the content to redact, then select the entity type to assign.
We have improved how model training works for model-based custom entity types. If you have a model that was trained on an earlier version, you must re-train the model.
Note for self-hosted customers
v374 includes a database migration that significantly improves application performance by moving from JSON blobs to a more normalized form.
Because of this migration, Textual services will be unavailable during the upgrade. The process takes only a few minutes for most customers, but can take up to several hours for organizations that have specific data profiles.
To determine the migration time, review the number of rows in the DeidentifyFileRowResults table. To determine the number of rows in this table, connect to your PostgreSQL metadata database and run
SELECT count(*) from "DeidentifyFileRowResults";
If the count is higher than 300,000 rows, expect the migration to take several hours.
To decrease the number of rows in this table, in your datasets, delete any CSV or XLSX files that you no longer use. These file types add a disproportionate number of rows to this table.
If you need any assistance, reach out to Tonic.ai.
For datasets that write JSON output, Textual now supports RTF files. You cannot use RTF files in datasets that generate synthesized files that are of the same type as the source files.
For JSON output datasets, updated the preview page to remove the Entities tab and to add the ability to download the redacted version of the Markdown. For RTF files, you can also download redacted HTML.
Note that v370 was not released because of technical issues.
Self-hosted users can now use AWS Bedrock models for annotation and suggestions during the model-based custom entity types creation and training flow.
Users can now delete training files for a model-based custom entity type, which excludes the file from future models.
Users can now delete test files for a model-based custom entity type, excluding the file from future models.
Fixed an issue where the detected entity counts did not update to reflect changes to the dataset settings.
Fixed an issue where the dataset entity type filter did not correctly return detected custom entity types.
Improved the entity grouping endpoint. Specifically, it is now less likely to cause exceptions in Textual.
On the dataset file list, when a file needs to be rescanned, Textual now displays a small warning icon in front of the column name and adds a clickable rescan icon to the options column.
Entities catalog for dataset entity values - Replaced the entity type value count and value modal with a new Entities catalog page for the dataset. The Entities catalog lists all of the detected entity values in the file. The list includes the context, transformed value, the Textual confidence in the detection, and the file where it was detected.
On the Entity Settings page for a dataset, restored the option to select the generator to use to generate synthesized values for custom entity types.
The dataset files list now displays the OCR method used for PDFs and images.
The dataset files list indicates when each file needs to be rescanned.
Fixed an issue where the preview for a dataset PDF file did not allow users to change the entity type handling.
On a self-hosted instance, you can now select the OCR engine to use for individual datasets. The setting is in the PDF settings section of the Project Settings page for the dataset.
Fixed an issue that caused an error when changing resource editing permissions when deprecated pipeline permissions were present.
Dataset details redesign - On the new dataset details page, the left panel contains a menu of options to display the list of dataset files, the entity type settings, and the project settings. Added more information to the file list. Added filters to the entity types list and changed the handling option from a toggle to a dropdown list. Enabling and disabling custom entity types is moved to a new separate side panel.
For dataset .docx and .xslx files, when the URL entity type is set to Redact or Synthesis, Textual now automatically replaces hyperlink destinations with google.com. If the URL entity type is set to Off, then Textual does not replace the hyperlinks.
Improved the grouping model to enable disambiguation of distinct entities that have identical names.
Replaced the Getting Started navigation link and modal with panels on the Home page that contain links to perform actions related to the API, datasets, guided redaction, and custom entity types.
On the Home page, when you upload a file that the redaction preview tool doesn't support (such as a PDF file), Textual now prompts you to create a dataset that contains the file.
Improved detection for date of birth values.
Added a new built-in entity type, US_ROUTING_TRANSIT_NUMBER, to identify the routing number of a bank in the United States.
Removed the pipeline functionality from the API and the SDK.
For guided redaction, added guided redaction permissions to control whether users can:
- Add and edit comments
- Mark redactions as reviewed
- View the audit log
On the Home page, for redacted values, the entity type is now followed by a token to distinguish the value. The same value always gets the same token.
When you preview a file, and an entity is flagged with both numeric_pii and a more specific entity type, the more specific entity type is preferred.
Model-based custom entity types now use a larger context to include prediction errors and provide better guideline suggestions.
On the dataset details page, standardized how regex-based and model-based custom entity types are displayed.
Added a user profile setting to specify the name of the organization or team that the user belongs to.
Guided redaction (beta release) - The new guided redaction feature, currently in beta, produces redacted files for use cases such as Freedom of Information Act requests. Guided redaction supports a process that includes redaction and review.
Before you create a guided redaction project, you set up reference codes to identify specific types of values, and can map those codes to Textual entity types. Projects and files are assigned statuses to indicate where they are in the process. Textual provides a basic set of status values that you can adjust.
For the redaction phase, Textual performs an initial scan to detect sensitive values. You can then add and remove values and adjust the assigned reference codes.
The review process verifies that the redaction is correct and complete.
You can preview each file to verify the redactions before you download the output. All output files are PDFs, regardless of the original format. You can choose the color of the box that hides the redacted values, and whether to include the reference codes.
Model-based custom entity types - Textual now supports a new way to create a custom entity type. Previously, custom entity types always looked for values that matched regular expressions. Those are now called regex-based custom entity types, and are useful for entity types that have a smaller set of values or have values that use a standard format. For the new model-based custom entity types, you define and train models to identify entity values for an entity type. Model-based custom entity types are useful for when there are a wide range of values that need to be identified more by context than by format.
Added the following environment variables related to how Textual processes PDF files:
SOLAR_PDF_PAGE_REDACTION_PARALLELISM- When you download or preview a PDF file, the total number of pages to redact simultaneously. The default is 4.SOLAR_PDF_DOC_PLAN_PARALLELISM- Before downloading or previewing PDF files, the total number of PDF documents that can be redacted simultaneously. The default is 3.
Textual customers who use Okta on Textual Cloud must now configure an organization-specific sign-in URL in their Okta application. The redirect URL is displayed on the Single Sign-On tab of the Permission Settings page.
You can now create a Textual dataset that connects to a SharePoint repository.
The dataset details page now includes the option to share access to the dataset. Previously, the option was only on the Datasets page.
Support cloud storage and JSON output for datasets - When you create a dataset, you can now choose whether to upload files from a local file system or to select files and folders from Amazon S3. For Amazon S3 datasets, Textual generates the output to a configured output location. You can also configure a dataset to generate JSON output instead of redacted files. The JSON output matches the output currently produced by Textual pipelines. Note that after we add support for Azure and Sharepoint datasets, we plan to deprecate and remove pipelines.
On self-hosted instances, you can now include SSO group names in the value of the SOLAR_ADMINISTRATORS environment variable, which determines the assignment of the Admin (Environment) permission set.
For .txt, .csv, and .docx files, the file preview now allows you to select the entity type handling option. To select the option, in the right-hand pane, click the value, then click the handling option. The selected option is applied to all instances of that entity type.
The Textual REST API now provides endpoints to perform the following actions:
- Retrieve information about and share dataset access
- Retrieve information about permission sets
- Retrieve information about an organization’s users
- Retrieve information about an organization’s SSO groups
Fixed an issue where the Datasets page did not provide an affordance to page through the datasets list.
Fixed an issue with font size in synthesized PDF values.
On the Datasets page, removed the Last Updated column. Added a Created By column, which displays the name of the user who created the dataset, and when the dataset was created.
Default permission sets - From the global permission sets list, you can now select a global permission set to assign to all Textual users in your organization. By default, all users are granted the built-in General User global permission set.
Default permission set for dataset and pipeline creators - You can now select the dataset and pipeline permission sets to assign to the user who creates a dataset or pipeline. By default, the creator is granted the Editor dataset or pipeline permission set.
