Tonic Structural release information

Enhancements

If data generation is not blocked on all schema changes, Tonic now displays a dismissible warning when there are non-conflicting schema changes. Conflicting schema changes always block data generation.

For the GenerateData API endpoint, added an optional clientResourceId query parameter. When you provide a value, then jobs that have the specified clientResourceId run serially instead of in parallel. The check applies to all jobs across the instance, regardless of whether they belong to the same workspace.

Other changes

Fixed an issue that made it difficult to click the Cancel Job button in the Job History list.On the workspace configuration view, the source and destination database details are now populated correctly when you refresh the page.

When granting access to a workspace, improved how we display long names and email addresses.

For tables that use Scale mode, removed Passthrough from the generator selection dropdown list. Previously, the option displayed in the dropdown list even though it couldn't be selected.

Improved performance for looking up tables and columns.

Corrected an issue where the same environment variable with different cases caused Tonic to crash.

When logging in using single sign-on (SSO), when the email address uses a different case from an existing username-password account, it now resolves to the same user.

For instances deployed using Docker Compose, Tonic now cleans up old, unused images.

Improved performance when running generators on de-identified tables.

Corrected an issue where the Event Timestamps generator produced unexpected values for linked columns.

Improved error handling for account creation.

MongoDB

• Updated MongoDB.Driver and MongoDB.Bson to 2.18.0.
• Subsetting jobs with foreign keys can now run on collections that have lower permissions. To enable this, set the environment variable TONIC_BYPASSDOCUMENTVALIDATION_ON_DOWNSTREAM_KEY_MERGES to false. The default value is true, which means that the destination database requires the dbAdmin role in addition to readWrite.

Oracle

• Tonic now automatically uses the optimal table prefix for schema queries. The TONIC_ORACLE_DICTIONARY_TABLE_PREFIX environment variable is removed.
• Improved error handling for missing permissions for table size queries.

SQL Server

• Improved performance for table queries.

Reverted an update from v580 that caused slow performance when retrieving the list of tables for a workspace. In some cases, this caused Tonic to indicate that no tables were available.

Fixed an issue where custom value processors interfered with the display of configuration options for the SSN generator.

Fixed an issue where the dropdown lists to select a column generator periodically scrolled to the top of the screen.

Improved performance when navigating among Tonic views.

Improved the list of suggested sub-generators for the Conditional generator.

Improved the data generation process to prevent jobs from hanging when an error occurs.

Fixed an issue where the post-subset generation row count always returned 0.

Updated the Tonic logging framework.

Google BigQuery

• Improved handling of timestamp and datetime fields.

MongoDB

• Fixed an issue where workspaces crashed when running data generation on new collections.

MySQL

• Corrected the handling of the BIT data type.
• Improved performance when loading workspaces.

Oracle

• The Constant generator now works correctly for RAW columns.
• Stopped using source database constraints that were in the recycle bin to populate the destination database.
• Improved performance when retrieving schema information and loading workspaces.
• Added the OracleDriverAnalyzer tool to analyze Oracle performance.

PostgreSQL

• Display an error when an excluded or included schema does not exist in the source database.
• Fixed a permissions error that occurred when creating a new workspace using Azure PostgreSQL for the destination database.

Spark

• Fixed an issue with the workspace configuration view that prevented users from creating and updating Spark-based workspaces.

SQL Server

• Fixed an issue that caused a data generation error when XML indexes had dependencies on other indexes.
• Upgraded SqlClient to 5.0.0.

Enhancements

The Subsetting view now shows for each table the percentage of data that is included in the destination database.

Other updates

The workspaces view no longer briefly flashes a message indicating that the workspace cannot be found.

Added the ability to display trace information in the log files. To enable the trace information, set the environment variable TONIC_LOG_TRACES to true.

Fixed an issue that caused an incorrect warning to display when linking columns that were assigned the Custom Categorical generator.

Fixed a data type error in AI Synthesizer for models that only contain categorical data.For the Event Timestamps generator, Tonic now prevents the generator from being assigned to a time-only value.

Amazon Redshift

• Added ServerCompatibilityMode to the Redshift connection string to prevent connection errors.

Databricks

• Added support for Databricks 10.x.

MongoDB

• Fixed an issue where Tonic generated collection names in Preserve Destination mode that exceeded the maximum length limit.

MySQL

• The MySQL data connector now supports the ability to use the DELIMITER command in post-job scripts.

Oracle

• Improved performance for schema queries.
• Improved performance for databases without database links.
• Restored missing information for loader errors.
• Improved performance for applying indexes and constraints.
• Added the environment variable TONIC_ORACLE_REDO_LOG_ENABLED, which by default disables recovery information writes to REDO LOG files.
• Improved performance for retrieving tables and columns.
• Fixed an issue with loading custom types.
• Fixed an issue where multiple indexes on the same column caused data generation to fail.
• Corrects a regression from V611.
• When fetching constraints, Tonic can now support primary key indexes that have the same name.
• Excluded views from table queries.

PostgreSQL

• Updated the process of loading tables to improve performance and eliminate an error indicating that PostgreSQL types could not be found.

SQL Server

• Fixed an issue where Preserve Destination and Incremental table mode could not be used in tables with foreign key relationships.
• Tonic now provides a more informative error when data generation fails because the database is not accessible.
• Tonic now properly handles identifiers with single quotes.
• Fixed an issue that caused a worker process to crash when using Azure to connect to the database.
• Added additional logging of the available databases.
• Improved handling of schema and table names in privacy scans.

Enhancements

You can now generate DEBUG level logs for the Tonic API. To do this, set the environment variable TONIC_CONSOLE_LOG_LEVEL to DEBUG.

Tonic now supports logging for long-running queries. The environment variable TONIC_LONG_RUNNING_QUERY_LOGGING_INTERVAL provides the interval in minutes for logging queries. By default, Tonic generates a log entry for a long-running query every 10 minutes. To see this information, TONIC_CONSOLE_LOG_LEVEL must be set to DEBUG.

Other updates

Terminology change - In the Tonic documentation, we have changed the term "mask generator" to "composite generator".

Fixed an issue that prevented the use of Preserve Destination and Truncation modes on tables and collections that had names that were close to the maximum length limit.

You can now assign the Integer Key generator as a sub-generator for string values in composite generators.

For Google SSO, fixed an issue where users who did not have a group membership could not access Tonic.

Improved performance for format-preserving encryption (FPE), which is primarily used for key generators.

Improved generation performance for tables that use De-Identify table mode.

Improved error display when reading data from the source database.

Fixed an issue where values for columns that were assigned the Passthrough generator were being masked with 1s and 0s.

The User Settings page no longer displays the password change option for SSO users.

For the JSON Mask generator, fixed an issue where a large matching value caused the buttons to move off of the configuration dialog.

Fixed an issue where data connection pooling caused data generation to fail because of colliding queries.

Google BigQuery

• Fixed an issue where source data was not loading properly into the Tonic user interface.

MongoDB

• Fixed an issue where the generator configuration panel closed unexpectedly after making edits.

Oracle

• Added a warning when a generator produces a value that is truncated in the destination database.
• When Tonic retrieves table sizes, the results now include the size of the associated large objects (LOBs).
• To help with troubleshooting, Tonic can now determine which Oracle patches were applied to a database.
• Fixed an issue that caused errors when reading a large object array.
• Added additional checks to the data connection tests.
• Updated our Oracle Driver to 3.21.80.
• Fixed an issue where primary keys could not be copied to another index.
• Additional performance improvements.
• Corrected how we check the current user's database privileges.

PostgreSQL

• Added an option to not refresh materialized views. To turn off the refresh, set the environment variable TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS to false.
• Fixed an issue with parsing PostgreSQL schemas in Windows environments.
• Fixed an issue where data generation failed when the data contained read-only arrays.

Snowflake

• Fixed an issue where data generation failed when the schema definition (DDL) contained row access policies.

SQL Server

• Added permission checks to the data connection tests.
• Fixed an issue where Tonic processes failed.

Improved the performance of Table View, particularly when scrolling.

Reworked data generation to better group generator errors.

On the Subsetting view:

• Improved accuracy of the row counts for small tables.
• The table list shows whether each table is in the subset and the number of pre-subset and post-subset rows even when subsetting is not enabled for data generation.

The health check for the PyML container can now use HTTP instead of HTTPS.

MongoDB

MySQL

• Improved performance when loading workspaces.

Oracle

• Fix to improve query performance.
• Added permission checks when testing database connections.

PostgreSQL

• Tonic now applies the source database check constraints on the destination database after data generation is complete.

Snowflake

• Fixed an issue where data generation failed when there were tables that used Preserve Destination table mode.

Enhancements

Workspace inheritance - For enterprise customers, the workspace inheritance function allows you to create child workspaces that automatically inherit source data and Tonic configuration from their parent workspace. Changes to the parent workspace configuration are copied to the child workspace. You can override the subsetting configuration and post-job scripts in a child workspace.

Links from Schema Changes to Database View - Added links from Schema Changes entries to Database View. The links automatically filter Database View to only include the affected column or table. The links only display for columns and tables that are in the source database. Removed columns or tables do not have links to Database View.

Schema filtering for PostgreSQL - For PostgreSQL workspaces, when you create or edit the workspace, you can specify a list of schemas to either include or exclude from the source database.

User profile pictures - From the User Settings view, Tonic users can now upload a user image for their account. For SSO providers that support user images, the image from the SSO is used by default.

Zip code configuration for HIPAA Address generator - A new configuration setting for the HIPAA Address generator allows you to determine how the generator sets the zip code. If the setting is off, the generator uses the current process, which replaces the last two digits of the zip code with zeros. For low population areas, the zip code is all zeros. If the setting is on, then the generator selects a real zip code that starts with the same three digits as the original zip code. For low population areas, if a state is provided in the data, the generator selects a random zip code from that state. Otherwise it selects a random zip code from the United States.

Other updates

Minor updates to the Subsetting view. New icons for the subsetting summary and the inbound and outbound relationship counts. Added a Use subsetting toggle to indicate to use the subset configuration for data generation. This toggle is synchronized with the same toggle on the Confirm Generation panel.

Fixed an issue where Tonic logged users out between browser sessions more often than expected.

Improved performance for the JSON Mask generator.

Fix to ensure that the AI Synthesizer is canceled when a data generation job that includes AI Synthesizer is canceled.

Fixed an issue where a data generation job would hang if there was a failure.

Fix to address an issue where after a one-click update, the Tonic version unexpectedly regressed to an earlier version.

Fix to ensure that cross-table commands do not run in parallel, which could cause deadlocks in the database.

MongoDB

• Fixed an issue that prevented the Regex Mask generator from being applied to fields in arrays.
• Updated to require 3.6 as the minimum server version for MongoDB.

PostgreSQL

• Sequence values are now copied over correctly.

Spark

• Fixed a data generation error in Hive and Livy that occurred when a source table had non-lowercase column names.

Enhancements

Assigning generators from Schema Changes view - On Schema Changes view, new columns, changes to column data type, and changes to column nullability have a Select dropdown list that includes an option to assign a different generator to the column and then resolve the issue.

Complete list of blocking issues for data generation - When data generation is blocked, the generation panel now displays all of the blocking issues. This allows you to correct all of the blocking issues before you attempt to run data generation again.

Other updates

Made some visual updates to the Tonic navigation pane and the Tonic login panel.

On the Job History page, the details popover for queued jobs now points to the correct job.

In Database View, the generator list for the Applied Generators filter is now correctly alphabetized.

Fixed an issue where the dropdown arrow for the table mode selector was not always clickable.

When importing a workspace, Tonic now validates that columns do not have multiple generators assigned to them.

Updated to provide clearer error messages when there is an issue with an assigned sub-generator.

Fixed an issue where Privacy Hub sometimes did not reload after a new sensitivity scan.

The database type filter now includes all of the available Spark database types.

Corrected the link to the Tonic privacy policy.

Fixed an issue where jobs failed when multiple tables with the same name in different schemas were assigned Preserve Destination table mode.

Timestamp Shift is now the recommended generator for Date and Timestamp columns.

Corrected the display of available buttons on the Tonic application.

MongoDB

• Improved error handling for database summary queries.
• Added a warning about subsetting performance for percentage subsetting target tables on MongoDB versions before 4.4.2.
• Fixed a memory usage issue with loading Collection View when the data contains large arrays.

Oracle

• A new environment variable, TONIC_ORACLE_DATA_PUMP_PARALLELISM, allows you to choose the maximum number of threads for parallelization for Oracle Data Pump.
• Optimized queries for better performance.

Spark

• Added the ability to specify the proxy user when using Livy.
• Fixed an issue where users could change the configuration type when editing a Spark workspace.

Enhancements

Subsetting results - The Subsetting view now displays the results of the most recent subsetting data generation run. The information includes:

• When the job began and ended.
• The percent reduction from the original source data to the subset destination data.
• The percentage of the source data that is included in the subset destination data.
• The volume of data in the source data and the subset destination data.

Schema changes filter on Database View - On Database View, the advanced filters now include an option to only display columns that have unresolved schema changes. This filter is not combined with other filters. When you filter for unresolved schema changes, the other column filters are disabled.

Larger WHERE clause editor for subsetting - For WHERE clause target tables, you can now display an editor with a larger text area for entering the WHERE clause. This provides better support for longer, more complex WHERE clauses.

Other updates

Fixed an issue that prevented users from deleting more than one tag from the Edit Workspace view.

Added the ability to run Tonic workers, the Tonic web server, and Tonic notifications on Heroku.

Updates to improve handling of canceled jobs, both when users cancel jobs and when jobs fail.Improvements to data generation memory handling and performance.

Improved the Synthesis Report for AI Synthesizer.

Fixed an issue where adding constraints to a destination database resulted in deadlocks.

Oracle

• Update to allow the JSON Mask generator to be used on user-defined types (UDTs).

PostgreSQL

• Fixed a regression that caused sequence fetching errors in v9.6 and earlier.

Spark

• Improved performance for the Continuous generator.
• Enabled partition filter validation on Hive.

SQL Server

• Added support for full text catalogs in Server Management Objects (SMOs).
• Fixed an issue to enable the correct handling of schema, table, and column names that contain single quotes.
• Made a fix to correctly display error messages.

Enhancements

Filtering workspaces by tags - On the Workspaces view, you can filter the workspaces by the assigned tags. In the Tags column heading, click the filter icon to display the list of applied tags. Check the checkbox for each tag to include. The list is filtered to include workspaces that have at least one of the selected tags.

Automatically resolving schema changes - Schema changes are now resolved or dismissed automatically when you update the table or column configuration.

• For a new table, the schema change is dismissed when you set the table mode to Truncate or Preserve Destination.
• For a new column, the schema change is dismissed when you assign a generator to the column.
• For a column that has a conflicting data type or nullability change, the schema change is resolved when you assign a different generator to the column.

Other updates

Corrected the number of rows for out-of-subset tables. No longer show 0 when out-of-subset tables are processed.

Improved error messaging when a where clause for a subset target table is invalid.

Corrected an issue where the subsetting table configuration was not handled correctly.

The job types filter on the Job History view now only shows valid types for the workspace.

Improved performance for data generation.

Improved performance for the AI Synthesizer.

Improved reporting of subsetting progress when parallelism is enabled.

MongoDB

• You can now assign the Preserve Destination mode to a collection.
• Subsetting and foreign key management is now enabled for all customers.

Spark

• For workspaces that use Livy, you can now assign the Continuous generator and Noise Generator as sub-generators for a mask generator.
• Improved performance for the Categorical generator.
• Improved performance for the HIPAA Address generator.

When you create a sample workspace, it now includes a tag called Sample Tag.

Fixed an issue where the Update Tonic button was not displayed correctly.

Improved the user interface for activating new hosted accounts.

Improved message to notify users that the subset configuration changed since the last subsetting data generation.

The HIPAA Address generator now works correctly for US addresses.

MongoDB

• You can now link fields in MongoDB collections.
• Conditional generators can now be applied to array elements.
• The Random Timestamp generator now works correctly on datetime columns.

Oracle

• Data generation can now work without DBLink.

PostgreSQL

• Completed additional updates to support cross-schema types.

Enhancements

You can now enable parallel processing for subsetting. The TONIC_SUBSETTING_PARALLELISM environment variable sets the number of steps to process in parallel.

Other updates

Made a couple of small improvements to the AI Synthesizer generator.

Fixed an issue where users could not change the percentage on a subsetting target table.

Improved how we sort tables for parallel processing to improve efficiency.

MongoDB

• The Categorical, Current Date, Date Truncation, HIPAA Address, and Unique Email generators now work on Mongo array fields.
• Linking for the Continuous generator now works correctly.

Spark

• Added support for Spark 2.3 on Livy.

Enhancements

On Privacy Hub, the term "Unprotected Sensitive" is replaced with "At-Risk". The protection status counts exclude columns that are not included in the destination database.

On Database View, added an option to filter by whether a column is included in the destination database. A new At-Risk toggle provides a shortcut to filter for columns that are included, marked as sensitive, and not assigned a generator. The Column Type filters, which filter columns based on whether they are a primary or foreign key, are changed from checkboxes to toggles.

In the Privacy Report, added Not Included as a value for ColumnPrivacyStatus, to identify columns that are not included in the destination database. The value Protected replaces the current values Masked and Anonymized, which are moved to a new ProtectionType column.

Other updates

Improved the display of the Database View advanced filter for smaller screens.

Fixed an issue with the generation of the API reference.

Began to log latencies each hour for source and destination databases.

MongoDB

• In Collection View, improved the display of key columns on smaller screens.
• Corrected an issue that resulted in duplicate schema change results.
• Fixed the collection scan status when a new scan is started.
• Single document view no longer reloads when you apply a generator.
• Corrected an issue where selecting a different collection to view applied updates from the previous collection.
• Fixed an issue where the Null generator could not be removed from a field.

MySQL

• Fixed an issue where the Tonic application could not load when table and row size estimates were not available.
• Increased the connection resiliency for write operations.
• Improved handling of different character sets in MySQL.

Enhancements

On Privacy Hub, the list of unprotected sensitive columns is replaced with a Database Tables list. The Database Tables list summarizes the protection status of each table in the source database. The Privacy Status column summarizes the protection status of the columns in the table. It provides access to the same column details and configuration options as the protection status panels at the top of Privacy Hub.

You can now filter the Workspaces view based on your assigned role in the workspace. For example, you can only display workspaces for which you are an Owner or an Editor. To use the role filter, click the filter icon in the Role column heading. For admin users, the role filter includes a Role assigned toggle to allow them to only see workspaces that they have a role in.

Other updates

Customers in the Basic license tier can now transfer ownership of workspaces and assign workspace roles to other users.

New users are now logged into Tonic immediately after they create their Tonic account.

For AI Synthesizer:

• The modeling now incorporates static features for each entity across a sequence of events.
• AI Synthesizer now queries to retrieve the actual minimum/maximum values of relevant columns, instead of taking the minimum and maximum of a sample of the data.

To update a license key, self-hosted instances that do not have an admin user can set the license key as the value of the TONIC_LICENSE environment variable. Tonic ignores the variable in instances that have an admin user.

The Workspaces view no longer waits for users to finish applying several new filters in quick succession before it fetches new results.

On the Job History view, fixed an issue where the copy job ID and download logs icons flashed on hover. Removed an error that flashed when job details were displayed.

Fixed the parallel processing for subsetting.

MongoDB

• Indexes are now copied to the destination database.
• Percentage-based subsetting is now supported in versions earlier than 4.4.2.
• Deleted collections are no longer displayed.

MySQL

• Improved memory handling for uploaded CSV files.

Oracle

• Long and long raw columns are no longer converted to blob or clob.

Spark

• Added Maven repository and artifact information to use the SDK Launch method to download the Spark SDK.
• Improved the performance of data generation on Databricks when using Job Cluster. Added support for SQL Warehouses on Databricks.

Added a new Max Categorical Dimension parameter to the AI Synthesizer configuration. This parameter controls the dimension of each column that has categorical or location encoding. If a column contains more distinct categories than this parameter, the most frequent categories are embedded as distinct one-hot vectors. The remaining categories are combined into a single one-hot vector.

Improved error identification for an invalid WHERE clause in subsetting configuration.

On the subsetting page, for tables that were not previously in the subset, the row count is now correctly represented as unknown instead of 0.

Fixed an issue with the Tonic update option in the Tonic application.

Configuring a subset target table to include 100% of the records no longer causes an error during data generation.

Removed connection pooling from Tonic workers to address database connection issues during data generation.

MongoDB

• Added support for partial indexes.
• Fixed an issue where the configured generators were not applied.
• Indexes that use the collation option are now properly recreated in the destination database.
• The subsetting user interface now uses the correct terminology for MongoDB.
• Improved performance for Mongo subsetting when handling downstream tables.
• Fixed percentage-based subsetting for Mongo versions before 4.4.2.

MySQL

• Added support for HASH partition parallelization.
• Running data generation on masked and passthrough tables with ranged sub-partitions no longer results in duplicated data.
• Added support for parallel uploads with ordering.

Oracle

• Added new environment variables (ORACLE_TRACE_LEVEL, ORACLE_TRACE_FILE_LOCATION, ORACLE_TRACE_FILE_MAX_SIZE, and ORACLE_TRACE_OPTION) to enable Oracle tracing.
• Added support for datetime components in composite keys for subsetting.
• Unique indexes are now detected and users cannot apply generators that might violate the enforced uniqueness.
• Improved performance for data generation.
• Improved handling of schema names during data generation.
• Users are no longer incorrectly removed from the destination database.

PostGreSQL

• Upgraded npgsql to address an issue with cross-schema types.

Spark

• Added support for Kerberos authentication for HDFS with Spark / Livy.
• Added support for repartition and coalesce options for Spark EMR and Livy.
• Repartition and coalesce options can now be saved on Databricks.
• Added support in Hive for varchar and char fields that have lengths.

Added Sequence Length Loss Factor and Order Column Loss Factor model configuration options for events data to the AI Synthesizer. Sequence Length Loss Factor indicates the importance of realistic sequence lengths in the model. Order Column Loss Factor indicates the importance of realistic column value ordering in the model.

For the Categorical generator, differential privacy is now off by default.

Increased the amount of time after which an inactive job is assumed to be canceled.

Amazon Redshift

• Fixed an issue that caused the system to crash when you clicked Test Connection during workspace creation.
• Fixed an issue that prevented more than one generation run per version.
• Fixed a HIPAA resource issue that caused data generation to fail.

MongoDB

• Fixed a subsetting issue that caused errors when there was missing data.

MySQL

• Tonic now handles the BIT data type correctly.

Oracle

• Decimal values that are larger than the dotnet decimal data type can now be handled.

Snowflake

• Fixed an issue that caused the system to crash when you clicked Test Connection during workspace creation.
• Fixed an issue that prevented more than one generation run per version.
• Fixed a HIPAA resource issue that caused data generation to fail.

Spark

SQL Server

• Synonyms are now created correctly during data generation.
• Fixed an issue with subsetting based on a percentage of a target table.

New features and enhancements

Redesigned the user experience for the subsetting feature. The new subsetting view displays a list of the source database tables and indicates whether each table is in the subset. When you click a table, the table details display in a new right-hand panel. From the details panel, you can configure target tables. You can also identify lookup tables (previously referred to as reference tables), and indicate how to handle tables that are not in the subset.

If subsetting is configured, when you run a data generation job, you can enable or disable subsetting.

Added latitude and longitude processing to the HIPAA Address generator.

You can now filter Database View based on the applied generator. On the filter panel, in the Applied Generator field, you can provide the list of generators to include.

Redesigned the Schema Changes page. The Actions list is now called Conflicting Schema Issues. Tonic provides clearer warnings when a schema change resolution will result in a change to the workspace configuration. Resolving a conflicting issue or bulk dismissing non-conflicting issues now includes a confirmation step.

Other updates

Renamed the Events generator to Event Timestamps.

Fixed some small display issues in the new Privacy Hub and Subsetting displays.

Improved event generation for the AI Synthesizer.

Improved how we retrieve CloudWatch logs to include the job ID and to use the correct Tonic version.

Corrected the processing of downstream, multi-index tables during subsetting. Corrected an issue in the initial fix.

Improved memory usage during data generation.

Amazon Redshift

• Corrected the handling of time zones in timestamps.
• Fixed test connection and generation in response to an AWS API change.

MongoDB

• UUIDs larger than 16 bytes are now truncated.
• Improved the use of MongoDB resources.
• Improved the display format of LUUIDs.

PostGreSQL

• Corrected how permissions are validated for Heroku.

Oracle

• Tonic now prevents data generation jobs from running when the Oracle versions for the source and destination database do not match.
• Improved the performance of writes to the destination database.
• After deleting a table in the destination database, Tonic now also clears out the recycle bin.
• Fixed generation for LONG and LONG RAW fields.

Snowflake

• Fixed test connection and generation in response to an AWS API change.

New features and enhancements

We enhanced Privacy Hub to add expanded top-level metric panels. These panels show the number of unprotected sensitive columns, protected columns, and unprotected non-sensitive columns. From these panels, you can display column details, select and configure the column generator, view sample data, and add column comments.

You can now add or update a Tonic license from the Tonic application. For a new instance, you are prompted to provide the license key before Tonic displays the login screen. Tonic displays a message when the current license is expired. The Admin Panel includes an option to update the license key.

For the Address generator, added City with State and City with State Abbr to the available options for the column format. You use these options for column values such as San Francisco, California or Boston, MA.

Tonic now supports subsetting for MongoDB databases.

Workspaces can now have a description (up to 200 characters) as well as a name. Use the description field to provide additional context for the workspace and how it is used.

In the Tonic API, you can now sort workspaces based on the last generation date.

Other updates

Tonic now prevents a job from running when the worker determines that the server is running a different version of Tonic.

Refactored the underlying implementation of the subsetting feature.

Made some small memory improvements for data generation.

Implemented performance improvements when applying parallel constraints.

Corrected errors for edge cases related to the Audit Trail.

When the selected workspace changes, the identifier in the URL is now updated correctly.

The workspace configuration is migrated to a data type that enables more efficient querying.

Tonic now validates uploaded foreign keys against the table definitions.

MongoDB

• MongoDB aggregations can now use temporary files on disk to store data that exceeds the MongoDB size limit. This expands the possible generations for a MongoDB database.
• Corrected the generation and display of UUIDs.
• ObjectIds can now be used as primary keys for subsetting in MongoDB.
• Improved the usability of the Add Foreign Key Relationships tab on the Foreign Key Relationships view.

Oracle

• A destination database can now have more than 1000 tables that have the table mode set to Preserve Destination.
• Improved how Tonic handles maximum lengths when it generates the following data types:NCHAR, NVARCHAR2, CHAR, VARCHAR2

PostgreSQL

• Batch sizes are now set dynamically based on the average row size.
• Memory improvements for PostgreSQL data generation that involve large rows.

Spark

• Improved support for foreign keys.
• Added support for Apache Livy and HDFS.
• Improved performance and added SDK support for the Integer Key generator.

SQL Server

• Eliminated duplicate default constraint URNs during database creation.

Refreshed the Audit Trail user interface on Privacy Hub. The new Protection Audit Trail provides a paginated list of the updates to the sensitivity designation and generator assignments.

Deep links now work correctly when you use Google SSO to authenticate.

Error messages from Oracle are now displayed in response to invalid where clauses in subset configuration.

Made minor memory improvements to the Address generator.

Snowflake

• Reduced the frequency of schema change detection on Snowflake databases. This can result in cost savings on Snowflake clusters, because the clusters can sleep more often.
• Snowflake generation now works correctly when there are foreign key constraints.

SQL Server

• Added support for security policies, sequences, check constraints, and system versioned temporal tables.

Improved cross-tab support for automatic logouts when you configure an inactivity period.

The Update option in the actions menu now takes you directly to the System tab on the Admin Panel instead of the Users tab.

Corrected the password length requirement to be 12 characters or greater instead of greater than 12 characters.

Improved the estimated row progress for scaled tables.

Eliminated a race condition that occurred when applying constraints.

Databricks:

• Can now run concurrent jobs that use different versions of Databricks.
• Tonic now supports ORC and Hive tables in Databricks.

SQL Server:

• The Categorical generator can now support more than 2 billion rows in a category.

Features

For PostgreSQL databases, Tonic now supports name and char data types.

For Tonic single sign-on, Tonic now supports Azure Active Directory.

From the administration screen, administrators for customers that run Tonic on Kubernetes and Docker can now download logs from all containers that run Tonic.

For Spark-powered integrations, Tonic now supports the Address generator as a sub-generator.

Bugs

Improved performance for:

• Random Hash generator on Spark and Databricks
• UUID Generator on Spark
• Subsetting

Improved the browsing experience on low resolution displays.

When users create a new password, Tonic now displays a panel with the password requirements, and indicates whether the password meets those requirements.

Improved the parallelization and concurrency for processing foreign key constraints.

Databricks and Spark EMR

• Improved the performance of the Noise generator.

MongoDB

• Improved display of longer key values in the Key column. Widened the column and added truncation.
• Optimized queries against the Tonic database.

Features

Add data type advanced search for Oracle, Amazon Redshift, and BigQuery

Performance improvements for constraint restoration

Improved performance of Random Integer generator on Spark

Bugs

Fix issue on SQL Server preserve destination mode when table names contain a ".".

PostgreSQL: Fix issue copying arrays with trailing slashes

Features

Skip batch instead of failing generations on Postgres in some cases when values fail to be inserted

Improvement to Synthesis Reports for AI Synthesizer generated data

Changing Lambda deployment from ECR to ZIP for integrations using Lambda functions. Images no longer need to be manually deployed to ECR.

Bugs

Fix for One-Click Tonic update when the PostgreSQL application database is deployed in a Docker container

Features

Support for arrays with Struct Mask generator on Amazon EMR/Spark and Databricks

Bugs

Fix issue causing privacy scans on Dremio to fail

Minor UI fixes to workspace page

Fix for data generation issue on Db2 iSeries when destination is empty

More robust handing of Databricks host URLs