Tonic Structural release information

Setting the default configuration for generators (Requires an Enterprise license)

The Generator Presets view allows you to configure the default configuration for generators. The current configuration is used whenever that generator is assigned to a column.

To update the configuration, you must be an owner or editor of a workspace in the instance.

In the configuration for a column, you can override the saved default configuration, which we call the baseline configuration. You can also revert to the current baseline configuration or save your configuration as the new baseline configuration.

Other updates

On the workspace management view, the workspace options are now included in the collapsed version of the heading.

For free trial users, the option to create a workspace is always visible in the Tonic heading.

For SAML SSO, if the value of NameID is not an email address, Tonic uses the email claim in the SAML response.

Databricks

• Added a TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE environment variable to determine whether new Databricks tables ErrorOnOverride by default. The default value is true, indicating that new tables ErrorOnOverride.

SQL Server

• Changed the default value of BYPASS_MS_XML_PARSING from false to true. The variable indicates whether to convert XML columns to nvarchar(max) to avoid potential XML parsing bugs.

NOTE: Releases 741 and 742 were removed.

Graph View for subsetting

The Configuration tab of Subsetting view now includes a toggle to switch between Table View and Graph View.

Table View is the existing tabular list of tables.

Graph View is a diagram view that displays the tables and the relationships between them. Similar to Table View, when you click a table in Graph View, the table details panel displays for that table. You can configure subsetting from either view.

On Graph View, Tonic adds a marker when the subset configuration for a table changed since the last subsetting data generation. A table might be added to the subset, removed from the subset, or modified within the subset.

Tonic data encryption

The new Tonic data encryption feature, available for Professional and Enterprise licenses, allows you to set up a configuration to decrypt source data before applying a generator, encrypt transformed data before writing it to the destination database, or both.

Tonic data encryption uses AES encryption.

Tonic data encryption requires you to set environment variables for the decryption key (TONIC_DATA_DECRYPTION_KEY) and encryption key (TONIC_DATA_ENCRYPTION_KEY). Both keys must use the same key size - either 128, 192, or 256 bits.

Admin users configure Tonic data encryption from the Data Encryption tab of the Admin Panel.

When you enable Tonic data encryption, the generator configuration includes a setting to indicate whether to use it for that column.

Other updates

Fixed an issue where navigating to Database View from Schema Changes view for a table-specific issue did not apply the correct filters.

Fixed an issue where workspace import could inadvertently add multiple subset targets in the same table.

Added support for ssh-rsa for data connections.

Improved error logging for the Timestamp Shift generator.

Fixed the display for the import workspace dialog.

MongoDB

• The new ObjectId Key generator allows you to de-identify object ID columns.
• Moved the scanning of MongoDB collections into the Tonic worker.

PostgreSQL

• Fixed an issue that caused data generation to fail with the error canceling statement due to statement timeout.

SQL Server

• Fixed an issue where a bug in the SQL Server Management Objects (SMO) library caused data generation to fail when using partitioning in Azure.
• Fixed an issue that caused data generation to fail when system tables were included.

When you log back in to Tonic, it now displays the workspace management view for the most recently viewed workspace.

Improved error messaging for rarely occurring user authentication issues.

Fixed display issues on the Subsetting view where part of the subset results graph did not display and the Configuration tab did not scroll.

Amazon Redshift

• Added the ability to provide filters on tables.

MongoDB

• Fixed an issue where indexes created with an expireAfterSections value are not created in the destination database.
• Fixed a regression where unscanned collections did not display in workspaces.
• Fixed an issue that prevented Tonic from being able to connect to MongoDB-compatible DocumentDB databases.
• Fixed an issue where DocumentDB jobs failed when run from Windows.

MySQL

• Updated to ensure that TONIC_WRITE_PARALLELISM is always 1, to prevent lock timeouts.
• Increased resilience around possible failures when retrieving a MySQL schema during data generation.

Snowflake

• Added the ability to provide filters on tables.

SQL Server

• Improved performance when reading large fields.
• Fixed an issue that caused data generation to fail with the error Sequence contains no matching element.
• Fixed an issue that caused data generation to fail on SQL Server 2014.

As of V724, the AI Synthesizer generator is by default not available. To enable the AI Synthesizer generator, set the environment variable TONIC_NN_GENERATOR_ENABLED to true. If the AI Synthesizer generator is assigned in a table, but the environment variable is false, then data generation fails.

During subset processing, when Tonic encounters a circular dependency, to break the circle, it nullifies the values of one of the foreign key columns. By default, it nullifies all of the foreign key column values. A new environment variable, TONIC_SUBSETTING_AGGRESSIVELY_NULL_CYCLICAL_FKS, controls the behavior. By default, the value is true. If you set the value to false, then Tonic only nullifies foreign key values that do not exist as primary key values in the other table.

Fixed an issue where authentication using Azure SSO failed if the user was a member of a large number of IdP groups.

Improved detection during sensitivity scans of columns that contain birth dates.

MongoDB

• Fixed an issue where adding a new null type field to the data caused data generation to fail.

Oracle

• Fixed an issue where data generation returned the following error: Specified argument was out of the range of valid values. (Parameter 'minSize').

PostgreSQL

• Fixed an issue where tables with generated columns, but no generator assignments, were not correctly copied to the destination database.
• Improved handling and messaging for connection timeouts.

Snowflake

• Fixed an issue where retrying a connection multiple times returned an incorrect error.

SQL Server

• Fixed issues where users could not configure data generation for tables that contained hierarchyid columns.
• Fixed an issue that prevented Preserve Destination from being used on schema-bound tables.
• Fixed an issue that prevented Incremental and Preserve Destination table modes on memory-optimized tables.
• Fixed an issue where column attributes such as the nullability of a column were not correctly reflected in the destination database.
• Fixed an issue where schema binding was not maintained in the destination database.

Redesigned Tonic navigation

As of V719, we have redesigned the Tonic navigation. The left navigation menu is removed. On Workspaces view, click the workspace name to display the workspace management view for that workspace. The workspace management view contains a horizontal navigation bar to provide access to the workspace configuration and generation tools, and a heading menu to provide access to other workspace actions.

On Workspaces view, the first column is now a checkbox to select workspaces to which to apply an option from the Actions menu.

The Tonic application header contains links to Workspaces view and the Admin Panel.

New table filtering for Google BigQuery

As of V717, Google BigQuery supports table filtering. On the table mode selection panel, for tables that use De-Identify mode, you can provide a WHERE clause to filter the records that are included in the destination data.

Other updates

Fixed an issue where Tonic workers did not start after an upgrade when a read-only file system was mounted for custom value processors.

Improved the detection of vehicle identification numbers (VINs) in source data.

Oracle

• Improved generator recommendations for columns that are part of a compound unique constraint.

SQL Server

• Fixed an issue that caused jobs to fail when none of the tables were assigned De-Identify mode.

Fixed an issue where when subsetting used parallel processing, the subsetting steps displayed incorrectly.

Improved the error message that is displayed when unsupported data types are present in a table.

On the Subsetting view, on the row count popup, provided a clearer explanation when the destination row count is larger than the source row count.

Fixed an issue where simultaneously updating workspace permissions for multiple users failed to apply the updates.
Made some small performance improvements to the Regex Mask and Array Regex Mask generators.

Improved the job cancellation logic to ensure that selecting the cancel option actually cancels the job.

When a SAML SSO login is initiated, Tonic now redirects the browser to the correct URL.

When Tonic detects an invalid or deprecated generator, it no longer returns the error message Unexpected generator id {generatorId}, create a dedicated Metadata class for this generator.

MySQL

• Fixed an issue that caused lockouts and timeouts on the destination database.

Oracle

• Reduced the number of destination database permissions that are required for data generation.

PostgreSQL

• When you use the Limit Schemas feature, and you choose to filter to only include specific schemas, Tonic no longer includes all of the extensions in the database. It only includes extensions that are related to the included schemas.
• Values are now truncated to fit the char type before they are written to the destination database.
• Generators that support consistency can now be made consistent with columns that are user-defined enum values.
• The Conditional generator can now use columns that are user-defined enums as conditions to apply sub-generators.
• Tonic now correctly determines whether to truncate a unicode string when fitting the string into a data type that is close in size.
• Improved resiliency to destination database connection failures.

NOTE: Releases v692 through v695 were removed.

Update to PII detection - PII container removed

As of v697, updated how Tonic performs PII detection. The name detection process now also scans for international names with origins from 103 different countries.

The PII detection process now runs from the Tonic worker instead of in a separate container.

To avoid errors when you upgrade, future versions up to v999 will have a placeholder PII container. The container is not active and is not used to run Tonic.

Before you upgrade to V1000 or later, you must remove the PII container from your instance.For a Docker deployment, in docker-compose.yaml, remove the tonic_pii_detection section.

For a Kubernetes deployment:

• Remove the file tonic-pii-detection-deployment.yaml.
• Remove the file tonic-pii-detection-service.yaml.
• Update tonic-worker-deployment.yaml to remove the entry for TONIC_PII_DETECTION_URL.
• Update values.sample.yaml to remove the entry for pii_detection.

If you do not remove the PII container before you upgrade to v1000 or later, then when you upgrade Tonic, you will encounter image pull errors.

Other updates

Improved error messaging on the Tonic UI for failed data generation jobs.

Reduced the telemetry logging for PII detection during sensitivity scans to reduce the performance impact on self-hosted instances with machines that do not have internet access.

Fixed an issue that prevented users from configuring the Random Timestamp generator Date Format when the generator was selected as a sub-generator of the JSON Mask and other composite generators.

Made a minor correction to a data generation log message.

Reduced Tonic slowdowns caused by telemetry.

MySQL

• Fixed an issue that caused data connections to fail with the error The ConnectionString property has not been initialized.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

Oracle

• Fixed an issue where long column names caused data generation to return the error ORA-00972: identifier is too long.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

PostgreSQL

• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

SQL Server

• Users can now assign generators to timestamp and rowversion columns.
• Fixed an issue where a database ID mismatch in the master and database tables prevented data generation from starting.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

NOTE: These releases were removed.

Enhancements

In a Databricks workspace, you can now choose to write all of the output tables to one of the following formats: Avro, JSON, Parquet, Delta, CSV, ORC. This setting replaces the previous option to write all of the output tables to Databricks Delta.

When sharing a workspace, free trial users can now invite other users with the same corporate email domain to start their own free trial.

Other updates

Added messaging to the Tonic application about changes to the Tonic license plans.

Fixed an issue with the JSON Mask generator where when users deleted a sub-generator, a different sub-generator was deleted.

Fixed cases where data generation jobs remained in the queued state indefinitely.

Fixed a performance regression that affected workspace loading.

Oracle

• Fixed an issue where a data generation fails when there are index-organized tables.

NOTE: Releases v684 and v685 were removed.

Enhancements

When creating new virtual foreign keys, you can now use the top level field name check box to select or deselect all of the fields that have that name.

Other updates

Fixed an issue where a warning did not display correctly when a subset table was configured with an invalid table mode for subsetting.

For the Custom Categorical generator, we no longer treat newlines as empty strings on numeric columns. Newlines are still treated as empty strings on string columns.

Fixed an issue where an error occurred when a column is assigned the Custom Value Processor generator.

Fixed an issue where data generation with subsetting failed with the following error: Could not load type from assembly 'Allos.Generators'.

Redesigned the generator selection dropdown to better separate the suggested generators from the other applicable generators.

Oracle

• Improved error logging during data generation.

SQL Server

• Fixed an issue where views that reference preserved tables (table mode is Preserve Destination) were not created.
• Added verification to ensure that all of the required tables and objects are created in the destination database before data generation.

Enhancements


Tonic now displays warnings at 30, 15, and 7 days before a Tonic license expires.

Other updates

After a one-click update of Tonic, containers for Docker Compose customers now include the version number in the name.

Addressed an issue for Docker Compose customers where Tonic did not restart properly after a one-click update.
Tonic can now download SAML IDP metadata from a URL. To configure the URL, set the environment variable TONIC_SSO_SAML_IDP_METADATA_XML_URL.

Added helper text to indicate the value format for a database server.

Added the ability to configure the SAML request issuer. To configure the issuer, set the environment variable TONIC_SSO_SAML_ENTITY_ID.

Improved error messaging for WHERE clause validation in subsetting configuration.

Moved some of the temporary files used for data generation from /tmp to /tmp/tonic.

MySQL

• Fixed an error where data generation occasionally failed with the error "Allos.Core.Exceptions.TonicException: No databases selected to overwrite".

Oracle

• Updated to ensure that system tables are ignored.

PostgreSQL

• Fixed an issue with data generation for partition tables.

Enhancements

New command-line tool for Tonic installation - Tonic now offers the Tonic Installation Manager (TIM), a command-line tool to deploy Tonic on either Amazon EKS or a VM.

Other updates

Updated how Tonic performs PII detection. Added additional name values to the information that the detection process looks for.

Fixed an issue where navigating to the workspace edit page sometimes threw an error.

Fixed an issue where SSO account creation erroneously returned errors even though the account was created successfully.

Improved our generator recommendations that are based on the security scan to prevent memory issues for larger databases.

Improved error messaging when Tonic-hosted users attempt to connect to a database on a local network.

Fixed an issue where job log timestamps displayed the wrong month value.

Amazon Redshift

• Corrected a permissions check issue that caused connection tests to fail.MySQL
  
  
  • Upgraded MySqlConnector to version 2.2.5.
  • Fixed an issue where subsetting jobs failed with a "No database selected" error.
  Oracle
  
  
  • Fixed an issue where data generation jobs did not stop when they were canceled.
  • Fixed an issue with applying the Event Timestamps generator to timestamps that have timezone values.
  • Update to prevent Tonic from running out of space during subsetting.
  • Fixed an issue where materialized views and materialized view logs are not torn down after data generation.
  • When TONIC_ORACLE_SKIP_CREATE_DB is true, we now properly truncate tables.
  PostgreSQL
  
  
  • Upgraded the Npgsql library to version 7.0. This is used for both workspace data and the Tonic application database.
  Snowflake
  
  
  • Corrected a permissions check issue that caused connection tests to fail.

Enhancements

New endpoints and expanded API documentation for generators - For generators, the generated Swagger documentation now contains descriptions for each endpoint and for all of the model properties. You can send a request to retrieve a list of available generators. You can now update the generator configuration for a single column, instead of having to provide the configuration for an entire table. There is also a new endpoint to revert a column to Passthrough.

On the Subsetting page, you can now sort the tables based on whether they are in or out of the subset.

Other updates

Improved UI support for timeouts when loading schema information for a source database.

Updated the subsetting process to ensure that a small percentage-based target table contributes at least one row to the subset.

The environment variable TONIC_SUBSET_PARALLELISM is deprecated. Tonic now uses the environment variable TONIC_TABLE_PARALLELISM to control parallel processing for subsetting.

When a data preview or workspace loading process is no longer needed, such as when there are network issues or the user leaves the application, Tonic now attempts to cancel the process.

MySQL

• Improved error messaging when a database connection times out.

Oracle

• Fix to ensure that data generation fails when drop table statements fail.

PostgreSQL

• Improved error messaging when a database connection times out.

SQL Server

• Fixed an issue where some tables were not created in the destination database.
• Improved error messaging when a database connection times out.

EnhancementsTonic now supports SAML SSO providers.

Other updates

Fixed an issue where containers do not start and the following error message is returned: "The directory named as part of the path /var/run/supervisor/supervisord.pid does not exist"

Corrected an issue where in some cases when the job failed because of a source connection issue, the job status was not set to Failed.

In the subsetting configuration, you can now configure a target table percentage value with up to 3 decimal places.

Resolved a potential database connection leak when canceling data generation jobs.

Improved error messaging when a deleted SSO user tries to log into Tonic.

Improved error display for subsetting WHERE clause validation.

Fixed an issue where running a query to validate a subsetting WHERE clause could cause the application to slow down.

Fixed an issue where jobs could fail when subsetting parallelism was enabled.

The Conditional generator now allows you to configure a selected sub-generator to be consistent with other columns. The selected generator must support consistency with other columns.

Databricks

• Updated the Databricks job names to use the format tonic_workspaceName/workspaceId + jobID.
• Improved performance for page loads.

Google BigQuery

• Added support for the JSON data type.

MongoDB

• Fixed a minor flickering issue on the Collection View.

MySQL

• Improved logging of errors that produce data generation warnings.

Oracle

• Corrected an issue where we displayed foreign keys that referenced a schema outside of the workspace.

PostgreSQL

• Fixed an issue with handling HStore types during data generation.
• Improved logging of errors that produce data generation warnings.
• Improved management of partition tables.

Spark

• Improved logging for data generation.

SQL Server

• Fixed authentication handling for Active Directory. Tonic supports NTLM v2 authentication.

Enhancements

From the Subsetting view, you can now view the table configuration and results summary for previous subsetting data generation runs. On the Previous Runs tab, you select the run to view the details for. Note that you can only view the details for jobs that you run after you upgrade.

Other updates

Corrected an issue for instances using a Docker version before 20.10.10 that prevented Tonic processes from starting after an upgrade from version 642 or below to version 643 through 647.

Corrected an issue where estimated data generation time was logged as a negative number.

Improved the performance and resiliency of the subset WHERE clause validation.

Improved the cleanup of database connections after they are used.

Improved error message when the source database version is newer than the destination database version.

Google BigQuery

• Fixed a user interface issue that prevented a new workspace from being created.
• Tonic now supports post-job scripts in BigQuery workspaces. Post-job scripts run inside of transactions. They are limited to statements that are supported within transactions.

Oracle

• The minimum supported version for the Oracle data connector is now Oracle 12c. Tonic no longer supports 11g. As a result of this change, the Oracle Helper is no longer needed. After upgrading, you should remove Oracle Helper from the list of containers to fetch.

Snowflake

• For database connections, the server name should not include https://. Tonic now detects and removes https:// if it is entered.

Spark

• Eliminated a vulnerability that could allow users to see the results of statistics jobs that they did not have access to.

SQL Server

• Fixed an issue with loading tables in SQL Server 2012 and below

Enhancements

Overriding table mode and generator configuration in child workspaces - The workspace inheritance feature now allows child workspaces to override the table mode and generator configuration from the parent workspace. Database View and Table View indicate when the parent configuration is overridden, and provide options to reset the configuration.

Other updates

You can now assign the SIN generator to fields that have uniqueness constraints.

The Address generator now only allows you to link columns that contain the following types of address values - City, City State, Country, Country Code, State, State Abbreviation, Zip Code, Latitude, Longitude.

Improved memory handling for subset processing of downstream tables.

On the Subsetting view, when the database returns an unrealistic value for the subset size, we replace the value with a warning message.

When using Helm to deploy Tonic via Kubernetes, integer environment variable values that are longer than 6 digits might be converted to scientific notation. To avoid this issue, Helm recommends that you enter values in quotes. Tonic can now parse scientific notation to better handle this behavior for values that are not in quotes.

Fixed an issue where linking columns that have the same name as columns in another table caused the generators to be removed from those other columns.

Made minor memory improvements throughout the Tonic application.

Corrected an issue where Tonic displayed a "subsetting in progress" message before the processing started.

Amazon Redshift

• Fixed an issue where data generation failed when the name of one table was part of the name of another table (for example, Test and Test1).

MongoDB

• Generators are now applied correctly to documents that are nested in arrays.

Oracle

• Removed the permissions check for reading truncated tables.
• Fixed an issue where DBLinks created for data generation were not removed when the generation failed or was canceled.

Snowflake

• The Categorical generator no longer returns errors when applied to binary, time, and date columns.
• Improved the performance for loading the user interface when the data contains a large number of foreign keys.

Spark

• The Audit Trail entries on Privacy Hub now display the correct table and column names.
• Corrected an issue where sub-generators for the Conditional generator were not applied correctly.
• Improved error messages when testing the connection to a Databricks database.

Improved user signup experience for the hosted version of Tonic.

Made some small styling updates to Database View. Removed the dropdown to uncheck or check all of the tables. Removed the highlighting on the table mode dropdown for tables that contain columns with assigned generators. Changed the Filter button labels to Filters.

For subsetting, verify that primary key fields are not assigned a non-primary key generator.

Amazon Redshift

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

MongoDB

• Fixed an issue that caused errors when applying the Timestamp Shift generator.

Snowflake

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

Spark

• Improved error messaging for unsupported versions.

SQL Server

• Improved handling of alphanumeric sequences.
• Case-sensitive databases now connect correctly to Tonic.
• Corrected how indexing recommendations for subsetting are determined.

Enhancements

Added filters for upstream subset records - In the subsetting configuration, for upstream related tables, you can now filter the records to include based on either a date value or a WHERE clause. Upstream tables contain data that has a foreign key that references a primary key in a target table. Upstream records are optional, and are not needed for referential integrity.

The Confirm Generation panel now provides access to tips to improve data generation performance.

Other updates

Tonic no longer offers DB2 as a data connector type.

On the Job History list, changed the job type "Privacy Scan" to "Sensitivity Scan".

Corrected a display issue in Privacy Hub where the column details panel extended past the bottom of the page.

For the JSON Mask generator, the path expression selection tool now works for arrays and for keys that contain spaces or special characters.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Added a message to notify users when queries used to validate a subset WHERE clause run for a long time.

Tonic now continues to record logs when a job fails.

Improved memory usage when running data generation.

Amazon Redshift

• Fixed an issue where Tonic failed to display an error for an invalid generator assignment.

MongoDB

• Corrected a subsetting issue that occurred when multiple foreign keys in one collection referred to the same primary key in another collection.

MySQL

• Updated the MySqlConnector driver to 2.2.0.

Oracle

• Downgraded the Oracle drivers to 3.21.70 after encountering issues with 3.21.80.
• Corrected an issue with the Event Timestamps generator.

PostgreSQL

• When executing psql and pgdump commands, we properly enforce the requirement to use SSL for connections.
• Corrected connection issues by escaping special characters in password files.

Snowflake

Spark

• You can now apply the Timestamp Shift generator to string fields that contain datetime values.

Enhancements

Tonic can now integrate with Keycloak for SSO authentication.

Other updates

Fixed an issue where users received the error HTTP/1.1 415 Unsupported Media Type from the /api/GenerateData/start endpoint.

Improved performance for the /api/job endpoint. As part of this update, the endpoint only returns the most recent 100 jobs.
Improved loading time for the Job History view, which now displays only the 100 most recent jobs.

Updated how Tonic determines whether to use SSL for connections.

The Add Foreign Key Relationships tab on the Foreign Keys view now expands to fill the height of the browser.

Fixed an issue where a failed cleanup step incorrectly marked a Tonic update as failed.

Amazon Redshift

• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.

MongoDB

• When performing a schema scan, Tonic now displays a progress indicator.
• Improved subsetting performance.
• Fixed an issue where subsetting failed because of duplicate keys.
• Corrected how Tonic handles views.

Snowflake

• Fixed an issue where the Tonic user interface could not load when the database included foreign keys.
• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.

Enhancements

If data generation is not blocked on all schema changes, Tonic now displays a dismissible warning when there are non-conflicting schema changes. Conflicting schema changes always block data generation.

For the GenerateData API endpoint, added an optional clientResourceId query parameter. When you provide a value, then jobs that have the specified clientResourceId run serially instead of in parallel. The check applies to all jobs across the instance, regardless of whether they belong to the same workspace.

Other changes

Fixed an issue that made it difficult to click the Cancel Job button in the Job History list.On the workspace configuration view, the source and destination database details are now populated correctly when you refresh the page.

When granting access to a workspace, improved how we display long names and email addresses.

For tables that use Scale mode, removed Passthrough from the generator selection dropdown list. Previously, the option displayed in the dropdown list even though it couldn't be selected.

Improved performance for looking up tables and columns.

Corrected an issue where the same environment variable with different cases caused Tonic to crash.

When logging in using single sign-on (SSO), when the email address uses a different case from an existing username-password account, it now resolves to the same user.

For instances deployed using Docker Compose, Tonic now cleans up old, unused images.

Improved performance when running generators on de-identified tables.

Corrected an issue where the Event Timestamps generator produced unexpected values for linked columns.

Improved error handling for account creation.

MongoDB

• Updated MongoDB.Driver and MongoDB.Bson to 2.18.0.
• Subsetting jobs with foreign keys can now run on collections that have lower permissions. To enable this, set the environment variable TONIC_BYPASSDOCUMENTVALIDATION_ON_DOWNSTREAM_KEY_MERGES to false. The default value is true, which means that the destination database requires the dbAdmin role in addition to readWrite.

Oracle

• Tonic now automatically uses the optimal table prefix for schema queries. The TONIC_ORACLE_DICTIONARY_TABLE_PREFIX environment variable is removed.
• Improved error handling for missing permissions for table size queries.

SQL Server

• Improved performance for table queries.

Enhancements

The Subsetting view now shows for each table the percentage of data that is included in the destination database.

Other updates

The workspaces view no longer briefly flashes a message indicating that the workspace cannot be found.

Added the ability to display trace information in the log files. To enable the trace information, set the environment variable TONIC_LOG_TRACES to true.

Fixed an issue that caused an incorrect warning to display when linking columns that were assigned the Custom Categorical generator.

Fixed a data type error in AI Synthesizer for models that only contain categorical data.For the Event Timestamps generator, Tonic now prevents the generator from being assigned to a time-only value.

Amazon Redshift

• Added ServerCompatibilityMode to the Redshift connection string to prevent connection errors.

Databricks

• Added support for Databricks 10.x.

MongoDB

• Fixed an issue where Tonic generated collection names in Preserve Destination mode that exceeded the maximum length limit.

MySQL

• The MySQL data connector now supports the ability to use the DELIMITER command in post-job scripts.

Oracle

• Improved performance for schema queries.
• Improved performance for databases without database links.
• Restored missing information for loader errors.
• Improved performance for applying indexes and constraints.
• Added the environment variable TONIC_ORACLE_REDO_LOG_ENABLED, which by default disables recovery information writes to REDO LOG files.
• Improved performance for retrieving tables and columns.
• Fixed an issue with loading custom types.
• Fixed an issue where multiple indexes on the same column caused data generation to fail.
• Corrects a regression from V611.
• When fetching constraints, Tonic can now support primary key indexes that have the same name.
• Excluded views from table queries.

PostgreSQL

• Updated the process of loading tables to improve performance and eliminate an error indicating that PostgreSQL types could not be found.

SQL Server

• Fixed an issue where Preserve Destination and Incremental table mode could not be used in tables with foreign key relationships.
• Tonic now provides a more informative error when data generation fails because the database is not accessible.
• Tonic now properly handles identifiers with single quotes.
• Fixed an issue that caused a worker process to crash when using Azure to connect to the database.
• Added additional logging of the available databases.
• Improved handling of schema and table names in privacy scans.

Enhancements

You can now generate DEBUG level logs for the Tonic API. To do this, set the environment variable TONIC_CONSOLE_LOG_LEVEL to DEBUG.

Tonic now supports logging for long-running queries. The environment variable TONIC_LONG_RUNNING_QUERY_LOGGING_INTERVAL provides the interval in minutes for logging queries. By default, Tonic generates a log entry for a long-running query every 10 minutes. To see this information, TONIC_CONSOLE_LOG_LEVEL must be set to DEBUG.

Other updates

Terminology change - In the Tonic documentation, we have changed the term "mask generator" to "composite generator".

Fixed an issue that prevented the use of Preserve Destination and Truncation modes on tables and collections that had names that were close to the maximum length limit.

You can now assign the Integer Key generator as a sub-generator for string values in composite generators.

For Google SSO, fixed an issue where users who did not have a group membership could not access Tonic.

Improved performance for format-preserving encryption (FPE), which is primarily used for key generators.

Improved generation performance for tables that use De-Identify table mode.

Improved error display when reading data from the source database.

Fixed an issue where values for columns that were assigned the Passthrough generator were being masked with 1s and 0s.

The User Settings page no longer displays the password change option for SSO users.

For the JSON Mask generator, fixed an issue where a large matching value caused the buttons to move off of the configuration dialog.

Fixed an issue where data connection pooling caused data generation to fail because of colliding queries.

Google BigQuery

• Fixed an issue where source data was not loading properly into the Tonic user interface.

MongoDB

• Fixed an issue where the generator configuration panel closed unexpectedly after making edits.

Oracle

• Added a warning when a generator produces a value that is truncated in the destination database.
• When Tonic retrieves table sizes, the results now include the size of the associated large objects (LOBs).
• To help with troubleshooting, Tonic can now determine which Oracle patches were applied to a database.
• Fixed an issue that caused errors when reading a large object array.
• Added additional checks to the data connection tests.
• Updated our Oracle Driver to 3.21.80.
• Fixed an issue where primary keys could not be copied to another index.
• Additional performance improvements.
• Corrected how we check the current user's database privileges.

PostgreSQL

• Added an option to not refresh materialized views. To turn off the refresh, set the environment variable TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS to false.
• Fixed an issue with parsing PostgreSQL schemas in Windows environments.
• Fixed an issue where data generation failed when the data contained read-only arrays.

Snowflake

• Fixed an issue where data generation failed when the schema definition (DDL) contained row access policies.

SQL Server

• Added permission checks to the data connection tests.
• Fixed an issue where Tonic processes failed.

Improved the performance of Table View, particularly when scrolling.

Reworked data generation to better group generator errors.

On the Subsetting view:

• Improved accuracy of the row counts for small tables.
• The table list shows whether each table is in the subset and the number of pre-subset and post-subset rows even when subsetting is not enabled for data generation.

The health check for the PyML container can now use HTTP instead of HTTPS.

MongoDB

MySQL

• Improved performance when loading workspaces.

Oracle

• Fix to improve query performance.
• Added permission checks when testing database connections.

PostgreSQL

• Tonic now applies the source database check constraints on the destination database after data generation is complete.

Snowflake

• Fixed an issue where data generation failed when there were tables that used Preserve Destination table mode.

Reverted an update from v580 that caused slow performance when retrieving the list of tables for a workspace. In some cases, this caused Tonic to indicate that no tables were available.

Fixed an issue where custom value processors interfered with the display of configuration options for the SSN generator.

Fixed an issue where the dropdown lists to select a column generator periodically scrolled to the top of the screen.

Improved performance when navigating among Tonic views.

Improved the list of suggested sub-generators for the Conditional generator.

Improved the data generation process to prevent jobs from hanging when an error occurs.

Fixed an issue where the post-subset generation row count always returned 0.

Updated the Tonic logging framework.

Google BigQuery

• Improved handling of timestamp and datetime fields.

MongoDB

• Fixed an issue where workspaces crashed when running data generation on new collections.

MySQL

• Corrected the handling of the BIT data type.
• Improved performance when loading workspaces.

Oracle

• The Constant generator now works correctly for RAW columns.
• Stopped using source database constraints that were in the recycle bin to populate the destination database.
• Improved performance when retrieving schema information and loading workspaces.
• Added the OracleDriverAnalyzer tool to analyze Oracle performance.

PostgreSQL

• Display an error when an excluded or included schema does not exist in the source database.
• Fixed a permissions error that occurred when creating a new workspace using Azure PostgreSQL for the destination database.

Spark

• Fixed an issue with the workspace configuration view that prevented users from creating and updating Spark-based workspaces.

SQL Server

• Fixed an issue that caused a data generation error when XML indexes had dependencies on other indexes.
• Upgraded SqlClient to 5.0.0.

Enhancements

Assigning generators from Schema Changes view - On Schema Changes view, new columns, changes to column data type, and changes to column nullability have a Select dropdown list that includes an option to assign a different generator to the column and then resolve the issue.

Complete list of blocking issues for data generation - When data generation is blocked, the generation panel now displays all of the blocking issues. This allows you to correct all of the blocking issues before you attempt to run data generation again.

Other updates

Made some visual updates to the Tonic navigation pane and the Tonic login panel.

On the Job History page, the details popover for queued jobs now points to the correct job.

In Database View, the generator list for the Applied Generators filter is now correctly alphabetized.

Fixed an issue where the dropdown arrow for the table mode selector was not always clickable.

When importing a workspace, Tonic now validates that columns do not have multiple generators assigned to them.

Updated to provide clearer error messages when there is an issue with an assigned sub-generator.

Fixed an issue where Privacy Hub sometimes did not reload after a new sensitivity scan.

The database type filter now includes all of the available Spark database types.

Corrected the link to the Tonic privacy policy.

Fixed an issue where jobs failed when multiple tables with the same name in different schemas were assigned Preserve Destination table mode.

Timestamp Shift is now the recommended generator for Date and Timestamp columns.

Corrected the display of available buttons on the Tonic application.

MongoDB

• Improved error handling for database summary queries.
• Added a warning about subsetting performance for percentage subsetting target tables on MongoDB versions before 4.4.2.
• Fixed a memory usage issue with loading Collection View when the data contains large arrays.

Oracle

• A new environment variable, TONIC_ORACLE_DATA_PUMP_PARALLELISM, allows you to choose the maximum number of threads for parallelization for Oracle Data Pump.
• Optimized queries for better performance.

Spark

• Added the ability to specify the proxy user when using Livy.
• Fixed an issue where users could change the configuration type when editing a Spark workspace.

Enhancements

Subsetting results - The Subsetting view now displays the results of the most recent subsetting data generation run. The information includes:

• When the job began and ended.
• The percent reduction from the original source data to the subset destination data.
• The percentage of the source data that is included in the subset destination data.
• The volume of data in the source data and the subset destination data.

Schema changes filter on Database View - On Database View, the advanced filters now include an option to only display columns that have unresolved schema changes. This filter is not combined with other filters. When you filter for unresolved schema changes, the other column filters are disabled.

Larger WHERE clause editor for subsetting - For WHERE clause target tables, you can now display an editor with a larger text area for entering the WHERE clause. This provides better support for longer, more complex WHERE clauses.

Other updates

Fixed an issue that prevented users from deleting more than one tag from the Edit Workspace view.

Added the ability to run Tonic workers, the Tonic web server, and Tonic notifications on Heroku.

Updates to improve handling of canceled jobs, both when users cancel jobs and when jobs fail.Improvements to data generation memory handling and performance.

Improved the Synthesis Report for AI Synthesizer.

Fixed an issue where adding constraints to a destination database resulted in deadlocks.

Oracle

• Update to allow the JSON Mask generator to be used on user-defined types (UDTs).

PostgreSQL

• Fixed a regression that caused sequence fetching errors in v9.6 and earlier.

Spark

• Improved performance for the Continuous generator.
• Enabled partition filter validation on Hive.

SQL Server

• Added support for full text catalogs in Server Management Objects (SMOs).
• Fixed an issue to enable the correct handling of schema, table, and column names that contain single quotes.
• Made a fix to correctly display error messages.