Tonic Structural release information

NOTE: Releases v849 through v854 are removed.

Removed caching of AWS credentials.

For the beta Data Pipeline V2 job processing (available for PostgreSQL only):

• When a job fails, Tonic no longer tries to fall back to the current job processing. In most cases, jobs fail for reasons that are not connected to the processing type. Falling back to the current job processing is not effective.
• Improved performance for subsetting.
• Adjusted the logging level for telemetry-related log messages to DEBUG.

On the Foreign Keys view, when you filter the keys, click the select all option, and then clear the filter, only the matching keys are selected. Previously, the select all option always selected all of the keys.

Fixed an issue where CSV files could not be uploaded to data science mode workspaces.

You can now configure parallelism for sensitivity scans. For relational databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_RDMBS, and the default is 4. For document-based databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_DOCUMENTDB, and the default is 1.

On the table configuration panel for subsetting, Tonic no longer displays a count of post-subset rows before a subset is generated.

Fixed an issue where subsetting data from one workspace appeared in a different workspace.

You can now use UUID columns in the conditions for the Conditional generator.

Fixed an issue where when you deleted a linked column from a column configuration panel, the other linked columns were deleted.

The Timestamp Shift generator can now be assigned to columns where the values use the date format MMddyyyy.

Fixed a regression that caused NPGSQL logging to occur even when it was disabled.

Google BigQuery

• Improved error handling when rows contain invalid data. Tonic now provides a method to look at the data that caused the error. Fixed the handling of rows to prevent errors on certain data.
• Fixed how we create views in the destination database.

MongoDB

• Fixed an issue where Tonic was unable to get the schema for the source data.

Oracle

• Fixed an issue where retries of Oracle commands from transient errors failed.

Snowflake

• Improved resource handling during data generation in order to enable parallel processing.
• Snowflake on Azure no longer requires CREATE SCHEMA permissions in order to support Preserve Destination mode for tables. Snowflake on AWS with Lambda processing continues to require CREATE SCHEMA permissions in order to support Preserve Destination mode.
• When Tonic is unable to create a view in the destination database, it now returns a warning instead of an error.
• Implemented a more accurate method to detect hexadecimal values.

SQL Server

• Tonic now displays the correct generators for columns that are part of a composite unique index.

The new Business Name generator produces realistic names of businesses or companies. The Business Name generator can be consistent with itself or with other columns. It improves on and is intended to replace the Company Name generator, which is now deprecated.

Fixed an issue on Table View where users could not use the delete icon to remove a generator assignment for a linked column.

Fixed an issue where the job details view for a subsetting job did not always show all of the steps as completed.Updated the version of pytorch, which is used for data science mode. This new version addresses some security vulnerabilities.

Fixed an issue where when the Tonic server was air gapped, the Admin Panel did not correctly display the current Tonic version.

Fixed an issue where jobs took longer than expected to complete.

Fixed an issue where the subsetting Graph View did not show how table participation in the subset changed since the most recent subsetting data generation.

Fixed an issue where after a single failure to write logs, the Download Job Logs feature stopped refreshing the logs. Tonic now continues to try to upload logs.

Google BigQuery

• Fixed an issue with the test connection function for the destination database.
• Materialized views and routines from the source database are now copied to the destination database.

MongoDB

• Improved performance for sensitivity scans.

MySQL

• Fixed an issue where subsetting failed because of a data type mismatch between a primary key and a foreign key.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB = true, foreign keys are now correctly enabled on the destination database.

PostgreSQL

• Fixed an issue where the source database permissions check provided a false error about insufficient privileges for sequences.
• For the beta Data Pipeline V2 data generation process, fixed an issue where the data generation process continued even after the job failed.

Snowflake

• Fixed an issue where the presence of comments caused data generation to fail.
• Improved performance for sensitivity scans.

SQL Server

• Added support for user-defined types.

Generator presets are now supported for Enterprise licenses on Tonic Cloud.

For Tonic data encryption, fixed an issue where the previous encryption key environment variable value was saved in the application database, which caused Tonic to use those values even after they were removed.

The Tonic diagnostic logs now include the Tonic worker ID.

Fixed an issue where the Tonic web server would not launch unless the Tonic application database used PostgreSQL v13 or later.

For data connectors other than MongoDB, the sensitivity scan is now parallelized.

Google BigQuery

• Data generation now works correctly when the region that hosts Google BigQuery for the destination database is different from the region for the source database.
• Fixed an issue where failed data generation jobs were incorrectly reported as successful.
• Tonic now handles the TIME data type correctly.

MySQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB=true, fixed an issue where the truncation of tables violated foreign key dependencies, which caused jobs to fail.
• Added an option to enable the TCPS protocol for Oracle database connections. Previously, only TCP was supported. If you enable TCPS, you must also provide a wallet file.
• Tonic now cleans up temporary destination database tables that were created during subsetting data generation.

PostgreSQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.
• During data generation, Tonic now warns users when an extension that the destination database needs is unavailable for installation.

Snowflake

• For both Snowflake on AWS and Snowflake on Azure, you can now configure workspaces to limit the schemas to include.

For Tonic data encryption, Tonic now only verifies the key for the enabled process. If you only enable decryption, then Tonic only verifies the value of TONIC_DATA_DECRYPTION_KEY. If you only enable encryption, then Tonic only verifies the value of TONIC_DATA_ENCRYPTION_KEY.

Upgraded our Docker images from Ubuntu 20 to Ubuntu 22.04.

Updated to ensure that the Tonic URL reflects the currently active workspace.

Recently started jobs no longer display a start time that occurred several years ago.

On the Data Encryption tab, the option to provide custom initialization vectors is now a toggle instead of radio buttons.

Resolved an issue where Tonic took an extremely long time to load.

Oracle

• Reduced the permissions required to test database connections.
• Changed the required permissions to better support when TONIC_ORACLE_DBLINK_ENABLED is false. For the source database user, you can either grant SELECT ANY DICTIONARY, grant SELECT_CATALOG_RULE, or (not recommended) grant access to the ALL_* views.

PostgreSQL

• Improved the error messaging when testing the connection to the destination database.

Snowflake

• You can now use a connection string to connect to the source and destination databases. Also added support for proxy connections.

NOTE: v805 and v806 are removed.

The new Snowflake on Azure data connector uses Azure Blob Storage to store interim uploaded and generated files.

A copied workspace now includes manual sensitivity designations. A manual sensitivity designation is when you change the sensitivity designation that was assigned by the sensitivity scan to either sensitive or not sensitive.

When the configured encryption or decryption key is not valid - for example, the key is not configured or uses the incorrect size - then Tonic does not allow you to configure Tonic data encryption.

Tonic Cloud now correctly enforces the supported data connectors.

Improved error messaging when subsetting data generation fails because the generator cannot be used with subsetting.When you change the type of Tonic data encryption (decryption, encryption, or both), Tonic no longer clears the decryption and encryption text fields.

MongoDB

• Fixed an issue where workspaces that contained views did not load.

Oracle

• Fixed an issue where the default Oracle NUMBER type was not compatible with the Integer Key Generator.
• Removed an invalid error that was returned when users tested data connections.

PostgreSQL

• For the beta Data Pipeline V2 generation process, improved the error logic to prevent jobs from hanging when errors occur.

SQL Server

• Improved the resilience of data generation to transient failures.

Fixed an issue where the column configuration panel on Privacy Hub incorrectly allowed users to change the configuration of foreign key columns.

The option to create a completely new workspace is no longer available from the workspace management view. You can only copy that workspace and create a child workspace. To create a completely new workspace, use the Create New Workspace option on the Workspaces view.

Fixed an issue where the generator for a column in a table that uses Scale mode could have a configuration option that is invalid for that mode.

Upgraded libraries to address security vulnerabilities.

Fixed an issue where when you applied the Random Timestamp generator to a column, a Bad format string error was returned.

Improved display of long table names in the table details panel on Subsetting view.

MongoDB

• Introduced a new configuration to collapse child fields into a single field based on a regular expression, to reduce the size of the schema. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX provides the regular expression to check the field keys against. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX_THRESHOLD specifies the number of matching fields that causes the fields to be collapsed. A value of 0 indicates to not collapse the fields.
• On Collection View, fixed an issue where toggling the data preview changed the frequency values for documents within documents.

Setting the default configuration for generators (Requires an Enterprise license)

The Generator Presets view allows you to configure the default configuration for generators. The current configuration is used whenever that generator is assigned to a column.

To update the configuration, you must be an owner or editor of a workspace in the instance.

In the configuration for a column, you can override the saved default configuration, which we call the baseline configuration. You can also revert to the current baseline configuration or save your configuration as the new baseline configuration.

Other updates

On the workspace management view, the workspace options are now included in the collapsed version of the heading.

For free trial users, the option to create a workspace is always visible in the Tonic heading.

For SAML SSO, if the value of NameID is not an email address, Tonic uses the email claim in the SAML response.

Databricks

• Added a TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE environment variable to determine whether new Databricks tables ErrorOnOverride by default. The default value is true, indicating that new tables ErrorOnOverride.

SQL Server

• Changed the default value of BYPASS_MS_XML_PARSING from false to true. The variable indicates whether to convert XML columns to nvarchar(max) to avoid potential XML parsing bugs.

When you log back in to Tonic, it now displays the workspace management view for the most recently viewed workspace.

Improved error messaging for rarely occurring user authentication issues.

Fixed display issues on the Subsetting view where part of the subset results graph did not display and the Configuration tab did not scroll.

Amazon Redshift

• Added the ability to provide filters on tables.

MongoDB

• Fixed an issue where indexes created with an expireAfterSections value are not created in the destination database.
• Fixed a regression where unscanned collections did not display in workspaces.
• Fixed an issue that prevented Tonic from being able to connect to MongoDB-compatible DocumentDB databases.
• Fixed an issue where DocumentDB jobs failed when run from Windows.

MySQL

• Updated to ensure that TONIC_WRITE_PARALLELISM is always 1, to prevent lock timeouts.
• Increased resilience around possible failures when retrieving a MySQL schema during data generation.

Snowflake

• Added the ability to provide filters on tables.

SQL Server

• Improved performance when reading large fields.
• Fixed an issue that caused data generation to fail with the error Sequence contains no matching element.
• Fixed an issue that caused data generation to fail on SQL Server 2014.

Added a confirmation step when updating the baseline configuration of a generator preset from a column configuration panel.

Fixed an issue where job failures sometimes caused the worker process to crash in a way that prevented new jobs from running.

To improve Tonic web application security, added X-Frame-Options/X-Content-Type-Option headers.

Databricks

• Corrected an issue on the workspace configuration page so that read-only items are no longer clickable.

PostgreSQL

• You can now use connection strings to connect to the source and destination databases.

SQL Server

• Fixed an issue where data generation failed when all URNs in the database have no dependencies.

NOTE: Releases 741 and 742 were removed.

Graph View for subsetting

The Configuration tab of Subsetting view now includes a toggle to switch between Table View and Graph View.

Table View is the existing tabular list of tables.

Graph View is a diagram view that displays the tables and the relationships between them. Similar to Table View, when you click a table in Graph View, the table details panel displays for that table. You can configure subsetting from either view.

On Graph View, Tonic adds a marker when the subset configuration for a table changed since the last subsetting data generation. A table might be added to the subset, removed from the subset, or modified within the subset.

Tonic data encryption

The new Tonic data encryption feature, available for Professional and Enterprise licenses, allows you to set up a configuration to decrypt source data before applying a generator, encrypt transformed data before writing it to the destination database, or both.

Tonic data encryption uses AES encryption.

Tonic data encryption requires you to set environment variables for the decryption key (TONIC_DATA_DECRYPTION_KEY) and encryption key (TONIC_DATA_ENCRYPTION_KEY). Both keys must use the same key size - either 128, 192, or 256 bits.

Admin users configure Tonic data encryption from the Data Encryption tab of the Admin Panel.

When you enable Tonic data encryption, the generator configuration includes a setting to indicate whether to use it for that column.

Other updates

Fixed an issue where navigating to Database View from Schema Changes view for a table-specific issue did not apply the correct filters.

Fixed an issue where workspace import could inadvertently add multiple subset targets in the same table.

Added support for ssh-rsa for data connections.

Improved error logging for the Timestamp Shift generator.

Fixed the display for the import workspace dialog.

MongoDB

• The new ObjectId Key generator allows you to de-identify object ID columns.
• Moved the scanning of MongoDB collections into the Tonic worker.

PostgreSQL

• Fixed an issue that caused data generation to fail with the error canceling statement due to statement timeout.

SQL Server

• Fixed an issue where a bug in the SQL Server Management Objects (SMO) library caused data generation to fail when using partitioning in Azure.
• Fixed an issue that caused data generation to fail when system tables were included.

NOTE: These releases were removed.

Enhancements

In a Databricks workspace, you can now choose to write all of the output tables to one of the following formats: Avro, JSON, Parquet, Delta, CSV, ORC. This setting replaces the previous option to write all of the output tables to Databricks Delta.

When sharing a workspace, free trial users can now invite other users with the same corporate email domain to start their own free trial.

Other updates

Added messaging to the Tonic application about changes to the Tonic license plans.

Fixed an issue with the JSON Mask generator where when users deleted a sub-generator, a different sub-generator was deleted.

Fixed cases where data generation jobs remained in the queued state indefinitely.

Fixed a performance regression that affected workspace loading.

Oracle

• Fixed an issue where a data generation fails when there are index-organized tables.

As of V724, the AI Synthesizer generator is by default not available. To enable the AI Synthesizer generator, set the environment variable TONIC_NN_GENERATOR_ENABLED to true. If the AI Synthesizer generator is assigned in a table, but the environment variable is false, then data generation fails.

During subset processing, when Tonic encounters a circular dependency, to break the circle, it nullifies the values of one of the foreign key columns. By default, it nullifies all of the foreign key column values. A new environment variable, TONIC_SUBSETTING_AGGRESSIVELY_NULL_CYCLICAL_FKS, controls the behavior. By default, the value is true. If you set the value to false, then Tonic only nullifies foreign key values that do not exist as primary key values in the other table.

Fixed an issue where authentication using Azure SSO failed if the user was a member of a large number of IdP groups.

Improved detection during sensitivity scans of columns that contain birth dates.

MongoDB

• Fixed an issue where adding a new null type field to the data caused data generation to fail.

Oracle

• Fixed an issue where data generation returned the following error: Specified argument was out of the range of valid values. (Parameter 'minSize').

PostgreSQL

• Fixed an issue where tables with generated columns, but no generator assignments, were not correctly copied to the destination database.
• Improved handling and messaging for connection timeouts.

Snowflake

• Fixed an issue where retrying a connection multiple times returned an incorrect error.

SQL Server

• Fixed issues where users could not configure data generation for tables that contained hierarchyid columns.
• Fixed an issue that prevented Preserve Destination from being used on schema-bound tables.
• Fixed an issue that prevented Incremental and Preserve Destination table modes on memory-optimized tables.
• Fixed an issue where column attributes such as the nullability of a column were not correctly reflected in the destination database.
• Fixed an issue where schema binding was not maintained in the destination database.

Redesigned Tonic navigation

As of V719, we have redesigned the Tonic navigation. The left navigation menu is removed. On Workspaces view, click the workspace name to display the workspace management view for that workspace. The workspace management view contains a horizontal navigation bar to provide access to the workspace configuration and generation tools, and a heading menu to provide access to other workspace actions.

On Workspaces view, the first column is now a checkbox to select workspaces to which to apply an option from the Actions menu.

The Tonic application header contains links to Workspaces view and the Admin Panel.

New table filtering for Google BigQuery

As of V717, Google BigQuery supports table filtering. On the table mode selection panel, for tables that use De-Identify mode, you can provide a WHERE clause to filter the records that are included in the destination data.

Other updates

Fixed an issue where Tonic workers did not start after an upgrade when a read-only file system was mounted for custom value processors.

Improved the detection of vehicle identification numbers (VINs) in source data.

Oracle

• Improved generator recommendations for columns that are part of a compound unique constraint.

SQL Server

• Fixed an issue that caused jobs to fail when none of the tables were assigned De-Identify mode.

NOTE: Releases v692 through v695 were removed.

Update to PII detection - PII container removed

As of v697, updated how Tonic performs PII detection. The name detection process now also scans for international names with origins from 103 different countries.

The PII detection process now runs from the Tonic worker instead of in a separate container.

To avoid errors when you upgrade, future versions up to v999 will have a placeholder PII container. The container is not active and is not used to run Tonic.

Before you upgrade to V1000 or later, you must remove the PII container from your instance.For a Docker deployment, in docker-compose.yaml, remove the tonic_pii_detection section.

For a Kubernetes deployment:

• Remove the file tonic-pii-detection-deployment.yaml.
• Remove the file tonic-pii-detection-service.yaml.
• Update tonic-worker-deployment.yaml to remove the entry for TONIC_PII_DETECTION_URL.
• Update values.sample.yaml to remove the entry for pii_detection.

If you do not remove the PII container before you upgrade to v1000 or later, then when you upgrade Tonic, you will encounter image pull errors.

Other updates

Improved error messaging on the Tonic UI for failed data generation jobs.

Reduced the telemetry logging for PII detection during sensitivity scans to reduce the performance impact on self-hosted instances with machines that do not have internet access.

Fixed an issue that prevented users from configuring the Random Timestamp generator Date Format when the generator was selected as a sub-generator of the JSON Mask and other composite generators.

Made a minor correction to a data generation log message.

Reduced Tonic slowdowns caused by telemetry.

MySQL

• Fixed an issue that caused data connections to fail with the error The ConnectionString property has not been initialized.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

Oracle

• Fixed an issue where long column names caused data generation to return the error ORA-00972: identifier is too long.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

PostgreSQL

• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

SQL Server

• Users can now assign generators to timestamp and rowversion columns.
• Fixed an issue where a database ID mismatch in the master and database tables prevented data generation from starting.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

Fixed an issue where when subsetting used parallel processing, the subsetting steps displayed incorrectly.

Improved the error message that is displayed when unsupported data types are present in a table.

On the Subsetting view, on the row count popup, provided a clearer explanation when the destination row count is larger than the source row count.

Fixed an issue where simultaneously updating workspace permissions for multiple users failed to apply the updates.
Made some small performance improvements to the Regex Mask and Array Regex Mask generators.

Improved the job cancellation logic to ensure that selecting the cancel option actually cancels the job.

When a SAML SSO login is initiated, Tonic now redirects the browser to the correct URL.

When Tonic detects an invalid or deprecated generator, it no longer returns the error message Unexpected generator id {generatorId}, create a dedicated Metadata class for this generator.

MySQL

• Fixed an issue that caused lockouts and timeouts on the destination database.

Oracle

• Reduced the number of destination database permissions that are required for data generation.

PostgreSQL

• When you use the Limit Schemas feature, and you choose to filter to only include specific schemas, Tonic no longer includes all of the extensions in the database. It only includes extensions that are related to the included schemas.
• Values are now truncated to fit the char type before they are written to the destination database.
• Generators that support consistency can now be made consistent with columns that are user-defined enum values.
• The Conditional generator can now use columns that are user-defined enums as conditions to apply sub-generators.
• Tonic now correctly determines whether to truncate a unicode string when fitting the string into a data type that is close in size.
• Improved resiliency to destination database connection failures.

NOTE: Releases v684 and v685 were removed.

Enhancements

When creating new virtual foreign keys, you can now use the top level field name check box to select or deselect all of the fields that have that name.

Other updates

Fixed an issue where a warning did not display correctly when a subset table was configured with an invalid table mode for subsetting.

For the Custom Categorical generator, we no longer treat newlines as empty strings on numeric columns. Newlines are still treated as empty strings on string columns.

Fixed an issue where an error occurred when a column is assigned the Custom Value Processor generator.

Fixed an issue where data generation with subsetting failed with the following error: Could not load type from assembly 'Allos.Generators'.

Redesigned the generator selection dropdown to better separate the suggested generators from the other applicable generators.

Oracle

• Improved error logging during data generation.

SQL Server

• Fixed an issue where views that reference preserved tables (table mode is Preserve Destination) were not created.
• Added verification to ensure that all of the required tables and objects are created in the destination database before data generation.

Enhancements


Tonic now displays warnings at 30, 15, and 7 days before a Tonic license expires.

Other updates

After a one-click update of Tonic, containers for Docker Compose customers now include the version number in the name.

Addressed an issue for Docker Compose customers where Tonic did not restart properly after a one-click update.
Tonic can now download SAML IDP metadata from a URL. To configure the URL, set the environment variable TONIC_SSO_SAML_IDP_METADATA_XML_URL.

Added helper text to indicate the value format for a database server.

Added the ability to configure the SAML request issuer. To configure the issuer, set the environment variable TONIC_SSO_SAML_ENTITY_ID.

Improved error messaging for WHERE clause validation in subsetting configuration.

Moved some of the temporary files used for data generation from /tmp to /tmp/tonic.

MySQL

• Fixed an error where data generation occasionally failed with the error "Allos.Core.Exceptions.TonicException: No databases selected to overwrite".

Oracle

• Updated to ensure that system tables are ignored.

PostgreSQL

• Fixed an issue with data generation for partition tables.

EnhancementsTonic now supports SAML SSO providers.

Other updates

Fixed an issue where containers do not start and the following error message is returned: "The directory named as part of the path /var/run/supervisor/supervisord.pid does not exist"

Corrected an issue where in some cases when the job failed because of a source connection issue, the job status was not set to Failed.

In the subsetting configuration, you can now configure a target table percentage value with up to 3 decimal places.

Resolved a potential database connection leak when canceling data generation jobs.

Improved error messaging when a deleted SSO user tries to log into Tonic.

Improved error display for subsetting WHERE clause validation.

Fixed an issue where running a query to validate a subsetting WHERE clause could cause the application to slow down.

Fixed an issue where jobs could fail when subsetting parallelism was enabled.

The Conditional generator now allows you to configure a selected sub-generator to be consistent with other columns. The selected generator must support consistency with other columns.

Databricks

• Updated the Databricks job names to use the format tonic_workspaceName/workspaceId + jobID.
• Improved performance for page loads.

Google BigQuery

• Added support for the JSON data type.

MongoDB

• Fixed a minor flickering issue on the Collection View.

MySQL

• Improved logging of errors that produce data generation warnings.

Oracle

• Corrected an issue where we displayed foreign keys that referenced a schema outside of the workspace.

PostgreSQL

• Fixed an issue with handling HStore types during data generation.
• Improved logging of errors that produce data generation warnings.
• Improved management of partition tables.

Spark

• Improved logging for data generation.

SQL Server

• Fixed authentication handling for Active Directory. Tonic supports NTLM v2 authentication.

Enhancements

New command-line tool for Tonic installation - Tonic now offers the Tonic Installation Manager (TIM), a command-line tool to deploy Tonic on either Amazon EKS or a VM.

Other updates

Updated how Tonic performs PII detection. Added additional name values to the information that the detection process looks for.

Fixed an issue where navigating to the workspace edit page sometimes threw an error.

Fixed an issue where SSO account creation erroneously returned errors even though the account was created successfully.

Improved our generator recommendations that are based on the security scan to prevent memory issues for larger databases.

Improved error messaging when Tonic-hosted users attempt to connect to a database on a local network.

Fixed an issue where job log timestamps displayed the wrong month value.

Amazon Redshift

• Corrected a permissions check issue that caused connection tests to fail.MySQL
  
  
  • Upgraded MySqlConnector to version 2.2.5.
  • Fixed an issue where subsetting jobs failed with a "No database selected" error.
  Oracle
  
  
  • Fixed an issue where data generation jobs did not stop when they were canceled.
  • Fixed an issue with applying the Event Timestamps generator to timestamps that have timezone values.
  • Update to prevent Tonic from running out of space during subsetting.
  • Fixed an issue where materialized views and materialized view logs are not torn down after data generation.
  • When TONIC_ORACLE_SKIP_CREATE_DB is true, we now properly truncate tables.
  PostgreSQL
  
  
  • Upgraded the Npgsql library to version 7.0. This is used for both workspace data and the Tonic application database.
  Snowflake
  
  
  • Corrected a permissions check issue that caused connection tests to fail.

Enhancements

From the Subsetting view, you can now view the table configuration and results summary for previous subsetting data generation runs. On the Previous Runs tab, you select the run to view the details for. Note that you can only view the details for jobs that you run after you upgrade.

Other updates

Corrected an issue for instances using a Docker version before 20.10.10 that prevented Tonic processes from starting after an upgrade from version 642 or below to version 643 through 647.

Corrected an issue where estimated data generation time was logged as a negative number.

Improved the performance and resiliency of the subset WHERE clause validation.

Improved the cleanup of database connections after they are used.

Improved error message when the source database version is newer than the destination database version.

Google BigQuery

• Fixed a user interface issue that prevented a new workspace from being created.
• Tonic now supports post-job scripts in BigQuery workspaces. Post-job scripts run inside of transactions. They are limited to statements that are supported within transactions.

Oracle

• The minimum supported version for the Oracle data connector is now Oracle 12c. Tonic no longer supports 11g. As a result of this change, the Oracle Helper is no longer needed. After upgrading, you should remove Oracle Helper from the list of containers to fetch.

Snowflake

• For database connections, the server name should not include https://. Tonic now detects and removes https:// if it is entered.

Spark

• Eliminated a vulnerability that could allow users to see the results of statistics jobs that they did not have access to.

SQL Server

• Fixed an issue with loading tables in SQL Server 2012 and below

Enhancements

New endpoints and expanded API documentation for generators - For generators, the generated Swagger documentation now contains descriptions for each endpoint and for all of the model properties. You can send a request to retrieve a list of available generators. You can now update the generator configuration for a single column, instead of having to provide the configuration for an entire table. There is also a new endpoint to revert a column to Passthrough.

On the Subsetting page, you can now sort the tables based on whether they are in or out of the subset.

Other updates

Improved UI support for timeouts when loading schema information for a source database.

Updated the subsetting process to ensure that a small percentage-based target table contributes at least one row to the subset.

The environment variable TONIC_SUBSET_PARALLELISM is deprecated. Tonic now uses the environment variable TONIC_TABLE_PARALLELISM to control parallel processing for subsetting.

When a data preview or workspace loading process is no longer needed, such as when there are network issues or the user leaves the application, Tonic now attempts to cancel the process.

MySQL

• Improved error messaging when a database connection times out.

Oracle

• Fix to ensure that data generation fails when drop table statements fail.

PostgreSQL

• Improved error messaging when a database connection times out.

SQL Server

• Fixed an issue where some tables were not created in the destination database.
• Improved error messaging when a database connection times out.

Enhancements

Overriding table mode and generator configuration in child workspaces - The workspace inheritance feature now allows child workspaces to override the table mode and generator configuration from the parent workspace. Database View and Table View indicate when the parent configuration is overridden, and provide options to reset the configuration.

Other updates

You can now assign the SIN generator to fields that have uniqueness constraints.

The Address generator now only allows you to link columns that contain the following types of address values - City, City State, Country, Country Code, State, State Abbreviation, Zip Code, Latitude, Longitude.

Improved memory handling for subset processing of downstream tables.

On the Subsetting view, when the database returns an unrealistic value for the subset size, we replace the value with a warning message.

When using Helm to deploy Tonic via Kubernetes, integer environment variable values that are longer than 6 digits might be converted to scientific notation. To avoid this issue, Helm recommends that you enter values in quotes. Tonic can now parse scientific notation to better handle this behavior for values that are not in quotes.

Fixed an issue where linking columns that have the same name as columns in another table caused the generators to be removed from those other columns.

Made minor memory improvements throughout the Tonic application.

Corrected an issue where Tonic displayed a "subsetting in progress" message before the processing started.

Amazon Redshift

• Fixed an issue where data generation failed when the name of one table was part of the name of another table (for example, Test and Test1).

MongoDB

• Generators are now applied correctly to documents that are nested in arrays.

Oracle

• Removed the permissions check for reading truncated tables.
• Fixed an issue where DBLinks created for data generation were not removed when the generation failed or was canceled.

Snowflake

• The Categorical generator no longer returns errors when applied to binary, time, and date columns.
• Improved the performance for loading the user interface when the data contains a large number of foreign keys.

Spark

• The Audit Trail entries on Privacy Hub now display the correct table and column names.
• Corrected an issue where sub-generators for the Conditional generator were not applied correctly.
• Improved error messages when testing the connection to a Databricks database.

Improved user signup experience for the hosted version of Tonic.

Made some small styling updates to Database View. Removed the dropdown to uncheck or check all of the tables. Removed the highlighting on the table mode dropdown for tables that contain columns with assigned generators. Changed the Filter button labels to Filters.

For subsetting, verify that primary key fields are not assigned a non-primary key generator.

Amazon Redshift

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

MongoDB

• Fixed an issue that caused errors when applying the Timestamp Shift generator.

Snowflake

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

Spark

• Improved error messaging for unsupported versions.

SQL Server

• Improved handling of alphanumeric sequences.
• Case-sensitive databases now connect correctly to Tonic.
• Corrected how indexing recommendations for subsetting are determined.

Enhancements

Added filters for upstream subset records - In the subsetting configuration, for upstream related tables, you can now filter the records to include based on either a date value or a WHERE clause. Upstream tables contain data that has a foreign key that references a primary key in a target table. Upstream records are optional, and are not needed for referential integrity.

The Confirm Generation panel now provides access to tips to improve data generation performance.

Other updates

Tonic no longer offers DB2 as a data connector type.

On the Job History list, changed the job type "Privacy Scan" to "Sensitivity Scan".

Corrected a display issue in Privacy Hub where the column details panel extended past the bottom of the page.

For the JSON Mask generator, the path expression selection tool now works for arrays and for keys that contain spaces or special characters.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Added a message to notify users when queries used to validate a subset WHERE clause run for a long time.

Tonic now continues to record logs when a job fails.

Improved memory usage when running data generation.

Amazon Redshift

• Fixed an issue where Tonic failed to display an error for an invalid generator assignment.

MongoDB

• Corrected a subsetting issue that occurred when multiple foreign keys in one collection referred to the same primary key in another collection.

MySQL

• Updated the MySqlConnector driver to 2.2.0.

Oracle

• Downgraded the Oracle drivers to 3.21.70 after encountering issues with 3.21.80.
• Corrected an issue with the Event Timestamps generator.

PostgreSQL

• When executing psql and pgdump commands, we properly enforce the requirement to use SSL for connections.
• Corrected connection issues by escaping special characters in password files.

Snowflake

Spark

• You can now apply the Timestamp Shift generator to string fields that contain datetime values.

Enhancements

Tonic can now integrate with Keycloak for SSO authentication.

Other updates

Fixed an issue where users received the error HTTP/1.1 415 Unsupported Media Type from the /api/GenerateData/start endpoint.

Improved performance for the /api/job endpoint. As part of this update, the endpoint only returns the most recent 100 jobs.
Improved loading time for the Job History view, which now displays only the 100 most recent jobs.

Updated how Tonic determines whether to use SSL for connections.

The Add Foreign Key Relationships tab on the Foreign Keys view now expands to fill the height of the browser.

Fixed an issue where a failed cleanup step incorrectly marked a Tonic update as failed.

Amazon Redshift

• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.

MongoDB

• When performing a schema scan, Tonic now displays a progress indicator.
• Improved subsetting performance.
• Fixed an issue where subsetting failed because of duplicate keys.
• Corrected how Tonic handles views.

Snowflake

• Fixed an issue where the Tonic user interface could not load when the database included foreign keys.
• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.