Tonic Structural release information

You can now export individual topics from the Tonic documentation to PDF files. To export a topic to PDF, click the actions menu next to the topic title, then click Export as PDF.

Fixed an issue with removing unique constraint conflicts in upsert where rows that didn’t have a conflict were excluded from the upsert process.

File connector

• For Amazon S3 and Google Cloud Storage, the permission to list all buckets is no longer required. However, if that permission is not present, users must manually type in the bucket name where the file is located.
• When you copy a file connector workspace, Tonic now copies the file groups to the new workspace.

Upsert data generation (beta feature)

Previously, the data generation process always replaced the entire destination database. The new upsert data generation option (currently in beta) allows you to add new records and update existing records without touching any of the other records in the destination database. For example, you might have a regular set of records that you use for testing that you want to maintain.

Upsert requires a connection to an intermediate database. When you run data generation with upsert, the initial data generation writes the transformed data to the intermediate database. It replaces the intermediate database, similar to regular data generation. After the generation to the intermediate database, the upsert process identifies the records to add to or update in the destination database. It ignores other records in the destination database.

Upsert requires a Professional or Enterprise license, and is only supported for the following data connectors:

• MySQL
• Oracle
• PostgreSQL
• SQL Server

New AI-enhanced documentation search option

The Tonic documentation now provides access to Lens, an AI-based search option. Instead of searching for specific words, you can ask questions such as "How do I create a workspace?". Lens searches the documentation for the answer. It generates a response that includes links to the topics that contain the information it used.

To use the Lens search, click the search field. At the top right of the search panel, click Lens. Then ask your question.

Other updates

The Custom Categorical generator now supports consistency with other columns. Previously, the generator only supported self-consistency.

Tonic now prevents you from starting data generation for a workspace that does not have a destination database specified.

Fixed a display issue where the generator preset details panel briefly showed the occurrences for the previously selected generator preset.

Tonic now suggests the Name generator for columns that Tonic detects as containing names, when the detection is based on the sampled data in the column. By default, the Name generator uses the First Last format.

A new configuration option allows webhooks to bypass SSL certificate validation and trust the server certificate.

File connector

• You cannot start data generation for a file connector workspace that has no source files specified.
• The Table View data preview for file groups that contain JSON or XML files no longer displays above the real data records an extra row that contains the value 0.
• Fixed an issue where generators such as the Categorical generator unexpectedly could not be used as sub-generators.
• Improved error handling when a file group is incorrectly configured as having a header row.
• Fixed an issue that caused data generation to fail for XML files because of missing metadata.
• Fixed an issue where Database View displayed duplicate columns.
• Table View no longer displays an extra data row.

MongoDB

• Fixed an issue that caused unscanned collections to not display on Collection View.
• The sensitivity scan no longer marks Null fields as sensitive.

PostgreSQL

• Improved performance when refreshing materialized views in PostgreSQL v11, v12 and v13.

On the generator configuration panel, changed the label of the Save As menu to Preset Options. The menu contains options related to configuring generator presets.

Free trial users now have access to the file connector.

Tonic now displays the error that occurs when an Algebraic generator configuration does not include any floating-point values.

For composite generators, the generator preset details panel now provides a clearer explanation that presets for composite generators must be configured from within a workspace.

Improved performance for the Address generator and the HIPAA Address generator.

Amazon Redshift

• Fixed an issue with clearing temporary tables.

File connector

• Fixed how Tonic handles EOF characters.

MongoDB

• Corrected the order of the available generator presets for a field.

Snowflake

• Fixed an issue where data generation returned the error The specified bucket does not exist.

SQL Server

• Database connections can now use the MultiSubnetFailover option.
• Improved error messaging when a database cannot be created because of permissions issues.

Removed support for TIM - The Tonic Installation Manager (TIM) command-line tool to install and configure Tonic is no longer available.

Free trial users can now use a public email address to create the free trial account. Users with public email addresses cannot invite other users or share workspaces. Public accounts are only allowed for free trials.

Users on a Professional instance can now share the Manager workspace permission set with users and groups.

Improved error handling and validation messages for the foreign key file upload process.

Counts of generator preset occurrences no longer include occurrences in deleted workspaces.

On the bulk update panel in Database View, the consistency and differential privacy options now display correctly.

Fixed an issue where you could not select Passthrough as a sub-generator in a composite generator.

Fixed an issue where custom presets could not be deleted.

Fixed a display issue where long post-job action names overflowed into the next column.

Fixed an issue where you could not assign Random Timestamp as a sub-generator for the Conditional generator.

Fixed an issue where the generator configuration panel displayed the generator preset options when the user did not have the Manage generator presets global permission.

Fixed an issue where when a constraint failed to be applied, data generation failed.

Improved display when users who do not have the Manage generator presets permission try to display the Occurrences tab on the preset details panel.

Improved how we handle unavailable options for workspace actions in Workspaces view and in the Tonic navigation options.

For the Conditional generator, Tonic now correctly compares MySQL date values.

Databricks

• Tonic cluster initialization scripts are now uploaded as workspace files instead of DBFS files. The new, optional Workspace Path setting for Databricks workspaces controls the parent directory where Tonic uploads initialization scripts. The default value is /Shared.

File connector

• The file connector can now support .txt files that contain CSV, XML, or JSON content.
• Fixed an issue where Tonic incorrectly identified how a file connector file was encoded.
• Improved error messages when uploading files for the file connector.
• Fixed an issue when configuring a file group from Amazon S3 where users saw the error "Failed to fetch files from S3. The continuation token provided is incorrect." but could still see the list of files in the S3 bucket.
• Tonic now correctly updates the file configuration for file groups. Previously, users could not add files that did not match the default configuration.
• Tonic now displays an error when it is unable to read files from Amazon S3.
• The file explorer for Amazon S3 can now list the files in folders that have names that contain special characters.
• Improved encoding detection and file parsing.
• Tonic now correctly handles EOF characters in .csv files.
• Tonic now preserves the encoding of .csv files.

MongoDB

• Fixed an issue where the protection status information at the top of Privacy Hub did not update correctly after a new sensitivity scan.

Custom generator presets

Earlier this year, for Enterprise instances, we introduced the concept of generator presets. A generator preset is a saved configuration of a generator. You can assign generator presets to columns.

The initial release only included built-in generator presets, which allowed you to set the default configuration for Tonic generators.

This update in v924 introduces custom generator presets, which allow you to set up multiple configurations of the same generator. You can create custom generator presets from Generator Presets view. From a generator configuration panel, you can also save the current configuration as a new custom generator preset.

Generator preset occurrences

From Generator Presets view, you can see how often each preset was used in a workspace configuration.

The Occurrences column of the generator presets list shows:

• The number of times the baseline configuration was used
• The number of times the baseline configuration was overridden, meaning that a user selected the generator preset and then made a change to the generator configuration

On the generator preset details panel, the Occurrences tab displays both the number of occurrences and the specific workspaces and columns where the generator preset was used. You cannot see workspace and column details for workspaces that you do not have access to.

Other updates

Tonic can now integrate with GitHub for SSO authentication.

To manage generator presets, users must now have the Manage generator presets global permission. Previously, you could also manage generator presets if you had the Manager or Editor workspace permission set for any workspace.

Fixed an issue where the table data in Table View was not updated correctly when switching the table mode to or from Scale mode.

Improved performance for the Regex Mask and Conditional generators.

MongoDB

• Fixed an issue where the subsetting Graph View did not display virtual foreign key relationships.
• You can now add collections to a subsetting rule before the sensitivity scan completes.

PostgreSQL

• Fixed an issue where certain database constraints were not handled correctly, which resulted in job warnings about the failure to add those constraints.

Permissions and permission sets

As of v922, Tonic now uses permissions and permission sets to manage access to Tonic features and functions.

A permission controls access to a single feature or function. A permission set is a saved collection of permissions.Tonic provides built-in global and workspace permission sets. You cannot change the configuration of built-in permission sets. Enterprise instances can create custom permission sets.

Global permission sets contain global permissions, which control access to actions outside the context of a specific workspace. The built-in Admin global permission set grants access to all global permissions. Users and groups configured in the TONIC_ADMINISTRATORS environment variable are granted the Admin (Environment) global permission set, which also grants access to all global permissions. These global permission sets replace the previous admin user concept.

The built-in General User global permission set is assigned to all Tonic users, and grants access to create workspaces. You can also designate a different global permission set to assign to all Tonic users.

Workspace permission sets are assigned in the context of a specific workspace. They provide access to workspace permissions, which are associated with workspace management functions. The built-in workspace permission sets (Manager, Editor, Viewer Auditor) mirror the previous workspace roles. Similar to the previous Owner role, the Manager workspace permission set is granted access to all workspace permissions. However, unlike the Owner role, the Manager workspace permission set can be assigned to any user or group. You use the workspace sharing function to assign workspace permission sets within a workspace.

Each workspace has a single owner. The user who creates the workspace is the initial owner. All owners are by default granted the Manager workspace permission set. You can also designate a different workspace permission set to assign to workspace owners. You use the transfer ownership function to select a different owner for a workspace.

On Tonic Settings view, the User Management tab is replaced by the Access Management tab. From the Access Management tab, you can:

• View and manage the list of Tonic users
• If you use SSO, view a list of SSO groups
• View, configure, and manage access to global permission sets
• View and configure workspace permission sets

API endpoint to track user access and permissions

A new API endpoint to track the following events related to user access and permissions:

• A user account is created.
• A user account is removed.
• A user logs in to Tonic.
• A permission is added to or removed from a permission set.
• A permission set is assigned to or removed from a user. This might be a global permission set, or a workspace permission set in the context of a specific workspace.
• A generator preset is updated.

The endpoint is:

GET /api/audit-events/search

Other updates

You can now assign the Business Name generator as a sub-generator for the Regex Mask generator.

For subsetting, Graph View and the table details panel now display information about cycle breaks, when the subsetting process needs to break a circular dependency.

Databricks

• Fixed an issue that prevented the use of partition filters in Databricks Unity Catalog workspaces.

File connector

• When a file connector workspace is deleted, Tonic now deletes files that were uploaded from a local file system.

Spark

• Fixed an issue with data generation for workspaces that use Hive.

The Admin Panel is renamed to Tonic Settings.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Fixed an issue where Tonic opened too many connections to the application database.

Fixed an issue with timestamps in the Tonic API specification.

Fixed a Tonic Cloud issue where using a different email domain to update a Tonic license caused issues with Tonic logging.

Enhanced the performance of the HIPAA Address generator.

The Data Pipeline V2 data generation process now respects the TONIC_PROCESS_PARALLELISM environment variable.

Improved performance for subsetting, especially for data that contains a large number of foreign key relationships.Made a small performance improvement to primary key generators.

File connector

• Improved error messages when uploading files for the file connector.
• The file connector now supports extended ASCII-encoded files.
• Fixed an issue where the file preview omitted the first row of the file.

MongoDB

• On the Jobs view, you can now filter for the Collection Schema Scan job type.
• Fixed an issue where sensitivity scans failed on large collections.

Oracle

• Tonic now provides a more meaningful error when it loads Table View for a table that the database account does not have access to query data from.

PostgreSQL

• Corrected the job history entries for subsetting jobs that run using the Data Pipeline V2 process.

Spark

• Fixed an issue that caused jobs to fail when an invalid Repartition or Coalesce value was specified.

NOTE: Releases v899 through v901 were removed.

Tonic Data Pipeline V2 for PostgreSQL ends beta

During the first half of 2023, Tonic has run a beta program for PostgreSQL for our new Data Pipeline V2. The beta program is now ending. Thank you to all of those who provided feedback.

Starting with version V905, Tonic.ai will progressively enable Data Pipeline V2 for all customers. To ensure a smooth transition for all our PostgreSQL customers, Tonic.ai controls the rollout remotely.

The remote rollout mechanism is controlled by an HTTPS request from your instance to https://feature-flag.tonic.ai. A JSON payload with a unique identifier for your deployment is sent, and the status of Data Pipeline V2 is returned. This request happens at the start of each data generation. If your Tonic server cannot reach https://feature-flag.tonic.ai, then the check is skipped.

What to expect for the enrollment:

• Before your instance is enrolled in Data Pipeline V2, Tonic Customer Success will contact you to confirm your enrollment.
• After your instance is enrolled, the Data Pipeline V2 toggle on the Confirm Generation panel is removed. All PostgreSQL jobs run using Data Pipeline V2.

For jobs that run on V2:

• The job type is Data Pipeline Generation instead of Data Generation.
• Jobs should run faster. Data Pipeline V2 has better resource handling, and can provide more parallel execution, especially for large tables and to apply constraints. Not all jobs will be faster, but for some jobs there should be a significant improvement.

We will continue to improve Data Pipeline V2 as we expand coverage to other data connectors.

Expanded Graph View for subsetting

The subsetting Graph View is expanded to use more of the available screen space. The Configure Subset panel, which includes the Options and Latest Results tabs, no longer displays on Graph View. It only displays on Table View.

Other updates

Fixed an issue where when a data generation job failed, tables that used Preserve Destination table mode were not restored.

The generated Tonic API documentation now includes the endpoints for managing file groups for file connector workspaces.

Fixed an issue that caused jobs for some workspaces to fail with the exception "Cannot modify workspace whose schema is not the latest version.".

Fixed an issue where the job details view displayed incorrect information.

Updated the /api/DataSource endpoint to not contain secure data such as the API key.

Updated our OpenAPI documents to ensure that all values of operationId are unique.

Improved error messages for failed data generation.

Databricks

• The Test Connection button on the workspace details view now works correctly.

MongoDB

• Fixed an issue where schema scans time out even when a timeout is not configured.
• Fixed an issue where MongoDB workspaces did not support MongoDB versions below v4.4.

MySQL

• Tonic data generation now supports generated columns in de-identified tables.

Oracle

• To improve the resiliency of Oracle commands during data generation, fixed the retry logic.

PostgreSQL

• Tonic no longer waits to process a job cancellation until after it finishes the constraint application that was in progress when the job was canceled.
• Fixed an issue where Tonic returned an "insufficient space" exception when writing numeric values to a destination database.

Snowflake

• Improved performance for Snowflake on Azure.

Spark

• Job tracking URLs now display correctly on the job details page.

SQL Server

• Fixed an issue where Tonic did not retrieve all of the compound keys from the source data.

NOTE: Releases v897 and v898 were removed.

On Subsetting view, Graph View now displays a loading animation as new data is loaded.

Improved performance for the UUID Key and Integer Key generators.

File connector

• Fixed an issue where files sometimes did not upload completely.

Google BigQuery

• Improved performance of destination database writes during data generation.

MySQL

• Improved performance for destination database writes during data generation.

Oracle

• Improved error handling when the tablespace in the source database is missing from the destination database.

File connector

The new file connector data connector allows you to use files from either Amazon S3, Google Cloud Storage, or a local file system as the source data. The file connector supports .csv, .json, and .xml files. Within a file connector workspace, you create file groups. Each file group contains files that have an identical format and structure. A file group is treated as a table for the purposes of table mode and generator configuration. The file connector is available with the Professional and Enterprise license plans.

Generic O​​IDC SSO Support

Tonic now supports authentication using a generic OpenID connection.

Tonic API versioning

We have introduced a versioning scheme for the Tonic API. API versions are released more or less quarterly, with the version identifier in the format vYYYY.MM.P (Year.Month.Patch). The current release candidate (v.RC) contains API updates in progress.

You should now specify an API version in your API requests. The System Status tab of the Admin Panel lists the latest available version. You can also select the version to use when you do not provide a version in the request. If you do not provide an API version in a request or select a default API version, then until January 31, 2024, Tonic automatically uses the latest version. After January 31, 2024, Tonic will return an error from the request.

Other updates

Fixed an issue where Tonic incorrectly returned the error No Destination DB has been configured for this Workspace for workspaces that used Preserve Destination.

For subsetting Graph View, updated the default zoom level to allow users to see more of the graph.

The Keycloak SSO provider now supports PKCE challenge.

Fixed an issue where deep links did not work for SAML SSO.

MongoDB

• For Mongo queries, Tonic now can now use disk as well as memory.

Oracle

• Tonic now refreshes materialized views even when SKIP_CREATE_DB is set to true.

PostgreSQL

• In the job progress steps, fixed an issue that caused the number of rows in a table to display as -1.
• When the Data Pipeline V2 processing is enabled, tables are now processed by size, with larger tables processed first.

SQL Server

• Fixed support for Kerberos authentication.

Spark

• Added support for the UUID Key generator to Livy and Databricks workspaces that use Spark 2.3.x and above.
• Added support for the UUID Key generator to the Tonic Java SDK.

The ASCII Key generator now includes an Exclude Lowercase Alphabet option to exclude lowercase letters from the destination values.

Fixed an issue that prevented free trial signups.

Data generation no longer fails when Tonic is unable to retrieve the destination database size.

Updated the FNR generator to prevent a possible leakage of PII.

Added a Date column to the usage report. The date column provides the date and time when the data generation job was completed.

Fixed an issue in the JSON Mask generator where it incorrectly changed the format of timestamps.

Subsetting is no longer prevented when a table that is not in the subset is assigned Preserve Destination mode.

Databricks

• On the workspace details view, you can now optionally specify the catalog where the source database is located. If you do not specify a catalog, then the default catalog is used.

MongoDB

• Fixed an issue where the collection statistics failed because the statistics object became too large.
• Updated to write collection records in batches.
• Tonic now continues to retrieve documents after a failure.

Oracle

• Removed the uniqueness check for individual columns that are part of a composite index.

PostgreSQL

• Fixed an issue where there was duplicated data from parent tables of inherited tables.
• Improved performance for query to retrieve tables and columns.

Snowflake

• Improved performance for data generation.
• Updated how Tonic uploads files to Amazon S3 to reduce memory usage.

SQL Server

• Fixed DNS resolution for Kerberos.
• Fixed an issue where Kerberos authentication failed with an error that the destination array was not long enough.

The new usage report summarizes the data processed for each table for data generation jobs. The report is a .csv file that you download from Tonic. To download the report, on the Admin Panel, click Download Usage Report.

When certain sensitive loggers are enabled, Tonic now disables log collection.

Fixed an issue on Database View where a column configuration panel would close unexpectedly.

For the TONIC_ADMINISTRATORS environment variable, you can now specify the names of SSO groups to grant administrator privileges to. Previously you could only specify user email addresses.

The Tonic SDK Javadoc now displays correctly.

Restored the ability to import a workspace configuration from Workspaces view.

MongoDB

• Added support for the DBRef datatype.
• Improved performance for collection management.

PostgreSQL

• For the beta Data Pipeline V2 data generation, adjusted the logging level for telemetry-related log messages to DEBUG.
• For the beta Data Pipeline v2 processing, improved parallel processing for constraints that cross tables.
• For the beta Data Pipeline V2 process, reduced the default maximum number of source and destination connections to 8. These are set as the values of the TONIC_JOBFLOW_MAX_SOURCE_CONNECTIONS and TONIC_JOBFLOW_MAX_DESTINATION_CONNECTIONS environment variables. We recommend that you set each value to the number of CPUs on the corresponding database.

Snowflake

• Fixed an issue where preserve destination tables were removed during data generation.

The new FNR generator transforms Norwegian national identity numbers. The FNR generator was added in V857. It included options to specify a range of birthdates and preserve the indicated gender. In V866, removed the date range configuration options. The destination values are now always within the same date range as the source values. The FNR generator also now can be used for columns that have uniqueness constraints. The final digits in the destination value are not a valid checksum.

For the beta Data Pipeline V2 processing, fixed an issue where jobs would hang if they were canceled before the job started.

Fixed an issue where the Foreign Keys view would freeze.

Fixed an issue where when you typed @ to add a user mention to a comment, suggestions for the user did not display.Upgraded to use .NET 7.

"Data science modeling" is changed to "data science mode".

When you configure the SSH tunnel settings for a workspace, Tonic now obscures the SSH passphrase.

MongoDB

• Added a configuration to prevent Tonic from retrieving other information about collections when retrieving a collection list. Addresses an issue where retrieving the collection list took a very long time. To disable the retrieval of other collection information, set the environment variable TONIC_MONGO_DISABLE_COLLECTION_INFORMATION_FETCHING to false.
• Fixed an issue with collection scanning that caused application pages to not load.
• Collection scans are now able to continue when the scan for an individual collection fails. The job logs include warnings for each failed collection.
• For collection scans, for each schema Tonic now limits the number of documents to scan and the length of time for the scan.

PostgreSQL

• Tonic now uses the estimated row count from PostgreSQL statistics to determine the parallelism for a table. Customers should ensure that they have up-to-date statistics for their source tables, especially for large ones.

Snowflake

• For source tables that are assigned Preserve Destination mode, Tonic no longer attempts to add existing constraints to the destination tables.
• Fixed a syntax error in the post-generation processing.
• Fixed an issue where data generation failed with the error "Unable to determine AWS Region".
• Fixed an issue where preserve destination tables were removed during data generation.

SQL Server

• During data generation, Tonic now warns users when a filegroup that exists in the source database does not exist in the destination database.

NOTE: Releases v849 through v854 are removed.

Removed caching of AWS credentials.

For the beta Data Pipeline V2 job processing (available for PostgreSQL only):

• When a job fails, Tonic no longer tries to fall back to the current job processing. In most cases, jobs fail for reasons that are not connected to the processing type. Falling back to the current job processing is not effective.
• Improved performance for subsetting.
• Adjusted the logging level for telemetry-related log messages to DEBUG.

On the Foreign Keys view, when you filter the keys, click the select all option, and then clear the filter, only the matching keys are selected. Previously, the select all option always selected all of the keys.

Fixed an issue where CSV files could not be uploaded to data science mode workspaces.

You can now configure parallelism for sensitivity scans. For relational databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_RDMBS, and the default is 4. For document-based databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_DOCUMENTDB, and the default is 1.

On the table configuration panel for subsetting, Tonic no longer displays a count of post-subset rows before a subset is generated.

Fixed an issue where subsetting data from one workspace appeared in a different workspace.

You can now use UUID columns in the conditions for the Conditional generator.

Fixed an issue where when you deleted a linked column from a column configuration panel, the other linked columns were deleted.

The Timestamp Shift generator can now be assigned to columns where the values use the date format MMddyyyy.

Fixed a regression that caused NPGSQL logging to occur even when it was disabled.

Google BigQuery

• Improved error handling when rows contain invalid data. Tonic now provides a method to look at the data that caused the error. Fixed the handling of rows to prevent errors on certain data.
• Fixed how we create views in the destination database.

MongoDB

• Fixed an issue where Tonic was unable to get the schema for the source data.

Oracle

• Fixed an issue where retries of Oracle commands from transient errors failed.

Snowflake

• Improved resource handling during data generation in order to enable parallel processing.
• Snowflake on Azure no longer requires CREATE SCHEMA permissions in order to support Preserve Destination mode for tables. Snowflake on AWS with Lambda processing continues to require CREATE SCHEMA permissions in order to support Preserve Destination mode.
• When Tonic is unable to create a view in the destination database, it now returns a warning instead of an error.
• Implemented a more accurate method to detect hexadecimal values.

SQL Server

• Tonic now displays the correct generators for columns that are part of a composite unique index.

The new Business Name generator produces realistic names of businesses or companies. The Business Name generator can be consistent with itself or with other columns. It improves on and is intended to replace the Company Name generator, which is now deprecated.

Fixed an issue on Table View where users could not use the delete icon to remove a generator assignment for a linked column.

Fixed an issue where the job details view for a subsetting job did not always show all of the steps as completed.Updated the version of pytorch, which is used for data science mode. This new version addresses some security vulnerabilities.

Fixed an issue where when the Tonic server was air gapped, the Admin Panel did not correctly display the current Tonic version.

Fixed an issue where jobs took longer than expected to complete.

Fixed an issue where the subsetting Graph View did not show how table participation in the subset changed since the most recent subsetting data generation.

Fixed an issue where after a single failure to write logs, the Download Job Logs feature stopped refreshing the logs. Tonic now continues to try to upload logs.

Google BigQuery

• Fixed an issue with the test connection function for the destination database.
• Materialized views and routines from the source database are now copied to the destination database.

MongoDB

• Improved performance for sensitivity scans.

MySQL

• Fixed an issue where subsetting failed because of a data type mismatch between a primary key and a foreign key.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB = true, foreign keys are now correctly enabled on the destination database.

PostgreSQL

• Fixed an issue where the source database permissions check provided a false error about insufficient privileges for sequences.
• For the beta Data Pipeline V2 data generation process, fixed an issue where the data generation process continued even after the job failed.

Snowflake

• Fixed an issue where the presence of comments caused data generation to fail.
• Improved performance for sensitivity scans.

SQL Server

• Added support for user-defined types.

Generator presets are now supported for Enterprise licenses on Tonic Cloud.

For Tonic data encryption, fixed an issue where the previous encryption key environment variable value was saved in the application database, which caused Tonic to use those values even after they were removed.

The Tonic diagnostic logs now include the Tonic worker ID.

Fixed an issue where the Tonic web server would not launch unless the Tonic application database used PostgreSQL v13 or later.

For data connectors other than MongoDB, the sensitivity scan is now parallelized.

Google BigQuery

• Data generation now works correctly when the region that hosts Google BigQuery for the destination database is different from the region for the source database.
• Fixed an issue where failed data generation jobs were incorrectly reported as successful.
• Tonic now handles the TIME data type correctly.

MySQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB=true, fixed an issue where the truncation of tables violated foreign key dependencies, which caused jobs to fail.
• Added an option to enable the TCPS protocol for Oracle database connections. Previously, only TCP was supported. If you enable TCPS, you must also provide a wallet file.
• Tonic now cleans up temporary destination database tables that were created during subsetting data generation.

PostgreSQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.
• During data generation, Tonic now warns users when an extension that the destination database needs is unavailable for installation.

Snowflake

• For both Snowflake on AWS and Snowflake on Azure, you can now configure workspaces to limit the schemas to include.

Fixed an issue where users could only select generators that supported uniqueness constraints for columns that were not unique.

Fixed an issue where admin users who did not have edit permissions on any workspaces could not edit presets from the Generator Presets view.

Improved data generation resiliency against transient failures.

Removed erroneous error messages.

To add AWS credentials to containers, you can now mount to ~/.aws/credentials.

Improved error messaging for Table View.

Fixed a display issue where the column configuration panel was too narrow and required horizontal scrolling.

Exporting or copying a workspace no longer requires the workspace to have a valid source database connection.

Reduced the amount of memory needed to run the Tonic web server.

MongoDB

• Better handling of errors that involve invalid UUIDs.

Oracle

• Updated the required permissions for destination database connections. If SELECT ANY DICTIONARY or SELECT_CATALOG_ROLE cannot be granted, then Tonic can use a selection of ALL_ views (not recommended).
• If TONIC_ORACLE_SKIP_CREATE_DB=true, then external tables are now excluded from the table list in Tonic. Tonic does not process those tables.

PostgreSQL

• Fixed an issue where the Data Pipeline V2 flow would hang.
• Fixed an issue where extensions such as pgcrypto were not transferred when data generation included schema filtering.
• Improved performance when handling constraints.

Snowflake on AWS

• As of V823, you can choose whether to use the Lambda process for data generation, which was previously the only option. By default, Snowflake on AWS uses a new, more resilient data generation process. You only need to use the Lambda data generation process for extremely large volumes of data (hundreds of gigabytes to terabytes).
  
  For existing workspaces, for versions before V826, the new default process is used. To use the Lambda data generation process, you must update your workspace configuration. As of V826, existing workspaces use the Lambda data generation process.
• For the temporary CSV files used to retrieve and write source and destination data, you can now specify to use an external stage instead of an S3 bucket. The option to use an external stage is not available when you use the Lambda data generation process.
• You can now specify different file storage locations for the temporary source and destination data files. In other words, you can have different S3 buckets or different external stages. Note that this option is not available when you use the Lambda data generation process.
• For the new data generation process, fixed an issue where data generation jobs would hang instead of failing.

Snowflake on Azure

• Before it runs a data generation, Tonic now verifies that there is a valid value for the Azure Blob Storage account key, which is set as the value of the environment variable TONIC_AZURE_BLOB_STORAGE_ACCOUNT_KEY.
• Fixed an issue where data generation jobs would hang instead of failing.

For Tonic data encryption, Tonic now only verifies the key for the enabled process. If you only enable decryption, then Tonic only verifies the value of TONIC_DATA_DECRYPTION_KEY. If you only enable encryption, then Tonic only verifies the value of TONIC_DATA_ENCRYPTION_KEY.

Upgraded our Docker images from Ubuntu 20 to Ubuntu 22.04.

Updated to ensure that the Tonic URL reflects the currently active workspace.

Recently started jobs no longer display a start time that occurred several years ago.

On the Data Encryption tab, the option to provide custom initialization vectors is now a toggle instead of radio buttons.

Resolved an issue where Tonic took an extremely long time to load.

Oracle

• Reduced the permissions required to test database connections.
• Changed the required permissions to better support when TONIC_ORACLE_DBLINK_ENABLED is false. For the source database user, you can either grant SELECT ANY DICTIONARY, grant SELECT_CATALOG_RULE, or (not recommended) grant access to the ALL_* views.

PostgreSQL

• Improved the error messaging when testing the connection to the destination database.

Snowflake

• You can now use a connection string to connect to the source and destination databases. Also added support for proxy connections.

NOTE: v805 and v806 are removed.

The new Snowflake on Azure data connector uses Azure Blob Storage to store interim uploaded and generated files.

A copied workspace now includes manual sensitivity designations. A manual sensitivity designation is when you change the sensitivity designation that was assigned by the sensitivity scan to either sensitive or not sensitive.

When the configured encryption or decryption key is not valid - for example, the key is not configured or uses the incorrect size - then Tonic does not allow you to configure Tonic data encryption.

Tonic Cloud now correctly enforces the supported data connectors.

Improved error messaging when subsetting data generation fails because the generator cannot be used with subsetting.When you change the type of Tonic data encryption (decryption, encryption, or both), Tonic no longer clears the decryption and encryption text fields.

MongoDB

• Fixed an issue where workspaces that contained views did not load.

Oracle

• Fixed an issue where the default Oracle NUMBER type was not compatible with the Integer Key Generator.
• Removed an invalid error that was returned when users tested data connections.

PostgreSQL

• For the beta Data Pipeline V2 generation process, improved the error logic to prevent jobs from hanging when errors occur.

SQL Server

• Improved the resilience of data generation to transient failures.

Fixed an issue where the column configuration panel on Privacy Hub incorrectly allowed users to change the configuration of foreign key columns.

The option to create a completely new workspace is no longer available from the workspace management view. You can only copy that workspace and create a child workspace. To create a completely new workspace, use the Create New Workspace option on the Workspaces view.

Fixed an issue where the generator for a column in a table that uses Scale mode could have a configuration option that is invalid for that mode.

Upgraded libraries to address security vulnerabilities.

Fixed an issue where when you applied the Random Timestamp generator to a column, a Bad format string error was returned.

Improved display of long table names in the table details panel on Subsetting view.

MongoDB

• Introduced a new configuration to collapse child fields into a single field based on a regular expression, to reduce the size of the schema. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX provides the regular expression to check the field keys against. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX_THRESHOLD specifies the number of matching fields that causes the fields to be collapsed. A value of 0 indicates to not collapse the fields.
• On Collection View, fixed an issue where toggling the data preview changed the frequency values for documents within documents.

Added a confirmation step when updating the baseline configuration of a generator preset from a column configuration panel.

Fixed an issue where job failures sometimes caused the worker process to crash in a way that prevented new jobs from running.

To improve Tonic web application security, added X-Frame-Options/X-Content-Type-Option headers.

Databricks

• Corrected an issue on the workspace configuration page so that read-only items are no longer clickable.

PostgreSQL

• You can now use connection strings to connect to the source and destination databases.

SQL Server

• Fixed an issue where data generation failed when all URNs in the database have no dependencies.

Setting the default configuration for generators (Requires an Enterprise license)

The Generator Presets view allows you to configure the default configuration for generators. The current configuration is used whenever that generator is assigned to a column.

To update the configuration, you must be an owner or editor of a workspace in the instance.

In the configuration for a column, you can override the saved default configuration, which we call the baseline configuration. You can also revert to the current baseline configuration or save your configuration as the new baseline configuration.

Other updates

On the workspace management view, the workspace options are now included in the collapsed version of the heading.

For free trial users, the option to create a workspace is always visible in the Tonic heading.

For SAML SSO, if the value of NameID is not an email address, Tonic uses the email claim in the SAML response.

Databricks

• Added a TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE environment variable to determine whether new Databricks tables ErrorOnOverride by default. The default value is true, indicating that new tables ErrorOnOverride.

SQL Server

• Changed the default value of BYPASS_MS_XML_PARSING from false to true. The variable indicates whether to convert XML columns to nvarchar(max) to avoid potential XML parsing bugs.

NOTE: Releases 741 and 742 were removed.

Graph View for subsetting

The Configuration tab of Subsetting view now includes a toggle to switch between Table View and Graph View.

Table View is the existing tabular list of tables.

Graph View is a diagram view that displays the tables and the relationships between them. Similar to Table View, when you click a table in Graph View, the table details panel displays for that table. You can configure subsetting from either view.

On Graph View, Tonic adds a marker when the subset configuration for a table changed since the last subsetting data generation. A table might be added to the subset, removed from the subset, or modified within the subset.

Tonic data encryption

The new Tonic data encryption feature, available for Professional and Enterprise licenses, allows you to set up a configuration to decrypt source data before applying a generator, encrypt transformed data before writing it to the destination database, or both.

Tonic data encryption uses AES encryption.

Tonic data encryption requires you to set environment variables for the decryption key (TONIC_DATA_DECRYPTION_KEY) and encryption key (TONIC_DATA_ENCRYPTION_KEY). Both keys must use the same key size - either 128, 192, or 256 bits.

Admin users configure Tonic data encryption from the Data Encryption tab of the Admin Panel.

When you enable Tonic data encryption, the generator configuration includes a setting to indicate whether to use it for that column.

Other updates

Fixed an issue where navigating to Database View from Schema Changes view for a table-specific issue did not apply the correct filters.

Fixed an issue where workspace import could inadvertently add multiple subset targets in the same table.

Added support for ssh-rsa for data connections.

Improved error logging for the Timestamp Shift generator.

Fixed the display for the import workspace dialog.

MongoDB

• The new ObjectId Key generator allows you to de-identify object ID columns.
• Moved the scanning of MongoDB collections into the Tonic worker.

PostgreSQL

• Fixed an issue that caused data generation to fail with the error canceling statement due to statement timeout.

SQL Server

• Fixed an issue where a bug in the SQL Server Management Objects (SMO) library caused data generation to fail when using partitioning in Azure.
• Fixed an issue that caused data generation to fail when system tables were included.

When you log back in to Tonic, it now displays the workspace management view for the most recently viewed workspace.

Improved error messaging for rarely occurring user authentication issues.

Fixed display issues on the Subsetting view where part of the subset results graph did not display and the Configuration tab did not scroll.

Amazon Redshift

• Added the ability to provide filters on tables.

MongoDB

• Fixed an issue where indexes created with an expireAfterSections value are not created in the destination database.
• Fixed a regression where unscanned collections did not display in workspaces.
• Fixed an issue that prevented Tonic from being able to connect to MongoDB-compatible DocumentDB databases.
• Fixed an issue where DocumentDB jobs failed when run from Windows.

MySQL

• Updated to ensure that TONIC_WRITE_PARALLELISM is always 1, to prevent lock timeouts.
• Increased resilience around possible failures when retrieving a MySQL schema during data generation.

Snowflake

• Added the ability to provide filters on tables.

SQL Server

• Improved performance when reading large fields.
• Fixed an issue that caused data generation to fail with the error Sequence contains no matching element.
• Fixed an issue that caused data generation to fail on SQL Server 2014.

As of V724, the AI Synthesizer generator is by default not available. To enable the AI Synthesizer generator, set the environment variable TONIC_NN_GENERATOR_ENABLED to true. If the AI Synthesizer generator is assigned in a table, but the environment variable is false, then data generation fails.

During subset processing, when Tonic encounters a circular dependency, to break the circle, it nullifies the values of one of the foreign key columns. By default, it nullifies all of the foreign key column values. A new environment variable, TONIC_SUBSETTING_AGGRESSIVELY_NULL_CYCLICAL_FKS, controls the behavior. By default, the value is true. If you set the value to false, then Tonic only nullifies foreign key values that do not exist as primary key values in the other table.

Fixed an issue where authentication using Azure SSO failed if the user was a member of a large number of IdP groups.

Improved detection during sensitivity scans of columns that contain birth dates.

MongoDB

• Fixed an issue where adding a new null type field to the data caused data generation to fail.

Oracle

• Fixed an issue where data generation returned the following error: Specified argument was out of the range of valid values. (Parameter 'minSize').

PostgreSQL

• Fixed an issue where tables with generated columns, but no generator assignments, were not correctly copied to the destination database.
• Improved handling and messaging for connection timeouts.

Snowflake

• Fixed an issue where retrying a connection multiple times returned an incorrect error.

SQL Server

• Fixed issues where users could not configure data generation for tables that contained hierarchyid columns.
• Fixed an issue that prevented Preserve Destination from being used on schema-bound tables.
• Fixed an issue that prevented Incremental and Preserve Destination table modes on memory-optimized tables.
• Fixed an issue where column attributes such as the nullability of a column were not correctly reflected in the destination database.
• Fixed an issue where schema binding was not maintained in the destination database.

Redesigned Tonic navigation

As of V719, we have redesigned the Tonic navigation. The left navigation menu is removed. On Workspaces view, click the workspace name to display the workspace management view for that workspace. The workspace management view contains a horizontal navigation bar to provide access to the workspace configuration and generation tools, and a heading menu to provide access to other workspace actions.

On Workspaces view, the first column is now a checkbox to select workspaces to which to apply an option from the Actions menu.

The Tonic application header contains links to Workspaces view and the Admin Panel.

New table filtering for Google BigQuery

As of V717, Google BigQuery supports table filtering. On the table mode selection panel, for tables that use De-Identify mode, you can provide a WHERE clause to filter the records that are included in the destination data.

Other updates

Fixed an issue where Tonic workers did not start after an upgrade when a read-only file system was mounted for custom value processors.

Improved the detection of vehicle identification numbers (VINs) in source data.

Oracle

• Improved generator recommendations for columns that are part of a compound unique constraint.

SQL Server

• Fixed an issue that caused jobs to fail when none of the tables were assigned De-Identify mode.

Fixed an issue where when subsetting used parallel processing, the subsetting steps displayed incorrectly.

Improved the error message that is displayed when unsupported data types are present in a table.

On the Subsetting view, on the row count popup, provided a clearer explanation when the destination row count is larger than the source row count.

Fixed an issue where simultaneously updating workspace permissions for multiple users failed to apply the updates.
Made some small performance improvements to the Regex Mask and Array Regex Mask generators.

Improved the job cancellation logic to ensure that selecting the cancel option actually cancels the job.

When a SAML SSO login is initiated, Tonic now redirects the browser to the correct URL.

When Tonic detects an invalid or deprecated generator, it no longer returns the error message Unexpected generator id {generatorId}, create a dedicated Metadata class for this generator.

MySQL

• Fixed an issue that caused lockouts and timeouts on the destination database.

Oracle

• Reduced the number of destination database permissions that are required for data generation.

PostgreSQL

• When you use the Limit Schemas feature, and you choose to filter to only include specific schemas, Tonic no longer includes all of the extensions in the database. It only includes extensions that are related to the included schemas.
• Values are now truncated to fit the char type before they are written to the destination database.
• Generators that support consistency can now be made consistent with columns that are user-defined enum values.
• The Conditional generator can now use columns that are user-defined enums as conditions to apply sub-generators.
• Tonic now correctly determines whether to truncate a unicode string when fitting the string into a data type that is close in size.
• Improved resiliency to destination database connection failures.

NOTE: Releases v692 through v695 were removed.

Update to PII detection - PII container removed

As of v697, updated how Tonic performs PII detection. The name detection process now also scans for international names with origins from 103 different countries.

The PII detection process now runs from the Tonic worker instead of in a separate container.

To avoid errors when you upgrade, future versions up to v999 will have a placeholder PII container. The container is not active and is not used to run Tonic.

Before you upgrade to V1000 or later, you must remove the PII container from your instance.For a Docker deployment, in docker-compose.yaml, remove the tonic_pii_detection section.

For a Kubernetes deployment:

• Remove the file tonic-pii-detection-deployment.yaml.
• Remove the file tonic-pii-detection-service.yaml.
• Update tonic-worker-deployment.yaml to remove the entry for TONIC_PII_DETECTION_URL.
• Update values.sample.yaml to remove the entry for pii_detection.

If you do not remove the PII container before you upgrade to v1000 or later, then when you upgrade Tonic, you will encounter image pull errors.

Other updates

Improved error messaging on the Tonic UI for failed data generation jobs.

Reduced the telemetry logging for PII detection during sensitivity scans to reduce the performance impact on self-hosted instances with machines that do not have internet access.

Fixed an issue that prevented users from configuring the Random Timestamp generator Date Format when the generator was selected as a sub-generator of the JSON Mask and other composite generators.

Made a minor correction to a data generation log message.

Reduced Tonic slowdowns caused by telemetry.

MySQL

• Fixed an issue that caused data connections to fail with the error The ConnectionString property has not been initialized.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

Oracle

• Fixed an issue where long column names caused data generation to return the error ORA-00972: identifier is too long.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

PostgreSQL

• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

SQL Server

• Users can now assign generators to timestamp and rowversion columns.
• Fixed an issue where a database ID mismatch in the master and database tables prevented data generation from starting.
• Fixed an issue where chained foreign keys caused subsetting jobs to fail.

NOTE: These releases were removed.

Enhancements

In a Databricks workspace, you can now choose to write all of the output tables to one of the following formats: Avro, JSON, Parquet, Delta, CSV, ORC. This setting replaces the previous option to write all of the output tables to Databricks Delta.

When sharing a workspace, free trial users can now invite other users with the same corporate email domain to start their own free trial.

Other updates

Added messaging to the Tonic application about changes to the Tonic license plans.

Fixed an issue with the JSON Mask generator where when users deleted a sub-generator, a different sub-generator was deleted.

Fixed cases where data generation jobs remained in the queued state indefinitely.

Fixed a performance regression that affected workspace loading.

Oracle

• Fixed an issue where a data generation fails when there are index-organized tables.

NOTE: Releases v684 and v685 were removed.

Enhancements

When creating new virtual foreign keys, you can now use the top level field name check box to select or deselect all of the fields that have that name.

Other updates

Fixed an issue where a warning did not display correctly when a subset table was configured with an invalid table mode for subsetting.

For the Custom Categorical generator, we no longer treat newlines as empty strings on numeric columns. Newlines are still treated as empty strings on string columns.

Fixed an issue where an error occurred when a column is assigned the Custom Value Processor generator.

Fixed an issue where data generation with subsetting failed with the following error: Could not load type from assembly 'Allos.Generators'.

Redesigned the generator selection dropdown to better separate the suggested generators from the other applicable generators.

Oracle

• Improved error logging during data generation.

SQL Server

• Fixed an issue where views that reference preserved tables (table mode is Preserve Destination) were not created.
• Added verification to ensure that all of the required tables and objects are created in the destination database before data generation.

Enhancements


Tonic now displays warnings at 30, 15, and 7 days before a Tonic license expires.

Other updates

After a one-click update of Tonic, containers for Docker Compose customers now include the version number in the name.

Addressed an issue for Docker Compose customers where Tonic did not restart properly after a one-click update.
Tonic can now download SAML IDP metadata from a URL. To configure the URL, set the environment variable TONIC_SSO_SAML_IDP_METADATA_XML_URL.

Added helper text to indicate the value format for a database server.

Added the ability to configure the SAML request issuer. To configure the issuer, set the environment variable TONIC_SSO_SAML_ENTITY_ID.

Improved error messaging for WHERE clause validation in subsetting configuration.

Moved some of the temporary files used for data generation from /tmp to /tmp/tonic.

MySQL

• Fixed an error where data generation occasionally failed with the error "Allos.Core.Exceptions.TonicException: No databases selected to overwrite".

Oracle

• Updated to ensure that system tables are ignored.

PostgreSQL

• Fixed an issue with data generation for partition tables.

Enhancements

New command-line tool for Tonic installation - Tonic now offers the Tonic Installation Manager (TIM), a command-line tool to deploy Tonic on either Amazon EKS or a VM.

Other updates

Updated how Tonic performs PII detection. Added additional name values to the information that the detection process looks for.

Fixed an issue where navigating to the workspace edit page sometimes threw an error.

Fixed an issue where SSO account creation erroneously returned errors even though the account was created successfully.

Improved our generator recommendations that are based on the security scan to prevent memory issues for larger databases.

Improved error messaging when Tonic-hosted users attempt to connect to a database on a local network.

Fixed an issue where job log timestamps displayed the wrong month value.

Amazon Redshift

• Corrected a permissions check issue that caused connection tests to fail.MySQL
  
  
  • Upgraded MySqlConnector to version 2.2.5.
  • Fixed an issue where subsetting jobs failed with a "No database selected" error.
  Oracle
  
  
  • Fixed an issue where data generation jobs did not stop when they were canceled.
  • Fixed an issue with applying the Event Timestamps generator to timestamps that have timezone values.
  • Update to prevent Tonic from running out of space during subsetting.
  • Fixed an issue where materialized views and materialized view logs are not torn down after data generation.
  • When TONIC_ORACLE_SKIP_CREATE_DB is true, we now properly truncate tables.
  PostgreSQL
  
  
  • Upgraded the Npgsql library to version 7.0. This is used for both workspace data and the Tonic application database.
  Snowflake
  
  
  • Corrected a permissions check issue that caused connection tests to fail.

Enhancements

New endpoints and expanded API documentation for generators - For generators, the generated Swagger documentation now contains descriptions for each endpoint and for all of the model properties. You can send a request to retrieve a list of available generators. You can now update the generator configuration for a single column, instead of having to provide the configuration for an entire table. There is also a new endpoint to revert a column to Passthrough.

On the Subsetting page, you can now sort the tables based on whether they are in or out of the subset.

Other updates

Improved UI support for timeouts when loading schema information for a source database.

Updated the subsetting process to ensure that a small percentage-based target table contributes at least one row to the subset.

The environment variable TONIC_SUBSET_PARALLELISM is deprecated. Tonic now uses the environment variable TONIC_TABLE_PARALLELISM to control parallel processing for subsetting.

When a data preview or workspace loading process is no longer needed, such as when there are network issues or the user leaves the application, Tonic now attempts to cancel the process.

MySQL

• Improved error messaging when a database connection times out.

Oracle

• Fix to ensure that data generation fails when drop table statements fail.

PostgreSQL

• Improved error messaging when a database connection times out.

SQL Server

• Fixed an issue where some tables were not created in the destination database.
• Improved error messaging when a database connection times out.

EnhancementsTonic now supports SAML SSO providers.

Other updates

Fixed an issue where containers do not start and the following error message is returned: "The directory named as part of the path /var/run/supervisor/supervisord.pid does not exist"

Corrected an issue where in some cases when the job failed because of a source connection issue, the job status was not set to Failed.

In the subsetting configuration, you can now configure a target table percentage value with up to 3 decimal places.

Resolved a potential database connection leak when canceling data generation jobs.

Improved error messaging when a deleted SSO user tries to log into Tonic.

Improved error display for subsetting WHERE clause validation.

Fixed an issue where running a query to validate a subsetting WHERE clause could cause the application to slow down.

Fixed an issue where jobs could fail when subsetting parallelism was enabled.

The Conditional generator now allows you to configure a selected sub-generator to be consistent with other columns. The selected generator must support consistency with other columns.

Databricks

• Updated the Databricks job names to use the format tonic_workspaceName/workspaceId + jobID.
• Improved performance for page loads.

Google BigQuery

• Added support for the JSON data type.

MongoDB

• Fixed a minor flickering issue on the Collection View.

MySQL

• Improved logging of errors that produce data generation warnings.

Oracle

• Corrected an issue where we displayed foreign keys that referenced a schema outside of the workspace.

PostgreSQL

• Fixed an issue with handling HStore types during data generation.
• Improved logging of errors that produce data generation warnings.
• Improved management of partition tables.

Spark

• Improved logging for data generation.

SQL Server

• Fixed authentication handling for Active Directory. Tonic supports NTLM v2 authentication.

Enhancements

From the Subsetting view, you can now view the table configuration and results summary for previous subsetting data generation runs. On the Previous Runs tab, you select the run to view the details for. Note that you can only view the details for jobs that you run after you upgrade.

Other updates

Corrected an issue for instances using a Docker version before 20.10.10 that prevented Tonic processes from starting after an upgrade from version 642 or below to version 643 through 647.

Corrected an issue where estimated data generation time was logged as a negative number.

Improved the performance and resiliency of the subset WHERE clause validation.

Improved the cleanup of database connections after they are used.

Improved error message when the source database version is newer than the destination database version.

Google BigQuery

• Fixed a user interface issue that prevented a new workspace from being created.
• Tonic now supports post-job scripts in BigQuery workspaces. Post-job scripts run inside of transactions. They are limited to statements that are supported within transactions.

Oracle

• The minimum supported version for the Oracle data connector is now Oracle 12c. Tonic no longer supports 11g. As a result of this change, the Oracle Helper is no longer needed. After upgrading, you should remove Oracle Helper from the list of containers to fetch.

Snowflake

• For database connections, the server name should not include https://. Tonic now detects and removes https:// if it is entered.

Spark

• Eliminated a vulnerability that could allow users to see the results of statistics jobs that they did not have access to.

SQL Server

• Fixed an issue with loading tables in SQL Server 2012 and below

Enhancements

Overriding table mode and generator configuration in child workspaces - The workspace inheritance feature now allows child workspaces to override the table mode and generator configuration from the parent workspace. Database View and Table View indicate when the parent configuration is overridden, and provide options to reset the configuration.

Other updates

You can now assign the SIN generator to fields that have uniqueness constraints.

The Address generator now only allows you to link columns that contain the following types of address values - City, City State, Country, Country Code, State, State Abbreviation, Zip Code, Latitude, Longitude.

Improved memory handling for subset processing of downstream tables.

On the Subsetting view, when the database returns an unrealistic value for the subset size, we replace the value with a warning message.

When using Helm to deploy Tonic via Kubernetes, integer environment variable values that are longer than 6 digits might be converted to scientific notation. To avoid this issue, Helm recommends that you enter values in quotes. Tonic can now parse scientific notation to better handle this behavior for values that are not in quotes.

Fixed an issue where linking columns that have the same name as columns in another table caused the generators to be removed from those other columns.

Made minor memory improvements throughout the Tonic application.

Corrected an issue where Tonic displayed a "subsetting in progress" message before the processing started.

Amazon Redshift

• Fixed an issue where data generation failed when the name of one table was part of the name of another table (for example, Test and Test1).

MongoDB

• Generators are now applied correctly to documents that are nested in arrays.

Oracle

• Removed the permissions check for reading truncated tables.
• Fixed an issue where DBLinks created for data generation were not removed when the generation failed or was canceled.

Snowflake

• The Categorical generator no longer returns errors when applied to binary, time, and date columns.
• Improved the performance for loading the user interface when the data contains a large number of foreign keys.

Spark

• The Audit Trail entries on Privacy Hub now display the correct table and column names.
• Corrected an issue where sub-generators for the Conditional generator were not applied correctly.
• Improved error messages when testing the connection to a Databricks database.

Improved user signup experience for the hosted version of Tonic.

Made some small styling updates to Database View. Removed the dropdown to uncheck or check all of the tables. Removed the highlighting on the table mode dropdown for tables that contain columns with assigned generators. Changed the Filter button labels to Filters.

For subsetting, verify that primary key fields are not assigned a non-primary key generator.

Amazon Redshift

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

MongoDB

• Fixed an issue that caused errors when applying the Timestamp Shift generator.

Snowflake

• Improved error messaging for permissions for data generation.
• Improved performance for data generation.

Spark

• Improved error messaging for unsupported versions.

SQL Server

• Improved handling of alphanumeric sequences.
• Case-sensitive databases now connect correctly to Tonic.
• Corrected how indexing recommendations for subsetting are determined.

Enhancements

Added filters for upstream subset records - In the subsetting configuration, for upstream related tables, you can now filter the records to include based on either a date value or a WHERE clause. Upstream tables contain data that has a foreign key that references a primary key in a target table. Upstream records are optional, and are not needed for referential integrity.

The Confirm Generation panel now provides access to tips to improve data generation performance.

Other updates

Tonic no longer offers DB2 as a data connector type.

On the Job History list, changed the job type "Privacy Scan" to "Sensitivity Scan".

Corrected a display issue in Privacy Hub where the column details panel extended past the bottom of the page.

For the JSON Mask generator, the path expression selection tool now works for arrays and for keys that contain spaces or special characters.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Added a message to notify users when queries used to validate a subset WHERE clause run for a long time.

Tonic now continues to record logs when a job fails.

Improved memory usage when running data generation.

Amazon Redshift

• Fixed an issue where Tonic failed to display an error for an invalid generator assignment.

MongoDB

• Corrected a subsetting issue that occurred when multiple foreign keys in one collection referred to the same primary key in another collection.

MySQL

• Updated the MySqlConnector driver to 2.2.0.

Oracle

• Downgraded the Oracle drivers to 3.21.70 after encountering issues with 3.21.80.
• Corrected an issue with the Event Timestamps generator.

PostgreSQL

• When executing psql and pgdump commands, we properly enforce the requirement to use SSL for connections.
• Corrected connection issues by escaping special characters in password files.

Snowflake

Spark

• You can now apply the Timestamp Shift generator to string fields that contain datetime values.

Enhancements

Tonic can now integrate with Keycloak for SSO authentication.

Other updates

Fixed an issue where users received the error HTTP/1.1 415 Unsupported Media Type from the /api/GenerateData/start endpoint.

Improved performance for the /api/job endpoint. As part of this update, the endpoint only returns the most recent 100 jobs.
Improved loading time for the Job History view, which now displays only the 100 most recent jobs.

Updated how Tonic determines whether to use SSL for connections.

The Add Foreign Key Relationships tab on the Foreign Keys view now expands to fill the height of the browser.

Fixed an issue where a failed cleanup step incorrectly marked a Tonic update as failed.

Amazon Redshift

• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.

MongoDB

• When performing a schema scan, Tonic now displays a progress indicator.
• Improved subsetting performance.
• Fixed an issue where subsetting failed because of duplicate keys.
• Corrected how Tonic handles views.

Snowflake

• Fixed an issue where the Tonic user interface could not load when the database included foreign keys.
• Improved memory management.
• The Lambda role now requires the s3:ListBucket permission.

Enhancements

If data generation is not blocked on all schema changes, Tonic now displays a dismissible warning when there are non-conflicting schema changes. Conflicting schema changes always block data generation.

For the GenerateData API endpoint, added an optional clientResourceId query parameter. When you provide a value, then jobs that have the specified clientResourceId run serially instead of in parallel. The check applies to all jobs across the instance, regardless of whether they belong to the same workspace.

Other changes

Fixed an issue that made it difficult to click the Cancel Job button in the Job History list.On the workspace configuration view, the source and destination database details are now populated correctly when you refresh the page.

When granting access to a workspace, improved how we display long names and email addresses.

For tables that use Scale mode, removed Passthrough from the generator selection dropdown list. Previously, the option displayed in the dropdown list even though it couldn't be selected.

Improved performance for looking up tables and columns.

Corrected an issue where the same environment variable with different cases caused Tonic to crash.

When logging in using single sign-on (SSO), when the email address uses a different case from an existing username-password account, it now resolves to the same user.

For instances deployed using Docker Compose, Tonic now cleans up old, unused images.

Improved performance when running generators on de-identified tables.

Corrected an issue where the Event Timestamps generator produced unexpected values for linked columns.

Improved error handling for account creation.

MongoDB

• Updated MongoDB.Driver and MongoDB.Bson to 2.18.0.
• Subsetting jobs with foreign keys can now run on collections that have lower permissions. To enable this, set the environment variable TONIC_BYPASSDOCUMENTVALIDATION_ON_DOWNSTREAM_KEY_MERGES to false. The default value is true, which means that the destination database requires the dbAdmin role in addition to readWrite.

Oracle

• Tonic now automatically uses the optimal table prefix for schema queries. The TONIC_ORACLE_DICTIONARY_TABLE_PREFIX environment variable is removed.
• Improved error handling for missing permissions for table size queries.

SQL Server

• Improved performance for table queries.

Enhancements

The Subsetting view now shows for each table the percentage of data that is included in the destination database.

Other updates

The workspaces view no longer briefly flashes a message indicating that the workspace cannot be found.

Added the ability to display trace information in the log files. To enable the trace information, set the environment variable TONIC_LOG_TRACES to true.

Fixed an issue that caused an incorrect warning to display when linking columns that were assigned the Custom Categorical generator.

Fixed a data type error in AI Synthesizer for models that only contain categorical data.For the Event Timestamps generator, Tonic now prevents the generator from being assigned to a time-only value.

Amazon Redshift

• Added ServerCompatibilityMode to the Redshift connection string to prevent connection errors.

Databricks

• Added support for Databricks 10.x.

MongoDB

• Fixed an issue where Tonic generated collection names in Preserve Destination mode that exceeded the maximum length limit.

MySQL

• The MySQL data connector now supports the ability to use the DELIMITER command in post-job scripts.

Oracle

• Improved performance for schema queries.
• Improved performance for databases without database links.
• Restored missing information for loader errors.
• Improved performance for applying indexes and constraints.
• Added the environment variable TONIC_ORACLE_REDO_LOG_ENABLED, which by default disables recovery information writes to REDO LOG files.
• Improved performance for retrieving tables and columns.
• Fixed an issue with loading custom types.
• Fixed an issue where multiple indexes on the same column caused data generation to fail.
• Corrects a regression from V611.
• When fetching constraints, Tonic can now support primary key indexes that have the same name.
• Excluded views from table queries.

PostgreSQL

• Updated the process of loading tables to improve performance and eliminate an error indicating that PostgreSQL types could not be found.

SQL Server

• Fixed an issue where Preserve Destination and Incremental table mode could not be used in tables with foreign key relationships.
• Tonic now provides a more informative error when data generation fails because the database is not accessible.
• Tonic now properly handles identifiers with single quotes.
• Fixed an issue that caused a worker process to crash when using Azure to connect to the database.
• Added additional logging of the available databases.
• Improved handling of schema and table names in privacy scans.

Enhancements

You can now generate DEBUG level logs for the Tonic API. To do this, set the environment variable TONIC_CONSOLE_LOG_LEVEL to DEBUG.

Tonic now supports logging for long-running queries. The environment variable TONIC_LONG_RUNNING_QUERY_LOGGING_INTERVAL provides the interval in minutes for logging queries. By default, Tonic generates a log entry for a long-running query every 10 minutes. To see this information, TONIC_CONSOLE_LOG_LEVEL must be set to DEBUG.

Other updates

Terminology change - In the Tonic documentation, we have changed the term "mask generator" to "composite generator".

Fixed an issue that prevented the use of Preserve Destination and Truncation modes on tables and collections that had names that were close to the maximum length limit.

You can now assign the Integer Key generator as a sub-generator for string values in composite generators.

For Google SSO, fixed an issue where users who did not have a group membership could not access Tonic.

Improved performance for format-preserving encryption (FPE), which is primarily used for key generators.

Improved generation performance for tables that use De-Identify table mode.

Improved error display when reading data from the source database.

Fixed an issue where values for columns that were assigned the Passthrough generator were being masked with 1s and 0s.

The User Settings page no longer displays the password change option for SSO users.

For the JSON Mask generator, fixed an issue where a large matching value caused the buttons to move off of the configuration dialog.

Fixed an issue where data connection pooling caused data generation to fail because of colliding queries.

Google BigQuery

• Fixed an issue where source data was not loading properly into the Tonic user interface.

MongoDB

• Fixed an issue where the generator configuration panel closed unexpectedly after making edits.

Oracle

• Added a warning when a generator produces a value that is truncated in the destination database.
• When Tonic retrieves table sizes, the results now include the size of the associated large objects (LOBs).
• To help with troubleshooting, Tonic can now determine which Oracle patches were applied to a database.
• Fixed an issue that caused errors when reading a large object array.
• Added additional checks to the data connection tests.
• Updated our Oracle Driver to 3.21.80.
• Fixed an issue where primary keys could not be copied to another index.
• Additional performance improvements.
• Corrected how we check the current user's database privileges.

PostgreSQL

• Added an option to not refresh materialized views. To turn off the refresh, set the environment variable TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS to false.
• Fixed an issue with parsing PostgreSQL schemas in Windows environments.
• Fixed an issue where data generation failed when the data contained read-only arrays.

Snowflake

• Fixed an issue where data generation failed when the schema definition (DDL) contained row access policies.

SQL Server

• Added permission checks to the data connection tests.
• Fixed an issue where Tonic processes failed.

Improved the performance of Table View, particularly when scrolling.

Reworked data generation to better group generator errors.

On the Subsetting view:

• Improved accuracy of the row counts for small tables.
• The table list shows whether each table is in the subset and the number of pre-subset and post-subset rows even when subsetting is not enabled for data generation.

The health check for the PyML container can now use HTTP instead of HTTPS.

MongoDB

MySQL

• Improved performance when loading workspaces.

Oracle

• Fix to improve query performance.
• Added permission checks when testing database connections.

PostgreSQL

• Tonic now applies the source database check constraints on the destination database after data generation is complete.

Snowflake

• Fixed an issue where data generation failed when there were tables that used Preserve Destination table mode.

Reverted an update from v580 that caused slow performance when retrieving the list of tables for a workspace. In some cases, this caused Tonic to indicate that no tables were available.

Fixed an issue where custom value processors interfered with the display of configuration options for the SSN generator.

Fixed an issue where the dropdown lists to select a column generator periodically scrolled to the top of the screen.

Improved performance when navigating among Tonic views.

Improved the list of suggested sub-generators for the Conditional generator.

Improved the data generation process to prevent jobs from hanging when an error occurs.

Fixed an issue where the post-subset generation row count always returned 0.

Updated the Tonic logging framework.

Google BigQuery

• Improved handling of timestamp and datetime fields.

MongoDB

• Fixed an issue where workspaces crashed when running data generation on new collections.

MySQL

• Corrected the handling of the BIT data type.
• Improved performance when loading workspaces.

Oracle

• The Constant generator now works correctly for RAW columns.
• Stopped using source database constraints that were in the recycle bin to populate the destination database.
• Improved performance when retrieving schema information and loading workspaces.
• Added the OracleDriverAnalyzer tool to analyze Oracle performance.

PostgreSQL

• Display an error when an excluded or included schema does not exist in the source database.
• Fixed a permissions error that occurred when creating a new workspace using Azure PostgreSQL for the destination database.

Spark

• Fixed an issue with the workspace configuration view that prevented users from creating and updating Spark-based workspaces.

SQL Server

• Fixed an issue that caused a data generation error when XML indexes had dependencies on other indexes.
• Upgraded SqlClient to 5.0.0.

Enhancements

Assigning generators from Schema Changes view - On Schema Changes view, new columns, changes to column data type, and changes to column nullability have a Select dropdown list that includes an option to assign a different generator to the column and then resolve the issue.

Complete list of blocking issues for data generation - When data generation is blocked, the generation panel now displays all of the blocking issues. This allows you to correct all of the blocking issues before you attempt to run data generation again.

Other updates

Made some visual updates to the Tonic navigation pane and the Tonic login panel.

On the Job History page, the details popover for queued jobs now points to the correct job.

In Database View, the generator list for the Applied Generators filter is now correctly alphabetized.

Fixed an issue where the dropdown arrow for the table mode selector was not always clickable.

When importing a workspace, Tonic now validates that columns do not have multiple generators assigned to them.

Updated to provide clearer error messages when there is an issue with an assigned sub-generator.

Fixed an issue where Privacy Hub sometimes did not reload after a new sensitivity scan.

The database type filter now includes all of the available Spark database types.

Corrected the link to the Tonic privacy policy.

Fixed an issue where jobs failed when multiple tables with the same name in different schemas were assigned Preserve Destination table mode.

Timestamp Shift is now the recommended generator for Date and Timestamp columns.

Corrected the display of available buttons on the Tonic application.

MongoDB

• Improved error handling for database summary queries.
• Added a warning about subsetting performance for percentage subsetting target tables on MongoDB versions before 4.4.2.
• Fixed a memory usage issue with loading Collection View when the data contains large arrays.

Oracle

• A new environment variable, TONIC_ORACLE_DATA_PUMP_PARALLELISM, allows you to choose the maximum number of threads for parallelization for Oracle Data Pump.
• Optimized queries for better performance.

Spark

• Added the ability to specify the proxy user when using Livy.
• Fixed an issue where users could change the configuration type when editing a Spark workspace.

Enhancements

Subsetting results - The Subsetting view now displays the results of the most recent subsetting data generation run. The information includes:

• When the job began and ended.
• The percent reduction from the original source data to the subset destination data.
• The percentage of the source data that is included in the subset destination data.
• The volume of data in the source data and the subset destination data.

Schema changes filter on Database View - On Database View, the advanced filters now include an option to only display columns that have unresolved schema changes. This filter is not combined with other filters. When you filter for unresolved schema changes, the other column filters are disabled.

Larger WHERE clause editor for subsetting - For WHERE clause target tables, you can now display an editor with a larger text area for entering the WHERE clause. This provides better support for longer, more complex WHERE clauses.

Other updates

Fixed an issue that prevented users from deleting more than one tag from the Edit Workspace view.

Added the ability to run Tonic workers, the Tonic web server, and Tonic notifications on Heroku.

Updates to improve handling of canceled jobs, both when users cancel jobs and when jobs fail.Improvements to data generation memory handling and performance.

Improved the Synthesis Report for AI Synthesizer.

Fixed an issue where adding constraints to a destination database resulted in deadlocks.

Oracle

• Update to allow the JSON Mask generator to be used on user-defined types (UDTs).

PostgreSQL

• Fixed a regression that caused sequence fetching errors in v9.6 and earlier.

Spark

• Improved performance for the Continuous generator.
• Enabled partition filter validation on Hive.

SQL Server

• Added support for full text catalogs in Server Management Objects (SMOs).
• Fixed an issue to enable the correct handling of schema, table, and column names that contain single quotes.
• Made a fix to correctly display error messages.

Enhancements

Workspace inheritance - For enterprise customers, the workspace inheritance function allows you to create child workspaces that automatically inherit source data and Tonic configuration from their parent workspace. Changes to the parent workspace configuration are copied to the child workspace. You can override the subsetting configuration and post-job scripts in a child workspace.

Links from Schema Changes to Database View - Added links from Schema Changes entries to Database View. The links automatically filter Database View to only include the affected column or table. The links only display for columns and tables that are in the source database. Removed columns or tables do not have links to Database View.

Schema filtering for PostgreSQL - For PostgreSQL workspaces, when you create or edit the workspace, you can specify a list of schemas to either include or exclude from the source database.

User profile pictures - From the User Settings view, Tonic users can now upload a user image for their account. For SSO providers that support user images, the image from the SSO is used by default.

Zip code configuration for HIPAA Address generator - A new configuration setting for the HIPAA Address generator allows you to determine how the generator sets the zip code. If the setting is off, the generator uses the current process, which replaces the last two digits of the zip code with zeros. For low population areas, the zip code is all zeros. If the setting is on, then the generator selects a real zip code that starts with the same three digits as the original zip code. For low population areas, if a state is provided in the data, the generator selects a random zip code from that state. Otherwise it selects a random zip code from the United States.

Other updates

Minor updates to the Subsetting view. New icons for the subsetting summary and the inbound and outbound relationship counts. Added a Use subsetting toggle to indicate to use the subset configuration for data generation. This toggle is synchronized with the same toggle on the Confirm Generation panel.

Fixed an issue where Tonic logged users out between browser sessions more often than expected.

Improved performance for the JSON Mask generator.

Fix to ensure that the AI Synthesizer is canceled when a data generation job that includes AI Synthesizer is canceled.

Fixed an issue where a data generation job would hang if there was a failure.

Fix to address an issue where after a one-click update, the Tonic version unexpectedly regressed to an earlier version.

Fix to ensure that cross-table commands do not run in parallel, which could cause deadlocks in the database.

MongoDB

• Fixed an issue that prevented the Regex Mask generator from being applied to fields in arrays.
• Updated to require 3.6 as the minimum server version for MongoDB.

PostgreSQL

• Sequence values are now copied over correctly.

Spark

• Fixed a data generation error in Hive and Livy that occurred when a source table had non-lowercase column names.

Enhancements

Filtering workspaces by tags - On the Workspaces view, you can filter the workspaces by the assigned tags. In the Tags column heading, click the filter icon to display the list of applied tags. Check the checkbox for each tag to include. The list is filtered to include workspaces that have at least one of the selected tags.

Automatically resolving schema changes - Schema changes are now resolved or dismissed automatically when you update the table or column configuration.

• For a new table, the schema change is dismissed when you set the table mode to Truncate or Preserve Destination.
• For a new column, the schema change is dismissed when you assign a generator to the column.
• For a column that has a conflicting data type or nullability change, the schema change is resolved when you assign a different generator to the column.

Other updates

Corrected the number of rows for out-of-subset tables. No longer show 0 when out-of-subset tables are processed.

Improved error messaging when a where clause for a subset target table is invalid.

Corrected an issue where the subsetting table configuration was not handled correctly.

The job types filter on the Job History view now only shows valid types for the workspace.

Improved performance for data generation.

Improved performance for the AI Synthesizer.

Improved reporting of subsetting progress when parallelism is enabled.

MongoDB

• You can now assign the Preserve Destination mode to a collection.
• Subsetting and foreign key management is now enabled for all customers.

Spark

• For workspaces that use Livy, you can now assign the Continuous generator and Noise Generator as sub-generators for a mask generator.
• Improved performance for the Categorical generator.
• Improved performance for the HIPAA Address generator.

Enhancements

You can now enable parallel processing for subsetting. The TONIC_SUBSETTING_PARALLELISM environment variable sets the number of steps to process in parallel.

Other updates

Made a couple of small improvements to the AI Synthesizer generator.

Fixed an issue where users could not change the percentage on a subsetting target table.

Improved how we sort tables for parallel processing to improve efficiency.

MongoDB

• The Categorical, Current Date, Date Truncation, HIPAA Address, and Unique Email generators now work on Mongo array fields.
• Linking for the Continuous generator now works correctly.

Spark

• Added support for Spark 2.3 on Livy.

Enhancements

On Privacy Hub, the term "Unprotected Sensitive" is replaced with "At-Risk". The protection status counts exclude columns that are not included in the destination database.

On Database View, added an option to filter by whether a column is included in the destination database. A new At-Risk toggle provides a shortcut to filter for columns that are included, marked as sensitive, and not assigned a generator. The Column Type filters, which filter columns based on whether they are a primary or foreign key, are changed from checkboxes to toggles.

In the Privacy Report, added Not Included as a value for ColumnPrivacyStatus, to identify columns that are not included in the destination database. The value Protected replaces the current values Masked and Anonymized, which are moved to a new ProtectionType column.

Other updates

Improved the display of the Database View advanced filter for smaller screens.

Fixed an issue with the generation of the API reference.

Began to log latencies each hour for source and destination databases.

MongoDB

• In Collection View, improved the display of key columns on smaller screens.
• Corrected an issue that resulted in duplicate schema change results.
• Fixed the collection scan status when a new scan is started.
• Single document view no longer reloads when you apply a generator.
• Corrected an issue where selecting a different collection to view applied updates from the previous collection.
• Fixed an issue where the Null generator could not be removed from a field.

MySQL

• Fixed an issue where the Tonic application could not load when table and row size estimates were not available.
• Increased the connection resiliency for write operations.
• Improved handling of different character sets in MySQL.

When you create a sample workspace, it now includes a tag called Sample Tag.

Fixed an issue where the Update Tonic button was not displayed correctly.

Improved the user interface for activating new hosted accounts.

Improved message to notify users that the subset configuration changed since the last subsetting data generation.

The HIPAA Address generator now works correctly for US addresses.

MongoDB

• You can now link fields in MongoDB collections.
• Conditional generators can now be applied to array elements.
• The Random Timestamp generator now works correctly on datetime columns.

Oracle

• Data generation can now work without DBLink.

PostgreSQL

• Completed additional updates to support cross-schema types.

Added a new Max Categorical Dimension parameter to the AI Synthesizer configuration. This parameter controls the dimension of each column that has categorical or location encoding. If a column contains more distinct categories than this parameter, the most frequent categories are embedded as distinct one-hot vectors. The remaining categories are combined into a single one-hot vector.

Improved error identification for an invalid WHERE clause in subsetting configuration.

On the subsetting page, for tables that were not previously in the subset, the row count is now correctly represented as unknown instead of 0.

Fixed an issue with the Tonic update option in the Tonic application.

Configuring a subset target table to include 100% of the records no longer causes an error during data generation.

Removed connection pooling from Tonic workers to address database connection issues during data generation.

MongoDB

• Added support for partial indexes.
• Fixed an issue where the configured generators were not applied.
• Indexes that use the collation option are now properly recreated in the destination database.
• The subsetting user interface now uses the correct terminology for MongoDB.
• Improved performance for Mongo subsetting when handling downstream tables.
• Fixed percentage-based subsetting for Mongo versions before 4.4.2.

MySQL

• Added support for HASH partition parallelization.
• Running data generation on masked and passthrough tables with ranged sub-partitions no longer results in duplicated data.
• Added support for parallel uploads with ordering.

Oracle

• Added new environment variables (ORACLE_TRACE_LEVEL, ORACLE_TRACE_FILE_LOCATION, ORACLE_TRACE_FILE_MAX_SIZE, and ORACLE_TRACE_OPTION) to enable Oracle tracing.
• Added support for datetime components in composite keys for subsetting.
• Unique indexes are now detected and users cannot apply generators that might violate the enforced uniqueness.
• Improved performance for data generation.
• Improved handling of schema names during data generation.
• Users are no longer incorrectly removed from the destination database.

PostGreSQL

• Upgraded npgsql to address an issue with cross-schema types.

Spark

• Added support for Kerberos authentication for HDFS with Spark / Livy.
• Added support for repartition and coalesce options for Spark EMR and Livy.
• Repartition and coalesce options can now be saved on Databricks.
• Added support in Hive for varchar and char fields that have lengths.

Enhancements

On Privacy Hub, the list of unprotected sensitive columns is replaced with a Database Tables list. The Database Tables list summarizes the protection status of each table in the source database. The Privacy Status column summarizes the protection status of the columns in the table. It provides access to the same column details and configuration options as the protection status panels at the top of Privacy Hub.

You can now filter the Workspaces view based on your assigned role in the workspace. For example, you can only display workspaces for which you are an Owner or an Editor. To use the role filter, click the filter icon in the Role column heading. For admin users, the role filter includes a Role assigned toggle to allow them to only see workspaces that they have a role in.

Other updates

Customers in the Basic license tier can now transfer ownership of workspaces and assign workspace roles to other users.

New users are now logged into Tonic immediately after they create their Tonic account.

For AI Synthesizer:

• The modeling now incorporates static features for each entity across a sequence of events.
• AI Synthesizer now queries to retrieve the actual minimum/maximum values of relevant columns, instead of taking the minimum and maximum of a sample of the data.

To update a license key, self-hosted instances that do not have an admin user can set the license key as the value of the TONIC_LICENSE environment variable. Tonic ignores the variable in instances that have an admin user.

The Workspaces view no longer waits for users to finish applying several new filters in quick succession before it fetches new results.

On the Job History view, fixed an issue where the copy job ID and download logs icons flashed on hover. Removed an error that flashed when job details were displayed.

Fixed the parallel processing for subsetting.

MongoDB

• Indexes are now copied to the destination database.
• Percentage-based subsetting is now supported in versions earlier than 4.4.2.
• Deleted collections are no longer displayed.

MySQL

• Improved memory handling for uploaded CSV files.

Oracle

• Long and long raw columns are no longer converted to blob or clob.

Spark

• Added Maven repository and artifact information to use the SDK Launch method to download the Spark SDK.
• Improved the performance of data generation on Databricks when using Job Cluster. Added support for SQL Warehouses on Databricks.

Added Sequence Length Loss Factor and Order Column Loss Factor model configuration options for events data to the AI Synthesizer. Sequence Length Loss Factor indicates the importance of realistic sequence lengths in the model. Order Column Loss Factor indicates the importance of realistic column value ordering in the model.

For the Categorical generator, differential privacy is now off by default.

Increased the amount of time after which an inactive job is assumed to be canceled.

Amazon Redshift

• Fixed an issue that caused the system to crash when you clicked Test Connection during workspace creation.
• Fixed an issue that prevented more than one generation run per version.
• Fixed a HIPAA resource issue that caused data generation to fail.

MongoDB

• Fixed a subsetting issue that caused errors when there was missing data.

MySQL

• Tonic now handles the BIT data type correctly.

Oracle

• Decimal values that are larger than the dotnet decimal data type can now be handled.

Snowflake

• Fixed an issue that caused the system to crash when you clicked Test Connection during workspace creation.
• Fixed an issue that prevented more than one generation run per version.
• Fixed a HIPAA resource issue that caused data generation to fail.

Spark

SQL Server

• Synonyms are now created correctly during data generation.
• Fixed an issue with subsetting based on a percentage of a target table.

New features and enhancements

Redesigned the user experience for the subsetting feature. The new subsetting view displays a list of the source database tables and indicates whether each table is in the subset. When you click a table, the table details display in a new right-hand panel. From the details panel, you can configure target tables. You can also identify lookup tables (previously referred to as reference tables), and indicate how to handle tables that are not in the subset.

If subsetting is configured, when you run a data generation job, you can enable or disable subsetting.

Added latitude and longitude processing to the HIPAA Address generator.

You can now filter Database View based on the applied generator. On the filter panel, in the Applied Generator field, you can provide the list of generators to include.

Redesigned the Schema Changes page. The Actions list is now called Conflicting Schema Issues. Tonic provides clearer warnings when a schema change resolution will result in a change to the workspace configuration. Resolving a conflicting issue or bulk dismissing non-conflicting issues now includes a confirmation step.

Other updates

Renamed the Events generator to Event Timestamps.

Fixed some small display issues in the new Privacy Hub and Subsetting displays.

Improved event generation for the AI Synthesizer.

Improved how we retrieve CloudWatch logs to include the job ID and to use the correct Tonic version.

Corrected the processing of downstream, multi-index tables during subsetting. Corrected an issue in the initial fix.

Improved memory usage during data generation.

Amazon Redshift

• Corrected the handling of time zones in timestamps.
• Fixed test connection and generation in response to an AWS API change.

MongoDB

• UUIDs larger than 16 bytes are now truncated.
• Improved the use of MongoDB resources.
• Improved the display format of LUUIDs.

PostGreSQL

• Corrected how permissions are validated for Heroku.

Oracle

• Tonic now prevents data generation jobs from running when the Oracle versions for the source and destination database do not match.
• Improved the performance of writes to the destination database.
• After deleting a table in the destination database, Tonic now also clears out the recycle bin.
• Fixed generation for LONG and LONG RAW fields.

Snowflake

• Fixed test connection and generation in response to an AWS API change.

New features and enhancements

We enhanced Privacy Hub to add expanded top-level metric panels. These panels show the number of unprotected sensitive columns, protected columns, and unprotected non-sensitive columns. From these panels, you can display column details, select and configure the column generator, view sample data, and add column comments.

You can now add or update a Tonic license from the Tonic application. For a new instance, you are prompted to provide the license key before Tonic displays the login screen. Tonic displays a message when the current license is expired. The Admin Panel includes an option to update the license key.

For the Address generator, added City with State and City with State Abbr to the available options for the column format. You use these options for column values such as San Francisco, California or Boston, MA.

Tonic now supports subsetting for MongoDB databases.

Workspaces can now have a description (up to 200 characters) as well as a name. Use the description field to provide additional context for the workspace and how it is used.

In the Tonic API, you can now sort workspaces based on the last generation date.

Other updates

Tonic now prevents a job from running when the worker determines that the server is running a different version of Tonic.

Refactored the underlying implementation of the subsetting feature.

Made some small memory improvements for data generation.

Implemented performance improvements when applying parallel constraints.

Corrected errors for edge cases related to the Audit Trail.

When the selected workspace changes, the identifier in the URL is now updated correctly.

The workspace configuration is migrated to a data type that enables more efficient querying.

Tonic now validates uploaded foreign keys against the table definitions.

MongoDB

• MongoDB aggregations can now use temporary files on disk to store data that exceeds the MongoDB size limit. This expands the possible generations for a MongoDB database.
• Corrected the generation and display of UUIDs.
• ObjectIds can now be used as primary keys for subsetting in MongoDB.
• Improved the usability of the Add Foreign Key Relationships tab on the Foreign Key Relationships view.

Oracle

• A destination database can now have more than 1000 tables that have the table mode set to Preserve Destination.
• Improved how Tonic handles maximum lengths when it generates the following data types:NCHAR, NVARCHAR2, CHAR, VARCHAR2

PostgreSQL

• Batch sizes are now set dynamically based on the average row size.
• Memory improvements for PostgreSQL data generation that involve large rows.

Spark

• Improved support for foreign keys.
• Added support for Apache Livy and HDFS.
• Improved performance and added SDK support for the Integer Key generator.

SQL Server

• Eliminated duplicate default constraint URNs during database creation.

Refreshed the Audit Trail user interface on Privacy Hub. The new Protection Audit Trail provides a paginated list of the updates to the sensitivity designation and generator assignments.

Deep links now work correctly when you use Google SSO to authenticate.

Error messages from Oracle are now displayed in response to invalid where clauses in subset configuration.

Made minor memory improvements to the Address generator.

Snowflake

• Reduced the frequency of schema change detection on Snowflake databases. This can result in cost savings on Snowflake clusters, because the clusters can sleep more often.
• Snowflake generation now works correctly when there are foreign key constraints.

SQL Server

• Added support for security policies, sequences, check constraints, and system versioned temporal tables.

Improved cross-tab support for automatic logouts when you configure an inactivity period.

The Update option in the actions menu now takes you directly to the System tab on the Admin Panel instead of the Users tab.

Corrected the password length requirement to be 12 characters or greater instead of greater than 12 characters.

Improved the estimated row progress for scaled tables.

Eliminated a race condition that occurred when applying constraints.

Databricks:

• Can now run concurrent jobs that use different versions of Databricks.
• Tonic now supports ORC and Hive tables in Databricks.

SQL Server:

• The Categorical generator can now support more than 2 billion rows in a category.

When users create a new password, Tonic now displays a panel with the password requirements, and indicates whether the password meets those requirements.

Improved the parallelization and concurrency for processing foreign key constraints.

Databricks and Spark EMR

• Improved the performance of the Noise generator.

MongoDB

• Improved display of longer key values in the Key column. Widened the column and added truncation.
• Optimized queries against the Tonic database.

Features

For PostgreSQL databases, Tonic now supports name and char data types.

For Tonic single sign-on, Tonic now supports Azure Active Directory.

From the administration screen, administrators for customers that run Tonic on Kubernetes and Docker can now download logs from all containers that run Tonic.

For Spark-powered integrations, Tonic now supports the Address generator as a sub-generator.

Bugs

Improved performance for:

• Random Hash generator on Spark and Databricks
• UUID Generator on Spark
• Subsetting

Improved the browsing experience on low resolution displays.

Features

Skip batch instead of failing generations on Postgres in some cases when values fail to be inserted

Improvement to Synthesis Reports for AI Synthesizer generated data

Changing Lambda deployment from ECR to ZIP for integrations using Lambda functions. Images no longer need to be manually deployed to ECR.

Bugs

Fix for One-Click Tonic update when the PostgreSQL application database is deployed in a Docker container

Features

Support for arrays with Struct Mask generator on Amazon EMR/Spark and Databricks

Bugs

Fix issue causing privacy scans on Dremio to fail

Minor UI fixes to workspace page

Fix for data generation issue on Db2 iSeries when destination is empty

More robust handing of Databricks host URLs

Features

Adds data types to all keys in Mongo Collection view and tooltip popovers to key names

Updates to data generation UI

Bugs

Fix shaky dialog when table mode resizes

Features

Add data type advanced search for Oracle, Amazon Redshift, and BigQuery

Performance improvements for constraint restoration

Improved performance of Random Integer generator on Spark

Bugs

Fix issue on SQL Server preserve destination mode when table names contain a ".".

PostgreSQL: Fix issue copying arrays with trailing slashes

Allow the console log level to be configured on the Web Server and Worker

User passwords can now be reset by an Admin in the Admin Panel

Features

Added compatibility with SQL Server 2012

Spark SDK: Support for validation of workspace when processing a dataframe

Bugs

Minor fixes for MongoDB integration

Features

For Amazon S3/EMR data source, support separately testing each component of the connection (Glue catalog, EMR cluster, and S3 bucket)

The Additive Noise Generator is renamed to Noise Generator with two noise options - Additive (existing) and Multiplicative (new)

Improve Javadocs for Spark SDK and other enhancements

Bugs

Fixes for the SQL beautifier on post-job scripts, including scenarios that may have crashed the app on Safari

Skip processing Temporal Types when not supported (SQL Server 2012) to prevent job failures

Features

Support graceful cancellation of privacy metric computation for AI Synthesizer

Java SDK documentation available from the SDK Setup dialog

Bugs

Only show SSO and Enterprise licensing information to users when appropriate

Minor UI fixes for scrolling on the workspace view table

Minor UI fixes for the Edit Workspace page

Add hourly heartbeat/status log message for web server and worker

Group database/integrations on the create workspace screen by type

Features

Add support for Azure Datalake Storage Gen 2 as an output destination for Databricks

Don't include statistics for Oracle schema copy

MongoDB: Support uploading foreign key file, support deleting configured foreign keys

Bugs

Do not stop generation in SQL Server if schema pre-fetch fails

Improved deletion of old logs to prevent timeouts

Features

Adds support for Spark 3.1

Improvements to Dremio for data type searching, Struct support, and a new getVersion method on SDK

Bugs

Fix workspace name not updating in dropdown after rename

Don't allow non-editor/owner to open table mode selector

Features

Update subsetting tooltip help text

Don't check destination recovery mode on Azure SQL

Improve performance of table constraint application when subsetting by applying single and cross table constraints in parallel

Bugs

Fix logo navigation redirect when signed in as a user with no workspaces

Fix issue with setting a partition filter on Spark

Fix issue with collection picker on MongoDB showing removed collections

Fix excessive memory usage of Hostname generator

Features

Improves performance of Custom Categorical generator on Spark and Databricks

Add support for Azure Databricks

Display SQL Server schema generation progress as a percentage

Bugs

Do not fetch column data on configuration popover open to reduce queries against the source database

Reduce the number of queries to the source database for table/column details during generation

Features

Spark 2.4 and Databricks Library Improvements

Add ability to remove users in the Admin Panel

Bugs

Fixed issue with the displayed port when connecting to MongoDB via connection string

Fix for data table column resizing not rendering properly

Support random latitude and longitudes with HIPAA Address generator

Features

New design for the workspace dropdown selector

Add warning when previewing AI Synthesizer on text which does not appear to be categorical

Bugs

MongoDB: Collections no longer uncollapse when a generator is applied in hybrid view

Features

Add support for Spark SDK on Spark v2.4

Bugs

Fix UI crash when selecting incremental as Table Mode

Fix AI Synthesizer when no categorical columns present

Ensure AI Synthesizer model is not trained if only one column is selected

Render HTML Mask in the generator model panel on Table View

Fix issue with not being able to set incremental table mode column if you refresh after setting it to incremental

Added a pre-job check to guard against cycles in applied generators

Better support for environments where websockets are not supported

• Add environment flag to disable websockets on API
• Gracefully fail if websockets can't connect

Fixed issue preventing connecting to PostgreSQL databases using an SSH tunnel

Fixed nullability error in US Phone generator

Fix issue where collection loads were attempted for non-MongoDB databases in some instances

Write pg_dump results directly to file instead of reading them via stdout to prevent process locks

Fixed isssue where copying a workspace on the Workspaces view doesn't update the table of workspaces

Features

Support applying generator values for PK fields in MongoDB to their FK fields

Bugs

Reduce frequency of Upstream table filtering logs

Remove double scroll bar on foreign key page when viewing foreign keys

Improved database introspection queries on IBM Db2 iSeries

Consistently clear local storage on authorization failures

Aggregate dropped row exceptions for more efficient logging

Features

Improved performance for schema scans on MongoDB

Bugs

Minor fixes to workspace view

Fix for concurrent Lambda function update issues

Fix brief non-ideal state while loading collections on MongoDB

Features

Improve usability of processing with Java SDK

Bugs

Close connection to application database when not being used

Removed excess querying of database version

Allow skipping of privacy scan on MongoDB using TONIC_TABLE_SKIP_REGEX environment variable

Features

Improves performance of the SSN generator on Amazon EMR Spark / Databricks

Bugs

Keep the password field empty when copying from source database settings to destination database

Improved exception handling when masking and preserving destination tables

More quickly alert user when a cycle has been found in their relationships

Handle PostgreSQL interval data types with a null boundary

Disable Add Foreign Keys tab for non-workspace owner users

Features

Improves performance of Name generator on Spark / Databricks

Change Dremio schema input to Tag Input and add an informational popover

Bugs

Do not null FKs between reference tables

Allow foreign keys to be added by editors

Features

The Workspaces view now supports bulk actions. Share, transfer, delete, and leave multiple workspaces by checking the boxes at the right end of each row, and choosing the desired action from the Actions menu.

Bugs

Fixed an issue where Safari would fail to load the application

Fixes to constraint application parallelism

Breaking Changes

The response for the workspace deletion endpoint (/api/workspace/{workspaceId}) has been modified slightly

Features

Enable Timestamp generator to work without statistics for Spark object masks

When a MongoDB foreign key field has a primary key with a generator applied, replace the example data with a link to the collection with the primary key

Bugs

Fix for MySQL generation hanging with empty password

Features

Improves performance of Name generator on Amazon EMR and Databricks

Bugs

Fix display bug in AI Synthesizer configuration panel

Fix application of constraints in Amazon Redshift

Features

Add the version of the worker that ran a job to the job details

On the MongoDB Collection View for foreign/primary keys: Added a hover tooltip for key icons, replaced the generator with a label

Refresh the last collection visited when a new foreign key file is uploaded and optimize single_doc foreign key

Bugs

Disable date truncation generator as a sub-generator for Spark

Fixes Notification service consuming all disk space when crashing

Show the appropriate Conditional sub-generator label instead of Passthrough

Fixed in memory table query for SQL Server

Features

Teardown and database creation performance improved for SQL Server

Skip preserve destination tables (just like truncated) for privacy scan

Bugs

Fix not allowing generators allowed on arrays that were scanned prior to v439

Features

Adds support for events (dependent rows) to AI Synthesizer (formerly known as Smart Linking)

Allow wildcard (%) in schema name for Dremio

Bugs

Support Date Truncation and Timestamp Shift on Snowflake TIMESTAMP_TZ and Amazon Redshift TIMESTAMPTZ columns

Disable the generator dropdown in the UI for columns with both primary and foreign keys

MongoDB - Fix errors when linking Categorical generators

Bug fixes for synthesis report

Fix data type mappings for Dremio Integer and Varchar

Features

Move single connections on connection pools behind a feature flag

Do not drop indexes on auto increment columns in MySQL to improve performance

Add type filter to JSON Mask generator

Remove unique constraint on workspace names and change default workspace name to "Untitled Workspace"

Display key icons for user uploaded foreign/primary key fields in MongoDB single document view

Bugs

Fix issue where 0 results in Workspaces table sometimes let you navigate to a negative page number

Fix issue where workspace permissions didn't update in the UI until a refresh

Fixes authentication error on PostgreSQL when username has special characters

Fixed reopening open connections when TONIC_ENABLE_SINGLE_CONNECTION is false

Remove owner section from exported workspace body

Improve readability of workspace tags in small-width window sizes

Features

Display whether webhooks were used in job details

Terminology changes

• Masked is changing to De-identify
• Synthesized is changing to Scale
• Truncated is changing to Truncate
• Generator name change: Smart Linking is changing to AI Synthesizer

Bugs

Fixes to SQL script beautifier

Features

Apply foreign key constraints serially on a different thread to avoid deadlocks and improve performance

Bugs

Clear bulk column search query on filters reset

Memory improvements to MongoDB schema serialization

Better handling of schema name in Java SDK

Features

Allow changes to order of JSON Mask sub-generators

Running a job is now disabled when the license expiration date has passed

Separate Workspace Sharing and Workspace Role-based Workspace access into different license features

Improve SQL Server performance by prefetching all tables & views

Bugs

When using Java SDK only show Java supported generators

Features

Improves performance of Email generator on EMR and Databricks

Add support from Dremio with Spark

Bugs

Allowing Kubernetes ImagePullBackOff for up to 5 minutes before throwing error when updating through UI

Fixes inability to run a data generator on Spark with default database as the source

Features

Display estimated time remaining on row-based tasks in job details page

Display key icons for user uploaded foreign/primary key fields in MongoDB hybrid documents

Bugs

Copy Generator in MongoDB: Fix bug where if generator was copying a field that wasn't in the document it would throw exception

Add exception handling for SQL Server datetime columns to drop invalid records, better SQL Server XML type exception handling

Create source pool just before processing tables to avoid any timeouts in SQL Server

Display row based progress on jobs whenever possible

Statistics jobs for the Java Spark SDK now appear in Jobs table/Job Details page

Skip pre-job health check for PyML service when not needed