Tonic Structural release information

New global permission to view organization users - A new global permission, View organization users, determines whether a user is able to see the lists of users and groups in the organization. This permission is required in order to use the Tonic application to grant access to and transfer ownership of a workspace, and to grant access to global permission sets. It is not required when you use the Tonic API to perform these tasks. The permission is granted to the built-in Admin, Admin (Environment), and General User permission sets. When you upgrade, Tonic automatically grants this permission to your custom global permission sets.

Other updates

On the workspace details view, added a new upsert processing option, Warn on Mismatched Constraints. When this is enabled, Tonic treats mismatched foreign key and unique constraints between the source and destination databases as warnings instead of errors, so that the upsert job does not fail.

Tonic now accepts all AWS RDS certificate authorities. Previously, we only accepted rds-ca-2019. The accepted certificates include:

• rds-ca-rsa2048-g1
• rds-ca-rsa4096-g1
• rds-ca-ecc384-g1

When job log recording (used to download job logs from the Tonic application) fails, it no longer creates a recording retry loop.

File connector

• Additional fixes to skip and log invalid rows instead of failing the data generation.
• Fixed an issue where when you added a file to an existing file group, and any column name contained a leading or trailing space, Tonic incorrectly displayed a schema mismatch error.
• You can now add .gzip files to a file group in a file connector workspace. The original file that was compressed must have the same format and structure as the other files in the file group. .gzip files are only supported for workspaces that use files from Amazon S3 or Google Cloud Storage. They are not supported in workspaces that use local files.

PostgreSQL

• During upsert, improved performance when de-conflicting unique constraints.

Create virtual foreign keys from Subsetting view - On Subsetting view, from a table details panel, you can now add a virtual foreign key to that table. To add a virtual foreign key, you select the foreign key column from the current table, then select the primary key column from the other table.

Other updates

Fixed an issue with TLSv1 and TLSv1.1 support in Tonic.

Improved performance of downstream processing during subsetting.

Fixed an issue where subsetting failed when a composite foreign key included a Boolean value.

On Table View:

• Added a column warning when the assigned generator fails to generate preview data.
• Increased the default width of the preview data column.

File connector

• Tonic now skips invalid CSV file rows and logs a warning instead of failing the entire file.
• Added a warning when the same source file is added to multiple file groups in the same workspace.
• Added JSON Mask and XML Mask as the recommended generators for JSON and XML files.

​​MongoDB

• You can now assign the Conditional and Null generators to binary fields.

MySQL

• Fixed an issue where Tonic did not clean up temporary file uploads after it wrote the data.

Oracle

• For a data generation job that ran SQL Loader (sqlldr), if sqlldr either failed or succeeded with errors, the job details include an option to download the sqlldr log files.

PostgreSQL

• Fixed an upsert issue where jobs failed with a unique constraint violation if the table contained a unique index but not a unique constraint.
• During upsert, improved performance when de-conflicting unique constraints.

Snowflake on Azure

• Fixed a bug that occurred when specifying the Azure Storage Account URL.

Removed the requirement that the authentication cookie goes over HTTPS (Secure Cookie). This fixed an issue where users could no longer log into Tonic over HTTP, but they could still log in over HTTPS.

Fixed an issue where users could not log out of Tonic from the email confirmation page.

Fixed an issue where upsert failed because of foreign key violations. Also improved upsert performance.

MongoDB

• When generator configurations are updated in single document view, Tonic now generates the preview data without re-fetching the data and refreshing the page.
• Fixed an issue that caused fields to disappear from hybrid view when the Null generator was applied.

In Table View, when a generator cannot be applied to a column in order to produce the preview data, the error message now includes the name of the column.

Expanded the table and collection dropdown lists to accommodate longer names.

The Privacy Report now marks a column as consistent when the generator is always consistent.

Fixed a migration issue with file connector files that were added before V920.

Fixed an issue where data generation could not run because of the permissions hierarchy.

Fixed a security issue related to JWT authentication.

Fix an issue where webhooks sometimes did not start when a job was canceled.

Improved error message when Tonic cannot display a date value.

PostgreSQL

• For the Tonic Data Pipeline V2 processing, Tonic now stops job execution after the initial error.
• Fixed an issue where check constraints failed to be applied in the destination database.
• Fixed an issue where views that depend on both a table and a view at the same time were not created in the destination database.
• Tonic no longer uses the TONIC_PAGE_PARALLELISM and TONIC_PARALLEL_READ_RANGES_TABLES environment variables for parallel processing.

SQL Server

• Fixed a caching issue that occurred when connecting to SQL Server.
• Improved error messaging when a view cannot be created.
• Improved the readability of SMO error messages.

You can now export individual topics from the Tonic documentation to PDF files. To export a topic to PDF, click the actions menu next to the topic title, then click Export as PDF.

Fixed an issue with removing unique constraint conflicts in upsert where rows that didn’t have a conflict were excluded from the upsert process.

File connector

• For Amazon S3 and Google Cloud Storage, the permission to list all buckets is no longer required. However, if that permission is not present, users must manually type in the bucket name where the file is located.
• When you copy a file connector workspace, Tonic now copies the file groups to the new workspace.

Upsert data generation (beta feature)

Previously, the data generation process always replaced the entire destination database. The new upsert data generation option (currently in beta) allows you to add new records and update existing records without touching any of the other records in the destination database. For example, you might have a regular set of records that you use for testing that you want to maintain.

Upsert requires a connection to an intermediate database. When you run data generation with upsert, the initial data generation writes the transformed data to the intermediate database. It replaces the intermediate database, similar to regular data generation. After the generation to the intermediate database, the upsert process identifies the records to add to or update in the destination database. It ignores other records in the destination database.

Upsert requires a Professional or Enterprise license, and is only supported for the following data connectors:

• MySQL
• Oracle
• PostgreSQL
• SQL Server

New AI-enhanced documentation search option

The Tonic documentation now provides access to Lens, an AI-based search option. Instead of searching for specific words, you can ask questions such as "How do I create a workspace?". Lens searches the documentation for the answer. It generates a response that includes links to the topics that contain the information it used.

To use the Lens search, click the search field. At the top right of the search panel, click Lens. Then ask your question.

Other updates

The Custom Categorical generator now supports consistency with other columns. Previously, the generator only supported self-consistency.

Tonic now prevents you from starting data generation for a workspace that does not have a destination database specified.

Fixed a display issue where the generator preset details panel briefly showed the occurrences for the previously selected generator preset.

Tonic now suggests the Name generator for columns that Tonic detects as containing names, when the detection is based on the sampled data in the column. By default, the Name generator uses the First Last format.

A new configuration option allows webhooks to bypass SSL certificate validation and trust the server certificate.

File connector

• You cannot start data generation for a file connector workspace that has no source files specified.
• The Table View data preview for file groups that contain JSON or XML files no longer displays above the real data records an extra row that contains the value 0.
• Fixed an issue where generators such as the Categorical generator unexpectedly could not be used as sub-generators.
• Improved error handling when a file group is incorrectly configured as having a header row.
• Fixed an issue that caused data generation to fail for XML files because of missing metadata.
• Fixed an issue where Database View displayed duplicate columns.
• Table View no longer displays an extra data row.

MongoDB

• Fixed an issue that caused unscanned collections to not display on Collection View.
• The sensitivity scan no longer marks Null fields as sensitive.

PostgreSQL

• Improved performance when refreshing materialized views in PostgreSQL v11, v12 and v13.

On the generator configuration panel, changed the label of the Save As menu to Preset Options. The menu contains options related to configuring generator presets.

Free trial users now have access to the file connector.

Tonic now displays the error that occurs when an Algebraic generator configuration does not include any floating-point values.

For composite generators, the generator preset details panel now provides a clearer explanation that presets for composite generators must be configured from within a workspace.

Improved performance for the Address generator and the HIPAA Address generator.

Amazon Redshift

• Fixed an issue with clearing temporary tables.

File connector

• Fixed how Tonic handles EOF characters.

MongoDB

• Corrected the order of the available generator presets for a field.

Snowflake

• Fixed an issue where data generation returned the error The specified bucket does not exist.

SQL Server

• Database connections can now use the MultiSubnetFailover option.
• Improved error messaging when a database cannot be created because of permissions issues.

Removed support for TIM - The Tonic Installation Manager (TIM) command-line tool to install and configure Tonic is no longer available.

Free trial users can now use a public email address to create the free trial account. Users with public email addresses cannot invite other users or share workspaces. Public accounts are only allowed for free trials.

Users on a Professional instance can now share the Manager workspace permission set with users and groups.

Improved error handling and validation messages for the foreign key file upload process.

Counts of generator preset occurrences no longer include occurrences in deleted workspaces.

On the bulk update panel in Database View, the consistency and differential privacy options now display correctly.

Fixed an issue where you could not select Passthrough as a sub-generator in a composite generator.

Fixed an issue where custom presets could not be deleted.

Fixed a display issue where long post-job action names overflowed into the next column.

Fixed an issue where you could not assign Random Timestamp as a sub-generator for the Conditional generator.

Fixed an issue where the generator configuration panel displayed the generator preset options when the user did not have the Manage generator presets global permission.

Fixed an issue where when a constraint failed to be applied, data generation failed.

Improved display when users who do not have the Manage generator presets permission try to display the Occurrences tab on the preset details panel.

Improved how we handle unavailable options for workspace actions in Workspaces view and in the Tonic navigation options.

For the Conditional generator, Tonic now correctly compares MySQL date values.

Databricks

• Tonic cluster initialization scripts are now uploaded as workspace files instead of DBFS files. The new, optional Workspace Path setting for Databricks workspaces controls the parent directory where Tonic uploads initialization scripts. The default value is /Shared.

File connector

• The file connector can now support .txt files that contain CSV, XML, or JSON content.
• Fixed an issue where Tonic incorrectly identified how a file connector file was encoded.
• Improved error messages when uploading files for the file connector.
• Fixed an issue when configuring a file group from Amazon S3 where users saw the error "Failed to fetch files from S3. The continuation token provided is incorrect." but could still see the list of files in the S3 bucket.
• Tonic now correctly updates the file configuration for file groups. Previously, users could not add files that did not match the default configuration.
• Tonic now displays an error when it is unable to read files from Amazon S3.
• The file explorer for Amazon S3 can now list the files in folders that have names that contain special characters.
• Improved encoding detection and file parsing.
• Tonic now correctly handles EOF characters in .csv files.
• Tonic now preserves the encoding of .csv files.

MongoDB

• Fixed an issue where the protection status information at the top of Privacy Hub did not update correctly after a new sensitivity scan.

Custom generator presets

Earlier this year, for Enterprise instances, we introduced the concept of generator presets. A generator preset is a saved configuration of a generator. You can assign generator presets to columns.

The initial release only included built-in generator presets, which allowed you to set the default configuration for Tonic generators.

This update in v924 introduces custom generator presets, which allow you to set up multiple configurations of the same generator. You can create custom generator presets from Generator Presets view. From a generator configuration panel, you can also save the current configuration as a new custom generator preset.

Generator preset occurrences

From Generator Presets view, you can see how often each preset was used in a workspace configuration.

The Occurrences column of the generator presets list shows:

• The number of times the baseline configuration was used
• The number of times the baseline configuration was overridden, meaning that a user selected the generator preset and then made a change to the generator configuration

On the generator preset details panel, the Occurrences tab displays both the number of occurrences and the specific workspaces and columns where the generator preset was used. You cannot see workspace and column details for workspaces that you do not have access to.

Other updates

Tonic can now integrate with GitHub for SSO authentication.

To manage generator presets, users must now have the Manage generator presets global permission. Previously, you could also manage generator presets if you had the Manager or Editor workspace permission set for any workspace.

Fixed an issue where the table data in Table View was not updated correctly when switching the table mode to or from Scale mode.

Improved performance for the Regex Mask and Conditional generators.

MongoDB

• Fixed an issue where the subsetting Graph View did not display virtual foreign key relationships.
• You can now add collections to a subsetting rule before the sensitivity scan completes.

PostgreSQL

• Fixed an issue where certain database constraints were not handled correctly, which resulted in job warnings about the failure to add those constraints.

Permissions and permission sets

As of v922, Tonic now uses permissions and permission sets to manage access to Tonic features and functions.

A permission controls access to a single feature or function. A permission set is a saved collection of permissions.Tonic provides built-in global and workspace permission sets. You cannot change the configuration of built-in permission sets. Enterprise instances can create custom permission sets.

Global permission sets contain global permissions, which control access to actions outside the context of a specific workspace. The built-in Admin global permission set grants access to all global permissions. Users and groups configured in the TONIC_ADMINISTRATORS environment variable are granted the Admin (Environment) global permission set, which also grants access to all global permissions. These global permission sets replace the previous admin user concept.

The built-in General User global permission set is assigned to all Tonic users, and grants access to create workspaces. You can also designate a different global permission set to assign to all Tonic users.

Workspace permission sets are assigned in the context of a specific workspace. They provide access to workspace permissions, which are associated with workspace management functions. The built-in workspace permission sets (Manager, Editor, Viewer Auditor) mirror the previous workspace roles. Similar to the previous Owner role, the Manager workspace permission set is granted access to all workspace permissions. However, unlike the Owner role, the Manager workspace permission set can be assigned to any user or group. You use the workspace sharing function to assign workspace permission sets within a workspace.

Each workspace has a single owner. The user who creates the workspace is the initial owner. All owners are by default granted the Manager workspace permission set. You can also designate a different workspace permission set to assign to workspace owners. You use the transfer ownership function to select a different owner for a workspace.

On Tonic Settings view, the User Management tab is replaced by the Access Management tab. From the Access Management tab, you can:

• View and manage the list of Tonic users
• If you use SSO, view a list of SSO groups
• View, configure, and manage access to global permission sets
• View and configure workspace permission sets

API endpoint to track user access and permissions

A new API endpoint to track the following events related to user access and permissions:

• A user account is created.
• A user account is removed.
• A user logs in to Tonic.
• A permission is added to or removed from a permission set.
• A permission set is assigned to or removed from a user. This might be a global permission set, or a workspace permission set in the context of a specific workspace.
• A generator preset is updated.

The endpoint is:

GET /api/audit-events/search

Other updates

You can now assign the Business Name generator as a sub-generator for the Regex Mask generator.

For subsetting, Graph View and the table details panel now display information about cycle breaks, when the subsetting process needs to break a circular dependency.

Databricks

• Fixed an issue that prevented the use of partition filters in Databricks Unity Catalog workspaces.

File connector

• When a file connector workspace is deleted, Tonic now deletes files that were uploaded from a local file system.

Spark

• Fixed an issue with data generation for workspaces that use Hive.

The Admin Panel is renamed to Tonic Settings.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Fixed an issue where Tonic opened too many connections to the application database.

Fixed an issue with timestamps in the Tonic API specification.

Fixed a Tonic Cloud issue where using a different email domain to update a Tonic license caused issues with Tonic logging.

Enhanced the performance of the HIPAA Address generator.

The Data Pipeline V2 data generation process now respects the TONIC_PROCESS_PARALLELISM environment variable.

Improved performance for subsetting, especially for data that contains a large number of foreign key relationships.Made a small performance improvement to primary key generators.

File connector

• Improved error messages when uploading files for the file connector.
• The file connector now supports extended ASCII-encoded files.
• Fixed an issue where the file preview omitted the first row of the file.

MongoDB

• On the Jobs view, you can now filter for the Collection Schema Scan job type.
• Fixed an issue where sensitivity scans failed on large collections.

Oracle

• Tonic now provides a more meaningful error when it loads Table View for a table that the database account does not have access to query data from.

PostgreSQL

• Corrected the job history entries for subsetting jobs that run using the Data Pipeline V2 process.

Spark

• Fixed an issue that caused jobs to fail when an invalid Repartition or Coalesce value was specified.

NOTE: Releases v899 through v901 were removed.

Tonic Data Pipeline V2 for PostgreSQL ends beta

During the first half of 2023, Tonic has run a beta program for PostgreSQL for our new Data Pipeline V2. The beta program is now ending. Thank you to all of those who provided feedback.

Starting with version V905, Tonic.ai will progressively enable Data Pipeline V2 for all customers. To ensure a smooth transition for all our PostgreSQL customers, Tonic.ai controls the rollout remotely.

The remote rollout mechanism is controlled by an HTTPS request from your instance to https://feature-flag.tonic.ai. A JSON payload with a unique identifier for your deployment is sent, and the status of Data Pipeline V2 is returned. This request happens at the start of each data generation. If your Tonic server cannot reach https://feature-flag.tonic.ai, then the check is skipped.

What to expect for the enrollment:

• Before your instance is enrolled in Data Pipeline V2, Tonic Customer Success will contact you to confirm your enrollment.
• After your instance is enrolled, the Data Pipeline V2 toggle on the Confirm Generation panel is removed. All PostgreSQL jobs run using Data Pipeline V2.

For jobs that run on V2:

• The job type is Data Pipeline Generation instead of Data Generation.
• Jobs should run faster. Data Pipeline V2 has better resource handling, and can provide more parallel execution, especially for large tables and to apply constraints. Not all jobs will be faster, but for some jobs there should be a significant improvement.

We will continue to improve Data Pipeline V2 as we expand coverage to other data connectors.

Expanded Graph View for subsetting

The subsetting Graph View is expanded to use more of the available screen space. The Configure Subset panel, which includes the Options and Latest Results tabs, no longer displays on Graph View. It only displays on Table View.

Other updates

Fixed an issue where when a data generation job failed, tables that used Preserve Destination table mode were not restored.

The generated Tonic API documentation now includes the endpoints for managing file groups for file connector workspaces.

Fixed an issue that caused jobs for some workspaces to fail with the exception "Cannot modify workspace whose schema is not the latest version.".

Fixed an issue where the job details view displayed incorrect information.

Updated the /api/DataSource endpoint to not contain secure data such as the API key.

Updated our OpenAPI documents to ensure that all values of operationId are unique.

Improved error messages for failed data generation.

Databricks

• The Test Connection button on the workspace details view now works correctly.

MongoDB

• Fixed an issue where schema scans time out even when a timeout is not configured.
• Fixed an issue where MongoDB workspaces did not support MongoDB versions below v4.4.

MySQL

• Tonic data generation now supports generated columns in de-identified tables.

Oracle

• To improve the resiliency of Oracle commands during data generation, fixed the retry logic.

PostgreSQL

• Tonic no longer waits to process a job cancellation until after it finishes the constraint application that was in progress when the job was canceled.
• Fixed an issue where Tonic returned an "insufficient space" exception when writing numeric values to a destination database.

Snowflake

• Improved performance for Snowflake on Azure.

Spark

• Job tracking URLs now display correctly on the job details page.

SQL Server

• Fixed an issue where Tonic did not retrieve all of the compound keys from the source data.

NOTE: Releases v897 and v898 were removed.

On Subsetting view, Graph View now displays a loading animation as new data is loaded.

Improved performance for the UUID Key and Integer Key generators.

File connector

• Fixed an issue where files sometimes did not upload completely.

Google BigQuery

• Improved performance of destination database writes during data generation.

MySQL

• Improved performance for destination database writes during data generation.

Oracle

• Improved error handling when the tablespace in the source database is missing from the destination database.

File connector

The new file connector data connector allows you to use files from either Amazon S3, Google Cloud Storage, or a local file system as the source data. The file connector supports .csv, .json, and .xml files. Within a file connector workspace, you create file groups. Each file group contains files that have an identical format and structure. A file group is treated as a table for the purposes of table mode and generator configuration. The file connector is available with the Professional and Enterprise license plans.

Generic O​​IDC SSO Support

Tonic now supports authentication using a generic OpenID connection.

Tonic API versioning

We have introduced a versioning scheme for the Tonic API. API versions are released more or less quarterly, with the version identifier in the format vYYYY.MM.P (Year.Month.Patch). The current release candidate (v.RC) contains API updates in progress.

You should now specify an API version in your API requests. The System Status tab of the Admin Panel lists the latest available version. You can also select the version to use when you do not provide a version in the request. If you do not provide an API version in a request or select a default API version, then until January 31, 2024, Tonic automatically uses the latest version. After January 31, 2024, Tonic will return an error from the request.

Other updates

Fixed an issue where Tonic incorrectly returned the error No Destination DB has been configured for this Workspace for workspaces that used Preserve Destination.

For subsetting Graph View, updated the default zoom level to allow users to see more of the graph.

The Keycloak SSO provider now supports PKCE challenge.

Fixed an issue where deep links did not work for SAML SSO.

MongoDB

• For Mongo queries, Tonic now can now use disk as well as memory.

Oracle

• Tonic now refreshes materialized views even when SKIP_CREATE_DB is set to true.

PostgreSQL

• In the job progress steps, fixed an issue that caused the number of rows in a table to display as -1.
• When the Data Pipeline V2 processing is enabled, tables are now processed by size, with larger tables processed first.

SQL Server

• Fixed support for Kerberos authentication.

Spark

• Added support for the UUID Key generator to Livy and Databricks workspaces that use Spark 2.3.x and above.
• Added support for the UUID Key generator to the Tonic Java SDK.

The ASCII Key generator now includes an Exclude Lowercase Alphabet option to exclude lowercase letters from the destination values.

Fixed an issue that prevented free trial signups.

Data generation no longer fails when Tonic is unable to retrieve the destination database size.

Updated the FNR generator to prevent a possible leakage of PII.

Added a Date column to the usage report. The date column provides the date and time when the data generation job was completed.

Fixed an issue in the JSON Mask generator where it incorrectly changed the format of timestamps.

Subsetting is no longer prevented when a table that is not in the subset is assigned Preserve Destination mode.

Databricks

• On the workspace details view, you can now optionally specify the catalog where the source database is located. If you do not specify a catalog, then the default catalog is used.

MongoDB

• Fixed an issue where the collection statistics failed because the statistics object became too large.
• Updated to write collection records in batches.
• Tonic now continues to retrieve documents after a failure.

Oracle

• Removed the uniqueness check for individual columns that are part of a composite index.

PostgreSQL

• Fixed an issue where there was duplicated data from parent tables of inherited tables.
• Improved performance for query to retrieve tables and columns.

Snowflake

• Improved performance for data generation.
• Updated how Tonic uploads files to Amazon S3 to reduce memory usage.

SQL Server

• Fixed DNS resolution for Kerberos.
• Fixed an issue where Kerberos authentication failed with an error that the destination array was not long enough.

The new usage report summarizes the data processed for each table for data generation jobs. The report is a .csv file that you download from Tonic. To download the report, on the Admin Panel, click Download Usage Report.

When certain sensitive loggers are enabled, Tonic now disables log collection.

Fixed an issue on Database View where a column configuration panel would close unexpectedly.

For the TONIC_ADMINISTRATORS environment variable, you can now specify the names of SSO groups to grant administrator privileges to. Previously you could only specify user email addresses.

The Tonic SDK Javadoc now displays correctly.

Restored the ability to import a workspace configuration from Workspaces view.

MongoDB

• Added support for the DBRef datatype.
• Improved performance for collection management.

PostgreSQL

• For the beta Data Pipeline V2 data generation, adjusted the logging level for telemetry-related log messages to DEBUG.
• For the beta Data Pipeline v2 processing, improved parallel processing for constraints that cross tables.
• For the beta Data Pipeline V2 process, reduced the default maximum number of source and destination connections to 8. These are set as the values of the TONIC_JOBFLOW_MAX_SOURCE_CONNECTIONS and TONIC_JOBFLOW_MAX_DESTINATION_CONNECTIONS environment variables. We recommend that you set each value to the number of CPUs on the corresponding database.

Snowflake

• Fixed an issue where preserve destination tables were removed during data generation.

The new FNR generator transforms Norwegian national identity numbers. The FNR generator was added in V857. It included options to specify a range of birthdates and preserve the indicated gender. In V866, removed the date range configuration options. The destination values are now always within the same date range as the source values. The FNR generator also now can be used for columns that have uniqueness constraints. The final digits in the destination value are not a valid checksum.

For the beta Data Pipeline V2 processing, fixed an issue where jobs would hang if they were canceled before the job started.

Fixed an issue where the Foreign Keys view would freeze.

Fixed an issue where when you typed @ to add a user mention to a comment, suggestions for the user did not display.Upgraded to use .NET 7.

"Data science modeling" is changed to "data science mode".

When you configure the SSH tunnel settings for a workspace, Tonic now obscures the SSH passphrase.

MongoDB

• Added a configuration to prevent Tonic from retrieving other information about collections when retrieving a collection list. Addresses an issue where retrieving the collection list took a very long time. To disable the retrieval of other collection information, set the environment variable TONIC_MONGO_DISABLE_COLLECTION_INFORMATION_FETCHING to false.
• Fixed an issue with collection scanning that caused application pages to not load.
• Collection scans are now able to continue when the scan for an individual collection fails. The job logs include warnings for each failed collection.
• For collection scans, for each schema Tonic now limits the number of documents to scan and the length of time for the scan.

PostgreSQL

• Tonic now uses the estimated row count from PostgreSQL statistics to determine the parallelism for a table. Customers should ensure that they have up-to-date statistics for their source tables, especially for large ones.

Snowflake

• For source tables that are assigned Preserve Destination mode, Tonic no longer attempts to add existing constraints to the destination tables.
• Fixed a syntax error in the post-generation processing.
• Fixed an issue where data generation failed with the error "Unable to determine AWS Region".
• Fixed an issue where preserve destination tables were removed during data generation.

SQL Server

• During data generation, Tonic now warns users when a filegroup that exists in the source database does not exist in the destination database.

NOTE: Releases v849 through v854 are removed.

Removed caching of AWS credentials.

For the beta Data Pipeline V2 job processing (available for PostgreSQL only):

• When a job fails, Tonic no longer tries to fall back to the current job processing. In most cases, jobs fail for reasons that are not connected to the processing type. Falling back to the current job processing is not effective.
• Improved performance for subsetting.
• Adjusted the logging level for telemetry-related log messages to DEBUG.

On the Foreign Keys view, when you filter the keys, click the select all option, and then clear the filter, only the matching keys are selected. Previously, the select all option always selected all of the keys.

Fixed an issue where CSV files could not be uploaded to data science mode workspaces.

You can now configure parallelism for sensitivity scans. For relational databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_RDMBS, and the default is 4. For document-based databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_DOCUMENTDB, and the default is 1.

On the table configuration panel for subsetting, Tonic no longer displays a count of post-subset rows before a subset is generated.

Fixed an issue where subsetting data from one workspace appeared in a different workspace.

You can now use UUID columns in the conditions for the Conditional generator.

Fixed an issue where when you deleted a linked column from a column configuration panel, the other linked columns were deleted.

The Timestamp Shift generator can now be assigned to columns where the values use the date format MMddyyyy.

Fixed a regression that caused NPGSQL logging to occur even when it was disabled.

Google BigQuery

• Improved error handling when rows contain invalid data. Tonic now provides a method to look at the data that caused the error. Fixed the handling of rows to prevent errors on certain data.
• Fixed how we create views in the destination database.

MongoDB

• Fixed an issue where Tonic was unable to get the schema for the source data.

Oracle

• Fixed an issue where retries of Oracle commands from transient errors failed.

Snowflake

• Improved resource handling during data generation in order to enable parallel processing.
• Snowflake on Azure no longer requires CREATE SCHEMA permissions in order to support Preserve Destination mode for tables. Snowflake on AWS with Lambda processing continues to require CREATE SCHEMA permissions in order to support Preserve Destination mode.
• When Tonic is unable to create a view in the destination database, it now returns a warning instead of an error.
• Implemented a more accurate method to detect hexadecimal values.

SQL Server

• Tonic now displays the correct generators for columns that are part of a composite unique index.

The new Business Name generator produces realistic names of businesses or companies. The Business Name generator can be consistent with itself or with other columns. It improves on and is intended to replace the Company Name generator, which is now deprecated.

Fixed an issue on Table View where users could not use the delete icon to remove a generator assignment for a linked column.

Fixed an issue where the job details view for a subsetting job did not always show all of the steps as completed.Updated the version of pytorch, which is used for data science mode. This new version addresses some security vulnerabilities.

Fixed an issue where when the Tonic server was air gapped, the Admin Panel did not correctly display the current Tonic version.

Fixed an issue where jobs took longer than expected to complete.

Fixed an issue where the subsetting Graph View did not show how table participation in the subset changed since the most recent subsetting data generation.

Fixed an issue where after a single failure to write logs, the Download Job Logs feature stopped refreshing the logs. Tonic now continues to try to upload logs.

Google BigQuery

• Fixed an issue with the test connection function for the destination database.
• Materialized views and routines from the source database are now copied to the destination database.

MongoDB

• Improved performance for sensitivity scans.

MySQL

• Fixed an issue where subsetting failed because of a data type mismatch between a primary key and a foreign key.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB = true, foreign keys are now correctly enabled on the destination database.

PostgreSQL

• Fixed an issue where the source database permissions check provided a false error about insufficient privileges for sequences.
• For the beta Data Pipeline V2 data generation process, fixed an issue where the data generation process continued even after the job failed.

Snowflake

• Fixed an issue where the presence of comments caused data generation to fail.
• Improved performance for sensitivity scans.

SQL Server

• Added support for user-defined types.

Generator presets are now supported for Enterprise licenses on Tonic Cloud.

For Tonic data encryption, fixed an issue where the previous encryption key environment variable value was saved in the application database, which caused Tonic to use those values even after they were removed.

The Tonic diagnostic logs now include the Tonic worker ID.

Fixed an issue where the Tonic web server would not launch unless the Tonic application database used PostgreSQL v13 or later.

For data connectors other than MongoDB, the sensitivity scan is now parallelized.

Google BigQuery

• Data generation now works correctly when the region that hosts Google BigQuery for the destination database is different from the region for the source database.
• Fixed an issue where failed data generation jobs were incorrectly reported as successful.
• Tonic now handles the TIME data type correctly.

MySQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB=true, fixed an issue where the truncation of tables violated foreign key dependencies, which caused jobs to fail.
• Added an option to enable the TCPS protocol for Oracle database connections. Previously, only TCP was supported. If you enable TCPS, you must also provide a wallet file.
• Tonic now cleans up temporary destination database tables that were created during subsetting data generation.

PostgreSQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.
• During data generation, Tonic now warns users when an extension that the destination database needs is unavailable for installation.

Snowflake

• For both Snowflake on AWS and Snowflake on Azure, you can now configure workspaces to limit the schemas to include.

Fixed an issue where users could only select generators that supported uniqueness constraints for columns that were not unique.

Fixed an issue where admin users who did not have edit permissions on any workspaces could not edit presets from the Generator Presets view.

Improved data generation resiliency against transient failures.

Removed erroneous error messages.

To add AWS credentials to containers, you can now mount to ~/.aws/credentials.

Improved error messaging for Table View.

Fixed a display issue where the column configuration panel was too narrow and required horizontal scrolling.

Exporting or copying a workspace no longer requires the workspace to have a valid source database connection.

Reduced the amount of memory needed to run the Tonic web server.

MongoDB

• Better handling of errors that involve invalid UUIDs.

Oracle

• Updated the required permissions for destination database connections. If SELECT ANY DICTIONARY or SELECT_CATALOG_ROLE cannot be granted, then Tonic can use a selection of ALL_ views (not recommended).
• If TONIC_ORACLE_SKIP_CREATE_DB=true, then external tables are now excluded from the table list in Tonic. Tonic does not process those tables.

PostgreSQL

• Fixed an issue where the Data Pipeline V2 flow would hang.
• Fixed an issue where extensions such as pgcrypto were not transferred when data generation included schema filtering.
• Improved performance when handling constraints.

Snowflake on AWS

• As of V823, you can choose whether to use the Lambda process for data generation, which was previously the only option. By default, Snowflake on AWS uses a new, more resilient data generation process. You only need to use the Lambda data generation process for extremely large volumes of data (hundreds of gigabytes to terabytes).
  
  For existing workspaces, for versions before V826, the new default process is used. To use the Lambda data generation process, you must update your workspace configuration. As of V826, existing workspaces use the Lambda data generation process.
• For the temporary CSV files used to retrieve and write source and destination data, you can now specify to use an external stage instead of an S3 bucket. The option to use an external stage is not available when you use the Lambda data generation process.
• You can now specify different file storage locations for the temporary source and destination data files. In other words, you can have different S3 buckets or different external stages. Note that this option is not available when you use the Lambda data generation process.
• For the new data generation process, fixed an issue where data generation jobs would hang instead of failing.

Snowflake on Azure

• Before it runs a data generation, Tonic now verifies that there is a valid value for the Azure Blob Storage account key, which is set as the value of the environment variable TONIC_AZURE_BLOB_STORAGE_ACCOUNT_KEY.
• Fixed an issue where data generation jobs would hang instead of failing.

For Tonic data encryption, Tonic now only verifies the key for the enabled process. If you only enable decryption, then Tonic only verifies the value of TONIC_DATA_DECRYPTION_KEY. If you only enable encryption, then Tonic only verifies the value of TONIC_DATA_ENCRYPTION_KEY.

Upgraded our Docker images from Ubuntu 20 to Ubuntu 22.04.

Updated to ensure that the Tonic URL reflects the currently active workspace.

Recently started jobs no longer display a start time that occurred several years ago.

On the Data Encryption tab, the option to provide custom initialization vectors is now a toggle instead of radio buttons.

Resolved an issue where Tonic took an extremely long time to load.

Oracle

• Reduced the permissions required to test database connections.
• Changed the required permissions to better support when TONIC_ORACLE_DBLINK_ENABLED is false. For the source database user, you can either grant SELECT ANY DICTIONARY, grant SELECT_CATALOG_RULE, or (not recommended) grant access to the ALL_* views.

PostgreSQL

• Improved the error messaging when testing the connection to the destination database.

Snowflake

• You can now use a connection string to connect to the source and destination databases. Also added support for proxy connections.

NOTE: v805 and v806 are removed.

The new Snowflake on Azure data connector uses Azure Blob Storage to store interim uploaded and generated files.

A copied workspace now includes manual sensitivity designations. A manual sensitivity designation is when you change the sensitivity designation that was assigned by the sensitivity scan to either sensitive or not sensitive.

When the configured encryption or decryption key is not valid - for example, the key is not configured or uses the incorrect size - then Tonic does not allow you to configure Tonic data encryption.

Tonic Cloud now correctly enforces the supported data connectors.

Improved error messaging when subsetting data generation fails because the generator cannot be used with subsetting.When you change the type of Tonic data encryption (decryption, encryption, or both), Tonic no longer clears the decryption and encryption text fields.

MongoDB

• Fixed an issue where workspaces that contained views did not load.

Oracle

• Fixed an issue where the default Oracle NUMBER type was not compatible with the Integer Key Generator.
• Removed an invalid error that was returned when users tested data connections.

PostgreSQL

• For the beta Data Pipeline V2 generation process, improved the error logic to prevent jobs from hanging when errors occur.

SQL Server

• Improved the resilience of data generation to transient failures.

Fixed an issue where the column configuration panel on Privacy Hub incorrectly allowed users to change the configuration of foreign key columns.

The option to create a completely new workspace is no longer available from the workspace management view. You can only copy that workspace and create a child workspace. To create a completely new workspace, use the Create New Workspace option on the Workspaces view.

Fixed an issue where the generator for a column in a table that uses Scale mode could have a configuration option that is invalid for that mode.

Upgraded libraries to address security vulnerabilities.

Fixed an issue where when you applied the Random Timestamp generator to a column, a Bad format string error was returned.

Improved display of long table names in the table details panel on Subsetting view.

MongoDB

• Introduced a new configuration to collapse child fields into a single field based on a regular expression, to reduce the size of the schema. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX provides the regular expression to check the field keys against. TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX_THRESHOLD specifies the number of matching fields that causes the fields to be collapsed. A value of 0 indicates to not collapse the fields.
• On Collection View, fixed an issue where toggling the data preview changed the frequency values for documents within documents.

Added a confirmation step when updating the baseline configuration of a generator preset from a column configuration panel.

Fixed an issue where job failures sometimes caused the worker process to crash in a way that prevented new jobs from running.

To improve Tonic web application security, added X-Frame-Options/X-Content-Type-Option headers.

Databricks

• Corrected an issue on the workspace configuration page so that read-only items are no longer clickable.

PostgreSQL

• You can now use connection strings to connect to the source and destination databases.

SQL Server

• Fixed an issue where data generation failed when all URNs in the database have no dependencies.