Tonic Structural release information

New Db2 for LUW data connector - Tonic now has a data connector for IBM Db2 for Linux, Unix, and Windows (Db2 for LUW). Tonic supports Db2 for LUW version 11.5.

Other updates

When the AI Synthesizer is used in a workspace, Tonic now verifies before data generation that the AI Synthesizer does not use more than the maximum allowed categories.

Amazon EMR

• Improved data generation performance when using Spark 2.4.0-3.2.x.Databricks
  
  
  • Fixed an issue in Databricks 11.3+ where a generator that was consistent on another column received modified input values if a generator was applied to the consistency column. Generators that are consistent on another column now correctly receive unmodified input values.
  • Improved data generation performance when using Spark 2.4.0-3.2.x.
  Google BigQuery
  
  
  • You can now de-identify views, except for views that are written in Google Legacy SQL. On the workspace settings view, to enable de-identification of views, toggle De-identify Views to the on position. You can then assign table modes and generators to the views. In the table lists, views are identified by (view) after the view name.
    
    For each view, Tonic:
    
    
    • Writes the de-identified data to a table called <view name>_tonic_table.
    • Creates a view that has the same name and metadata as the view in the source data, but is populated from the destination table.
  • Fixed an issue where Passthrough tables that had table constraints failed to copy to the destination database.
  • Fixed an issue where cloned tables did not appear in the Tonic application, but caused job failures. Cloned tables now display within Tonic and can be managed in the same way as any other table.
  Oracle
  
  
  • For Oracle 12, improved the schema remapping across multiple object types and configurations.
  • Fixed an issue where Tonic failed to copy database link objects that were configured using passwords. Tonic no longer processes database links. If required, users must manually create database links in the destination schema.
  • Fixed an issue where tables that contained computed columns dropped rows during data generation.
  Snowflake
  
  
  • You can now assign the JSON Mask generator to Snowflake variant columns.

Fixed an issue with the Name generator where capitalization was not preserved if consistency was disabled.

For Table View, fixed an issue where the delete button to remove the generator assignment was sometimes hidden.

Oracle

• Tonic now supports Oracle Advanced Queues. Queue tables and queues are created in the destination database. Messages are not copied from the source database to the destination database. Queue subscribers are not currently supported. You must add them to the destination database manually.

PostgreSQL

• Tonic can now de-identify the ltree data type.

SQL Server

• Fixed an issue where indexes were not created on schema bound views.

Redesigned Database View

We redesigned Database View to improve the display and the filtering.

In the updated columns list, the Column column contains the schema, table, and column name, and the column data type. It provides access to the data preview option.

The Applied Generator column shows the applied generator. Applied Generator indicates when a column is unprotected, when the column is a primary or foreign key, and when the configuration overrides the parent workspace. If the table mode is not De-Identify, it shows the table mode. It provides access to the commenting option.

Filters other than the column name filter are moved under the Filters option. There are also new filters for the sensitivity type (the type of sensitive data that Tonic detected in the column) and whether the column has a recommended generator.

Privacy Report updates

In the Privacy Report, new column, Column Privacy Rank, indicates the privacy ranking for a column based on the assigned generator and generator configuration. The generator summary and generator reference include the possible privacy ranking values for each generator.

Added a new column, Tonic Detected Sensitivity, that indicates whether the Tonic sensitivity scan identified the column as sensitive. Renamed the Is Sensitive column to Current Sensitivity. Current Sensitivity indicates whether the column is currently marked as sensitive.

Also corrected an earlier issue with the order of the columns.

Other updates

Fixed an issue that caused all subset runs to record the percentage of rows in the subset as 100%. Subset runs that occur after updating to this version display the correct percentage.

The option to write output data to a container repository is out of beta.

Databricks

• For Databricks 13+, writing output to delta tables now uses the table’s liquid clustering configuration to cluster the insertion.
• Fixed an issue with loading tables into the Tonic application.

Google BigQuery

• Improved batch size calculations to reduce memory pressure during data generation jobs.
• Fixed an issue with applying the Timestamp Shift and Date Truncation generators on datetime or timestamp data when TONIC_GRPC_ENABLED was false.
• For the Timestamp Shift and Date Truncation generators, improved support for time values that include fractional seconds.
• Fixed an issue where the Character Substitution generator failed when assigned to a NUMERIC or BIGNUMERIC column.

Oracle

• Tonic now supports IDENTITY columns. Before this change, IDENTITYM columns caused errors during destination database creation.

For the Custom Categorical generator, you can now add a NULL value to the available custom category values. To indicate a NULL value, use the keyword {NULL}.

Made the following API updates to better accommodate users of the previous version of the API:

• jobs/{id}/workspace_snapshot now returns the WorkspaceDataModel object.
• Fixed an issue where workspace_snapshot could return empty replacements.
• Added a new endpoint, GET jobs/{id}/workspace_snapshot?api-version=v2023_07_00, that returns V17WorkspaceDataModel

Databricks

• When writing output to delta tables, the destination tables should now retain the TBLPROPERTIES from the source delta table, including 'delta.feature.allowColumnDefaults'.

Redesigned data model for generator assignments - The new version of the Tonic API includes a redesign of the data model for generator assignments. To use the previous version of the generator assignment data model, make sure that your API calls specify version 2023.07.0.

The generator assignment data model redesign includes the following changes:

• For all generators, moved the following fields to the metadata object in the link object:
  • presetId
  • generatorId
  • customValueProcessor
  • encryptionProcessor
• For the <value type>: Mask composite generators:
  • Moved pathExpression to the metadata object in the link object.
  • Removed the following fields from the link object:
    • subPresetId
    • subGeneratorId
    • customSubGeneratorValueProcessor
  • Added the following fields to the subGeneratorMetadata object under metadata:
    • presetId
    • generatorId
    • customValueProcessor
  Other updates
  
  Updated the sensitivity scan to better identify company and organization names and suggest the appropriate generator.
  
  Databricks
  
  
  • The Conditional Generator is now supported on version 11.3+.
  Google BigQuery
  
  
  • Fixed some cases where a copied View did not have its definition updated to reference resources within the destination dataset. This could result in failures when attempting to copy the View into the destination dataset.
  • Added support for Preserve Destination table mode.
  • Fixed an issue where sub-queries in table filter clauses did not work if TONIC_GRPC_ENABLED was set to true.
  MySQL
  
  
  • You can now enable diagnostic logging for a MySQL data generation job.
  PostgreSQL
  
  
  • Improved the destination database permissions check on Google Cloud SQL to handle additional superuser authentication setups.
  SQL Server
  
  
  • Data generation now preserves hidden properties of columns.

NOTE: v1074 was removed.

If your instance of Tonic is deployed on Docker, you can now use an external Kubernetes cluster to enable the option to write destination data to container artifacts.

You can now assign the Integer Key generator to a column with a decimal data type. The actual column values must still be integers.

Fixed an issue in Table View where an error displayed if you changed the selected table while the data was loading.

Databricks

• For cluster versions 11.3 and above, Tonic now displays accurate lists of available and suggested generators.

File connector

• Fixed an issue that prevented users from adding files to existing local file groups.

SQL Server

• Fixed an issue that caused errors when creating inline functions in the destination database.

NOTE: v1051 through v1053 were removed.

Enable administration functions in Tonic Cloud - For Tonic Cloud customers, the new Account Admin permission set provides access to Tonic administration functions for their organization. The Account Admin can reset passwords, delete users, copy and share all workspaces, and download the usage report. The Account Admin permission set is initially granted to the first user in the organization.

Databricks

• For the Test Cluster Connection option, added an error message when the specified cluster cannot describe all the tables in the specified catalog + database.
• Improved error messaging for the Test Cluster Connection option.

File connector

• When you use the API to add files to a file group, Tonic now validates that the file exists and does not duplicate a file that is already in the group.

MySQL

• Added a new environment setting, TONIC_MYSQL_MAX_CONCURRENT_INDEX_CREATION, to limit the number of concurrent indexes that are created. The default value is 0, which indicates that there is no limit.

SQL Server

• External tables are no longer displayed on the application or carried over to the destination database.

The Enable Diagnostic Logging global permission is now granted to the built-in Account Admin permission set.

Databricks

• CREATE CATALOG or CREATE SCHEMA permissions are no longer required if the destination catalog or schema already exists.
• For Databricks 11.3 and higher, added support for the Random Double generator.
• For Databricks 11.3 and higher, added support for the IP Address generator.

Diagnostic logging for data generation - By default, Tonic now redacts sensitive data in data generation log files.

When users start a data generation or upsert job, if they have the new global permission Enable diagnostic logging, they can choose to enable diagnostic logging, which does not redact the logs. The Enable diagnostic logging permission is also required to download the diagnostic logs. By default, the permission is only granted to the Admin and Admin (Environment) global permission sets.

In addition to the option for individual jobs, there are environment settings that enable diagnostic logging for specific data connectors.

Other updates

In the Release Candidate version of the API, the response model for the GET /api/workspace/minimal endpoint has been updated for more straightforward de-serialization.

Fixed an issue where a non-unique composite primary key column could only be assigned unique generators.

Users can now press Enter to finish copying a workspace or a generator preset, instead of having to click Copy.

File connector

• Added support for the Conditional generator for file groups that contain CSV files.

Google BigQuery

• Post-job scripts now support DDL and EXPORT.

Oracle

• Fixed an issue with the permissions check that prevented connecting to Amazon RDS for Oracle databases.

SQL Server

• In Data Definition Language (DDL) that applies to the destination database, Tonic now strips WITH INLINE clauses from definitions of user-defined functions (UDF). Inlining does not require these clauses. WITH_INLINE clauses in UDF definitions that do not meet the requirements for inlining can prevent the UDF from being restored properly in the destination database.
• Fixed an issue where the order of XML columns was changed in the destination database.

For the OpenID Connect (OIDC) SSO integration, Tonic now supports authentication by client secret that uses HTTP basic authentication (client_secret_basic). To provide the client secret, configure the TONIC_SSO_CLIENT_SECRET environment setting.

SQL Server

• A new environment setting, TONIC_SQL_SERVER_SKIP_CREATE_DB, indicates whether to skip schema creation for the destination database. If true, then Tonic does not create the schema. It uses the existing schema to populate the destination database. The default is false. You can configure this environment setting from the Environment Settings tab on Tonic Settings.

NOTE: These releases were removed.

During free trial signup, the data connector options now include an option to use local files for the source data. This creates a file connector workspace for local files, and displays the File Groups view to allow the free trial user to start to add file groups to the workspace.

Added an environment setting, Tonic Test Connection Timeout In Seconds (TONIC_TEST_CONNECTION_TIMEOUT_IN_SECONDS), that you can set from the Environment Settings tab on Tonic Settings. This setting configures the timeout for testing a database connection. Previously, connection test attempts timed out after 5 seconds. The new default is 15 seconds.

When you configure a workspace to write the output to container artifacts, you can now specify custom resources for the Kubernetes pod, including the ephemeral storage, memory, and CPU millicores.

Improved performance when marking a large number of columns as not sensitive.

Fixed an issue that caused Tonic workers that are deployed on Docker to crash unexpectedly.

For numeric columns that support arbitrary precision and scale, when the scale is 0 (for example, NUMERIC(N,0)), or when the underlying values are all integers, these columns are now supported as primary keys for the purpose of subsetting.

Amazon EMR

• You can now use the Timestamp Shift generator as a sub-generator within the Struct Mask generator.

Amazon EMR and Databricks

• When files are saved to non-job-specific file destinations, the new environment setting TONIC_WORKSPACE_DEFAULT_SAVE_MODE indicates the mode to use. If set to a value other than null (Ignore, Append, Overwrite), this setting takes precedence over TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE.

Google BigQuery

• Fixed an issue introduced in v982 where data generation failed with an HTTP 404 Not Found error for the destination table when the source and destination are in different BigQuery projects.

MongoDB

• Added an environment setting, TONIC_DOCUMENT_MAX_DEPTH, to configure the maximum depth of JSON document that can be handled. The default value, which is also the recommended minimum value, is 32.
• For bulk apply, fields with the same selection path but different data types no longer share a selection state.

SQL Server

• Fixed an issue to properly restore schema-bound views in dependency order.

When you select the option to write destination data to container artifacts, you can now use Google Artifact Registry (GAR) authorization using Google Cloud Platform (GCP) service account keys.

For the JSON Mask and XML Mask generators, fixed the data preview for JSON or XML field samples that are larger than 120MB by generating a smaller subset of the field.

The Name generator now supports consistency with other columns.

Added new API endpoints to retrieve and set table replacements. These new endpoints are compatible with workspaces for data connectors that do not have schemas, such as Spark-based databases and the file connector. The existing endpoints, which require you to provide a schema, eventually will be deprecated.

Amazon EMR

• You can now write data generation output either to a single, static location or to a job-specific folder. Previously, all data generation output was exported to a job-specific folder.

File connector

• You can now use the Continuous generator in file connector workspaces.

MySQL

• When writing destination data to a container repository, you can now provide the content of a my.cnf file that contains custom configuration.
• Fixed an issue where indexes that use expressions in their definition caused an error during data generation.

Oracle

• To fix an issue where turning off database links caused triggers to break, we now run the ALTER TRIGGER statements after the TRIGGERS are validated.

PostgreSQL

• For Azure PostgreSQL, fixed an issue with the destination database connection test.
• For Azure PostgreSQL, fixed the connection test check for schemas that are not admin owned.

SQL Server

• Data generation jobs that are canceled during the schema gathering phase now cancel more promptly.

Snowflake

• On Tonic Cloud, the Snowflake with Azure option is now enabled correctly.
• Free trial users can now use Snowflake on AWS and Snowflake on Azure.

Added an environment setting TONIC_DELETE_COLUMN_SCHEMA_ON_WORKSPACE_DELETE. If the setting is true, then when a workspace is deleted, Tonic also deletes the associated rows from the ColumnSchemas table in the Tonic application database.

The new environment setting TONIC_NOTIFICATION_SMTP_TRUST_CERTIFICATE indicates whether to allow the SMTP server certificate to be trusted.

Improved the performance of previewing data in Privacy Hub.

Fixed an issue where SSO groups were not removed when the value of TONIC_SSO_GROUP_FILTER_REGEX changed in a way that excluded previously imported groups. The removed groups are removed from any workspaces that they were granted access to.

For the Timestamp Shift Generator, added Month and Year as options for the date part to use to set the allowed range.

When writing data to container artifacts, Tonic now first shuts down the temporary database before it begins to write data to the container.

Amazon EMR

• Fixed an issue where, when the Athena workgroup used engine version 3, you could not preview data for tables that contained struct data types. Tonic now supports both Athena engine versions 2 and 3.
• Fixed an issue where Professional plan customers could not save workspaces.
• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.
• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

Databricks

• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.
• The FNR generator is now supported.
• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

MongoDB

• Fixed an issue where erroneous schema change items were reported when the frequency of a field changed.

Oracle

• Fixed an issue where when Oracle database links were not enabled (TONIC_ORACLE_DBLINK_ENABLED is false), privileges were not copied from the source to the destination.

Amazon EMR

• Fixed an issue where jobs failed if Tonic could not retrieve the tracking link for the EMR Step logs. To ensure maximum compatibility with all AWS regions, tracking links are now valid for 7 days instead of 14 days.
• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Databricks

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

MySQL

• On destination databases, Tonic no longer checks for REPLICATION CLIENT and REPLICATION SLAVE grants.

Spark SDK

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Spark with Livy

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

When an email notification contains a link to a comment, clicking the link now correctly navigates to and displays the comment.

Added a flag to mark log files that might contain sensitive information.

You can now assign the Timestamp Shift generator to primary key columns, including columns that are part of a composite primary key.

When writing destination data to a container repository, based on the size of the source database, Tonic attaches resource requests to the datapacker pod to verify that the cluster has sufficient resources for the data.

Tonic now supports writing destination data to container registries on Amazon Elastic Container Registry (Amazon ECR).

Fixed an issue where users could not assign access to permission sets when they did not have access to create and manage custom permission sets.

For foreign keys that have multiple primary keys, Tonic now prevents data generation if the assigned generation configuration for the primary key columns is inconsistent.

Standardized the display format of the heading area of the workspace management views.

File connector

• Fixed an issue with deleting multiple files from a file group.

MongoDB

• From Privacy Hub, you can now use the option to review and apply recommended generators to all of the detected sensitive fields.
• When there are schema changes that are not yet scanned, such as a new collection or new fields in a collection, they are now handled as Passthrough. The schema changes no longer cause data generation to fail.

MySQL

• You can now configure a MySQL workspace to write the destination data to a container repository.

PostgreSQL

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.
• Customers are now by default automatically enrolled in Data Pipeline V2 mode, and do not see the option to enable or disable it. The ability to enable or disable Data Pipeline V2 is granted to individual customers as needed.

Snowflake

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.

Apply recommended generators to multiple detected sensitive columns - On Privacy Hub, a new banner displays the number of detected (not manually designated) sensitive columns that are not protected. From the banner, you can display the list of columns, grouped by sensitivity type. For each sensitivity type, for the selected columns, you can:

There is also an option to apply the recommended generators to all of the selected columns across all of the sensitivity types.

Writing destination data to container registries (beta feature) - For data connectors that support it, you can now configure a workspace to write destination data to container artifacts on a container registry instead of to a database server. The Job History view provides access to the generated artifacts for each job.

Other updates

Fixed an issue where generated API documentation did not exclude endpoints and schemas from API versions other than the version being viewed.

Fixed an issue where the Protection Audit Trail displayed the incorrect override status of columns in child workspaces.

On the System Status tab of Tonic Settings, fixed an issue where the Data Sharing section incorrectly showed logging as unable to connect.

Databricks

• On the table mode selection panel, for workspaces that write to Databricks Delta tables, disabled the Error on Overwrite table configuration for tables. The setting is not used in this case.

File connector

• In the file group configuration, added options to skip lines from the beginning or end of CSV files.

MongoDB

• On Collection View, fixed an issue that prevented array fields from displaying in the Hybrid Document View of a collection.

Oracle

• For new workspaces, the Preserve source database file storage preferences setting is now off by default instead of on. Existing workspaces are not affected by this change.

Toggle between database server and container repository - On the workspace configuration view, if the data connector supports writing to a container repository, you can now switch between writing to a database server and writing to a container repository. Tonic saves the information you provide for each option.

Workspace override for statistics seed - From the workspace configuration view, you now can override the Tonic-wide statistics seed that is set as the value of the TONIC_STATISTICS_SEED environment setting. You can either provide a custom seed value for the workspace, or disable consistency across data generation runs for the workspace. Consistency also applies across workspaces that have the same custom seed value.

New telemetry URL - Tonic telemetry now routes through https://telemetry.tonic.ai/ instead of https://api2.amplitude.com/. The following IP addresses must be allowed:

• 75.2.74.76
• 99.83.246.105

The following IP addresses no longer need to be allowed: 52.43.241.47, 54.186.140.101, 54.203.75.164, 44.236.122.176, 34.215.78.194, 54.149.61.206, 54.191.147.220, 52.24.22.222, 52.37.168.36, 54.213.191.53, 54.68.108.104, 52.10.121.164, 52.27.184.186, 44.239.225.209, 54.148.216.233, 52.88.224.247Other updates

Fixed an issue in the Kubernetes client configuration that caused Tonic to reject the SSL certificate of a Kubernetes context.

Fixed an issue where, during subset configuration, an error was returned incorrectly if the user had permission to edit subsetting but not to view source data.

Improved the error message that displays when an uploaded virtual foreign key file is invalid.

Fixed an issue that prevented the Character Substitution generator from being used on primary and foreign key columns when subsetting.

When a workspace import encounters an unhandled exception, the error now displays correctly.

For the Address generator, the Country and Country Code options can now be linked. When linked, the country and country code are either “United States” or “US” to match the other linkable components, which are locations in the United States.

Fixed an issue where SSO login using Okta did not work when a custom authorization server is used.

Fixed an issue where, when using data science mode on Tonic Cloud, users could not download CSV files that contained synthetic data.

Data generation jobs that write to a container now include the datapacker logs, if the worker has permissions to read the pods and logs.

Fixed an issue with uploading container output to Harbor.

For string data type columns:

• Added yyyy/MM/dd as a valid format for the Timestamp Shift generator.
• Added yyyy/MM/dd and MMddyyyy as available output formats for the Random Timestamp generator.

Databricks

• For Databricks 11.3 and later, the Databricks data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark SDK

• The Spark SDK data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark with Livy

• For Spark 2.3.x and 2.4.2, the Spark with Livy data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

SQL Server

• Added a new worker environment setting, SQL_SERVER_SCRIPT_CROSS_DATABASE_REFERENCES. The default is true, which preserves the existing behavior. To prevent scripting of objects that are defined in other databases that the source database references, change the setting to false.
• Fixed issues where SQL Server connections did not honor the TONIC_SQL_COMMAND_TIMEOUT environment setting. The default is 0, which indicates an infinite timeout.

Configure Tonic environment settings from Tonic Settings - On the Tonic Settings view, a new Environment Settings tab allows you to configure a subset of Tonic environment settings (previously referred to as Tonic environment variables). To use the Environment Settings tab to configure settings, you must have the new Manage environment settings global permission. When you change the values from Tonic Settings, you do not need to restart Tonic.

Other updates

For the Regex Mask generator, fixed an issue where quotes in regular expressions were malformed.

Fixed an issue where the Test Webhook option failed when it shouldn’t.

Fixed issue where the Confirm Data Generation panel erroneously indicated that Tonic could not connect to the destination database when it actually was only unable to connect to the source database.

For the Address generator, removed an inappropriate value from the possible street names.

Improved error messages when Tonic containers fail to start because of missing or incorrect environment variable values.

Databricks

• Fixed source catalog workspace handling when the Databricks Unity source catalog contains a table with a key constraint.
• For Databricks 10.4 and earlier, generations to output Databricks tables now correctly create a new destination Databricks database if the specified one is not found.

File connector

• For file connector workspaces, the post-job scripts option is now hidden. The file connector does not support post-job scripts.
• For Amazon S3, fixed an issue where Tonic did not delete a temporary file that was created to test permissions.

Google BigQuery

• External tables are now supported with some restrictions. Destination tables must be native BigQuery tables and cannot be external tables, whether masked or passthrough. Performance might be affected because of Google’s implementation of external tables.

MongoDB

• Fixed an issue where an error was thrown when generators were applied.

Snowflake on AWS and Azure

• Added an option to use key pair authentication to connect to the source and destination databases.

Logging and telemetry connection status - On the System Status tab of Tonic Settings, a new Data Sharing section provides a summary of the logging and telemetry connectivity to the Tonic backend. The new section indicates:

• Whether sending logs and telemetry to Tonic.ai is enabled
• If they are enabled, whether Tonic is able to connect in order to send the logs and telemetry

Other updates

Tonic now checks for invalid virtual foreign keys for all data generation. Previously it only ran the check for subsetting data generation.

Fixed an issue where the Continuous generator failed when a column contained only NULL values.

Improved performance for the Continuous Generator.

Fixed an issue where the Cell Count in the Usage Report could throw an integer overflow error.

Fixed a subsetting issue where Tonic displayed the error “Error fetching Subset preview” when users navigated to a workspace.

For subsetting, updated the Graph View display to make it easier to see the connections between the tables.

Improved messaging when Tonic cannot reach the database when you test a database connection.

Databricks

• On the workspace settings view, Test Cluster Connection no longer requires that you re-enter your API key.
• On the workspace settings view, the default job cluster specification now recommends a Databricks 14.0 Spark version.
• Databricks is now enabled for Professional and Enterprise users on Tonic Cloud, as well as for free trial users.
• Fixed source catalog workspace handling when the source catalog contains a table with a key constraint.
• Added a warning to the workspace settings view to prevent specifying the same source and destination locations.

File connector

• Fixed the validation used to prevent duplicate files in Amazon S3 file groups.

PostgreSQL

• Improved handling when the database does not contain a public schema.
• Improved subsetting performance for the Data Pipeline V2 data generation process.
• Improved performance for the Data Pipeline V2 data generation process.

SQL Server

• Fixed how Tonic handles system time periods.
• Improved messaging when Tonic cannot access SQL objects.

The new TONIC_ENABLE_SECURE_COOKIES setting indicates whether to enable the "Secure" attribute on Tonic authorization and analytics cookies. The default value is false. Do not set this to true if you access Tonic over an HTTP connection. When TONIC_HTTPS_ONLY is set to true, the “Secure’” attribute is always enabled on Tonic authorization and analytics cookies, and the value of TONIC_ENABLE_SECURE_COOKIES is ignored.

Updated to prevent simultaneous updates to the same workspace configuration.

For the Constant generator, fixed an issue for JSON columns where setting the constant to an empty string caused data generation to fail without setting the job status to failed.

The upsert pre-job check that validates the constraints on the intermediate and destination databases no longer fails when a database has constraints with duplicate signatures.

Fixed an issue where an empty upstream filter WHERE clause caused subsetting to fail if the schema changed so that the table was no longer upstream.

To use the API to obtain data encryption settings, the API user must now have the required global permission.

When users log out of Tonic, we now automatically invalidate any JSON Web Tokens (JWTs) that are not expired.

The API endpoint /api/permission-sets now requires the ManageUserAccess (Manage user access to Tonic and to any workspace) permission. Added a new endpoint /api/permission-sets/public, which returns the subset of the data needed by users who do not have that permission.

Amazon EMR

• Fixed an issue where when data generation was run from the SDK, some generators, including the Categorical generator, would not work.some text
  • You can now configure a Databricks workspace to write destination data to Databricks Delta tables.
  some text
  • On the file group details for CSV files, added configuration options to quote spaces and trim whitespace.
  • Fixed an issue where Tonic was not able to display a preview of extremely large files.
  some text
  • On the Collection View, the preview icon is now hidden for types that cannot be previewed.
  some text
  • For Data Pipeline V2 data generation, fixed an issue where we did not correctly truncate destination tables.
  • Fixed an upsert issue for tables that have generated identity columns.
  some text
  • For a Snowflake on AWS workspace, you can now provide specific AWS credentials for the file storage locations (S3 buckets or external stages).
  • The Snowflake data connectors are now available to Tonic Cloud users with a Professional or Enterprise license. They are not available to free trial users.
DatabricksFile connectorMongoDBPostgreSQLSnowflake

Upsert data generation (beta feature)

Previously, the data generation process always replaced the entire destination database. The new upsert data generation option (currently in beta) allows you to add new records and update existing records without touching any of the other records in the destination database. For example, you might have a regular set of records that you use for testing that you want to maintain.

Upsert requires a connection to an intermediate database. When you run data generation with upsert, the initial data generation writes the transformed data to the intermediate database. It replaces the intermediate database, similar to regular data generation. After the generation to the intermediate database, the upsert process identifies the records to add to or update in the destination database. It ignores other records in the destination database.

Upsert requires a Professional or Enterprise license, and is only supported for the following data connectors:

• MySQL
• Oracle
• PostgreSQL
• SQL Server

New AI-enhanced documentation search option

The Tonic documentation now provides access to Lens, an AI-based search option. Instead of searching for specific words, you can ask questions such as "How do I create a workspace?". Lens searches the documentation for the answer. It generates a response that includes links to the topics that contain the information it used.

To use the Lens search, click the search field. At the top right of the search panel, click Lens. Then ask your question.

Other updates

The Custom Categorical generator now supports consistency with other columns. Previously, the generator only supported self-consistency.

Tonic now prevents you from starting data generation for a workspace that does not have a destination database specified.

Fixed a display issue where the generator preset details panel briefly showed the occurrences for the previously selected generator preset.

Tonic now suggests the Name generator for columns that Tonic detects as containing names, when the detection is based on the sampled data in the column. By default, the Name generator uses the First Last format.

A new configuration option allows webhooks to bypass SSL certificate validation and trust the server certificate.

File connector

• You cannot start data generation for a file connector workspace that has no source files specified.
• The Table View data preview for file groups that contain JSON or XML files no longer displays above the real data records an extra row that contains the value 0.
• Fixed an issue where generators such as the Categorical generator unexpectedly could not be used as sub-generators.
• Improved error handling when a file group is incorrectly configured as having a header row.
• Fixed an issue that caused data generation to fail for XML files because of missing metadata.
• Fixed an issue where Database View displayed duplicate columns.
• Table View no longer displays an extra data row.

MongoDB

• Fixed an issue that caused unscanned collections to not display on Collection View.
• The sensitivity scan no longer marks Null fields as sensitive.

PostgreSQL

• Improved performance when refreshing materialized views in PostgreSQL v11, v12 and v13.

NOTE: v980 through v982 were removed.

New monthly pay-as-you-go plan on Tonic Cloud - Tonic now offers a pay-as-you-go subscription plan for Tonic Cloud. Free trial users are offered the option to use a credit card to purchase a pay-as-you-go license. The monthly subscription grants a Professional level license. With the pay-as-you-go plan, you can configure generators for up to 20 tables across all of your workspaces. Tonic bills you separately for each additional table that you configure. The license renews automatically each month. On Tonic Settings, a new Billing tab displays the next renewal date.

Data migration option for upsert - For upsert, Enterprise users can now connect a workspace to their own data migration script or tool to ensure that schema changes are automatically reflected in the intermediate database.

Other updates

Timestamp Shift is now the suggested generator for birthdate fields. Previously, the suggested generator was Random Timestamp.

Fixed an issue where editing workspace settings could cause you to be logged out.

File connector

• On Tonic Cloud, you can now use Amazon S3 as a source of files. Previously, Tonic Cloud only supported Google Cloud Storage and uploaded local files.
• For the Categorical generator, linked columns are now in the correct order.

Google BigQuery

• Tonic now supports arrays of a supported type. The STRUCT and INTERVAL types are still not supported.

MongoDB

• Fixed issue where a UI refresh was required in order to show automatic de-identification of foreign keys from de-identified primary keys.

Oracle

• The Download SqlLdr Files workspace permission is now assigned to the built-in Manager and Editor workspace permission sets.
• Downloaded sqlldr files no longer include .bad files.

PostgreSQL

• Fixed an upsert issue for tables that have generated identity columns.

SQL Server

• The Constant generator can now handle bit values.

New global permission to view organization users - A new global permission, View organization users, determines whether a user is able to see the lists of users and groups in the organization. This permission is required in order to use the Tonic application to grant access to and transfer ownership of a workspace, and to grant access to global permission sets. It is not required when you use the Tonic API to perform these tasks. The permission is granted to the built-in Admin, Admin (Environment), and General User permission sets. When you upgrade, Tonic automatically grants this permission to your custom global permission sets.

Other updates

On the workspace details view, added a new upsert processing option, Warn on Mismatched Constraints. When this is enabled, Tonic treats mismatched foreign key and unique constraints between the source and destination databases as warnings instead of errors, so that the upsert job does not fail.

Tonic now accepts all AWS RDS certificate authorities. Previously, we only accepted rds-ca-2019. The accepted certificates include:

• rds-ca-rsa2048-g1
• rds-ca-rsa4096-g1
• rds-ca-ecc384-g1

When job log recording (used to download job logs from the Tonic application) fails, it no longer creates a recording retry loop.

File connector

• Additional fixes to skip and log invalid rows instead of failing the data generation.
• Fixed an issue where when you added a file to an existing file group, and any column name contained a leading or trailing space, Tonic incorrectly displayed a schema mismatch error.
• You can now add .gzip files to a file group in a file connector workspace. The original file that was compressed must have the same format and structure as the other files in the file group. .gzip files are only supported for workspaces that use files from Amazon S3 or Google Cloud Storage. They are not supported in workspaces that use local files.

PostgreSQL

• During upsert, improved performance when de-conflicting unique constraints.

Create virtual foreign keys from Subsetting view - On Subsetting view, from a table details panel, you can now add a virtual foreign key to that table. To add a virtual foreign key, you select the foreign key column from the current table, then select the primary key column from the other table.

Other updates

Fixed an issue with TLSv1 and TLSv1.1 support in Tonic.

Improved performance of downstream processing during subsetting.

Fixed an issue where subsetting failed when a composite foreign key included a Boolean value.

On Table View:

• Added a column warning when the assigned generator fails to generate preview data.
• Increased the default width of the preview data column.

File connector

• Tonic now skips invalid CSV file rows and logs a warning instead of failing the entire file.
• Added a warning when the same source file is added to multiple file groups in the same workspace.
• Added JSON Mask and XML Mask as the recommended generators for JSON and XML files.

​​MongoDB

• You can now assign the Conditional and Null generators to binary fields.

MySQL

• Fixed an issue where Tonic did not clean up temporary file uploads after it wrote the data.

Oracle

• For a data generation job that ran SQL Loader (sqlldr), if sqlldr either failed or succeeded with errors, the job details include an option to download the sqlldr log files.

PostgreSQL

• Fixed an upsert issue where jobs failed with a unique constraint violation if the table contained a unique index but not a unique constraint.
• During upsert, improved performance when de-conflicting unique constraints.

Snowflake on Azure

• Fixed a bug that occurred when specifying the Azure Storage Account URL.

In Table View, when a generator cannot be applied to a column in order to produce the preview data, the error message now includes the name of the column.

Expanded the table and collection dropdown lists to accommodate longer names.

The Privacy Report now marks a column as consistent when the generator is always consistent.

Fixed a migration issue with file connector files that were added before V920.

Fixed an issue where data generation could not run because of the permissions hierarchy.

Fixed a security issue related to JWT authentication.

Fix an issue where webhooks sometimes did not start when a job was canceled.

Improved error message when Tonic cannot display a date value.

PostgreSQL

• For the Tonic Data Pipeline V2 processing, Tonic now stops job execution after the initial error.
• Fixed an issue where check constraints failed to be applied in the destination database.
• Fixed an issue where views that depend on both a table and a view at the same time were not created in the destination database.
• Tonic no longer uses the TONIC_PAGE_PARALLELISM and TONIC_PARALLEL_READ_RANGES_TABLES environment variables for parallel processing.

SQL Server

• Fixed a caching issue that occurred when connecting to SQL Server.
• Improved error messaging when a view cannot be created.
• Improved the readability of SMO error messages.