Fixed an issue introduced in v1587 where in certain environments, jobs would not fail properly, and continued to be marked as running.
For HashiCorp Vault secrets managers, added support for LDAP authentication.
Amazon Redshift - Added support for subsetting for workspaces that use the Data Pipeline V2 data generation process.
Amazon Redshift - Added support to run data generation using the Data Pipeline V2 process, which does not require the AWS Lambda infrastructure. For Amazon Redshift workspaces, the Confirm Generation panel now provides an option to select whether to use the classic data generation or Data Pipeline V2.
Oracle: NUMBER(*,scale) columns now force the use of Sql*Loader instead of the faster BulkCopy.
When sharing a workspace or changing the shared permissions, users cannot grant or revoke access to revoke permissions that they do not have for the workspace. For example, if your assigned permission set for a workspace does not include the Run data generation permission, then you cannot assign or remove a workspace permission set that includes the Run data generation permission.
PostgreSQL - Fixed an error that could occur when tearing down a destination database that contained a very large number of Large Objects.
MongoDB - Fixed an issue that prevented data generation or schema diff execution when the workspace contained a schema change in a foreign key column that had an applied generator.
Subsetting no longer revisits lookup tables during the subset traversal. The lookup table is downloaded immediately as a direct target, and no further passes are made.
Snowflake - Fixed handling of schemas that use various constraint options in their table definitions.
Fixed an issue where OutOfMemoryException could be thrown when JSON data contained arrays larger than 2GB.
MongoDB and Document View: You can use path expressions to protect unscanned fields. If a path expression generator applies to an unscanned field, data generation does not fail and the generator is used to mask the field value.
To use this feature for MongoDB, set the environment setting TONIC_MONGO_DISABLE_FIELD_PROJECTION to true. Otherwise, unscanned fields are skipped automatically during data generation. Note that when this setting is true, any unscanned field that is not protected by a path expression generator causes data generation to fail.
Adjusted the Timestamp Shift generator output when the generator is configured to use self-consistency:
Date-only values that represent the same logical date shift consistently
For all types, the preview values in Table View are now more consistently aligned with the generated values
This change might affect the consistency of values for existing generator assignments.
MySQL - Fixed an issue where preserve destination temp objects in schemas that started with the prefix tonic_temp*were not always cleaned up immediately after parts of a job execution failed. This fix DOES NOT clean up tonic_temp* schemas and objects that were left behind by previous builds. Those must be cleaned up manually in the destination database.
Redesigned secret selection for workspace settings - On the workspace settings view, when a password field supports using a secret from a secrets manager, a new Use Secret link displays at the top right of the field. To select the secret to use, click the link. On the secret selection panel, instead of selecting the secret from a dropdown list, you must provide either the name or ARN of the secret. If the secret is part of a structured key-value pair, then you also provide the name of the property that contains the secret. When a password field uses a secret, the field is disabled and the Use Secret link changes to Using Secret. To display a panel from which you can update the selection or remove the secret, click Using Secret.
Databricks - Enabled TCP keepalive by default for Databricks connections.
On Structural Cloud, the Custom Categorical generator now includes an option to use an LLM to generate the available values. For example, for an Occupation column, you might prompt Structural to generate “25 common professions”. This option is not available on self-hosted instances.
You can now configure the retention limits for generation job artifacts. To set the limits, use the environment settings MAXIMUM_GENERATION_ARTIFACT_STORAGE_LIMIT_IN_BYTES and MAXIMUM_GENERATION_DEBUG_PACKAGE_RETENTION_IN_DAYS.
On Table View, when the width of the table is more than 1.5 times the visible display, a new Jump to column option allows you to select a column to scroll into view.
Schema retrieval jobs now display the reason they ran. Updated the description of the schema caching feature on Workspace Settings to clarify how often these refreshes can occur.
For workspaces that write output to container artifacts, the artifacts are now listed on the workspace Outputs tab.
On Jobs view, the jobs list now includes tabs that filter the jobs based on the job category. The All Jobs tab lists all of the jobs.
From the Confirm Generation panel, the option to upload job logs to Tonic.ai is now available even if diagnostic logging is not enabled.
File connector - As a safety precaution, data generation for Amazon S3 or GCS workspaces do not process any input files for which the output path points to itself, which could happen in rare output path configurations. Previously, any files in the output location were overwritten.
On the workspace settings and Schema Changes view, “sensitive schema changes” are now referred to as “schema changes that could expose data”. This is to avoid confusion with the general sensitivity designation for columns.
File connector - On the Workspace Settings page, added support for testing credentials and configurations.
Files connector - For Amazon S3 or GCS workspaces, fixed a regression that prevented users without permission to list all buckets from being able to load the workspace properly.
Fixed an issue where a new deployment could cause the application to fail during navigation.
Added a Format Type configuration option when the Timestamp Shift generator is assigned to numeric columns.
For jobs that generate diagnostic logs, the job details page now includes an option to send the logs directly to Tonic.ai.
Snowflake - Fixed an issue where subsetting data generation failed when primary or foreign key columns had names that conflicted with particular SQL keywords.
File connector - Fixed an issue that caused invalid modifications to file groups.
When a workspace is configured to not block data generation when there are schema changes, then before a data generation can run, there must be a completed schema diff.
MySQL - You can now exclude specific views from data generation. To identify the views to exclude, use the environment setting TONIC_MYSQL_EXCLUDED_TABLES_FROM_DUMP. The value is a comma-separated list of tables and views to exclude.
In Document View, fixed an issue where fields in sparse documents could be dropped from Hybrid view when the Preview option was toggled on.
For webhooks, fixed an issue where using typeahead suggestions on a custom property that contained multiple variables might result in unexpected behavior.
For webhooks, fixed an issue where in workspaces that used upsert, you could not select Upsert jobs as a trigger job type.
Salesforce - Added troubleshooting text for when the destination user is missing the "Update Records with Inactive Owners" permission.
The Conditional generator now supports non-integer numbers in conditions.
Salesforce - Added optional user obfuscation.
MySQL: Structural now drops rows that contain invalid JSON. When diagnostic logging is enabled, Structural also logs the invalid JSON. Previously, the invalid JSON caused the data generation to fail.
For post-job scripts, fixed an issue where if the script was originally created with warnings disabled, you could not subsequently update the post-job-script to enable warnings.
Fixed a race condition that could cause a job to fail when it cleaned up the postgres binary importer for a given table in high-performance environments.
In Document View, fixed a data generation preview issue with overlapping path generators.
MySQL: Structural now drops rows that contain invalid JSON. When diagnostic logging is enabled, Structural also logs the invalid JSON. Previously, the invalid JSON caused the data generation to fail.
Oracle: Removed support for Oracle 12.1. The minimum supported version is 12.2.
Oracle: Resolved a potential race condition that could cause errors during the creation of GLOBAL TEMPORARY TABLEs in the destination database.
You can now use the Timestamp Shift generator for integer and continuous numeric columns.
PostgreSQL - Fixed an issue during data generation where partitioned tables did not use the appropriate indexes for parallelized reading. This is a follow-up to a fix in v1496.
PostgreSQL - Fixed a data generation issue where a small number of rows might be dropped when a table used parallelized range query reading that used an index that contained timestamp columns.
Fixed an issue where applying generator recommendations to a large number of tables or columns could cause the page to become unresponsive or crash.
