Tonic Structural release information

Bug fixes and other internal updates.

Oracle - When TONIC_ORACLE_SKIP_CREATE_DB is true, tables in the destination database are preserved. If a table filter is provided in TONIC_TABLE_WHITELIST, then destination tables that are not in the whitelist are preserved. Before this change, those tables were truncated.

Bug fixes and other internal updates.

MongoDB - Added the environment setting TONIC_MONGO_MASK_FIELD_IDS. When set to true, Structural assigns the Mongo ObjectId Key generator to ObjectId field names. Consistency is enabled. The default value is false.

Bug fixes and other internal updates.

Improved error handling for PII detectors.

Google BigQuery - Fixed an issue where some BigQuery workspaces did not release rows from memory in a timely manner.

MySQL - The new environment setting TONIC_MYSQL_USE_NATIVE_DUMP_TOOL, when set true, prefers the mysql native tools over the existing default of mariadb.

MySQL - You can now configure the following environment settings to override the Structural default behavior when a connection opens and a session is established:

• TONIC_MYSQL_NETWORK_READ_TIMEOUT

• TONIC_MYSQL_NETWORK_WRITE_TIMEOUT

• TONIC_MYSQL_WAIT_TIMEOUT

• TONIC_MYSQL_LOCK_WAIT_TIMEOUT

• TONIC_MYSQL_INNODB_LOCK_WAIT_TIMEOUT

Azure SSO - Added support for authenticating application service principals using the EntraID client-credentials flow. Service principals can access the Structural API. For configuration requirements, refer to the Azure/EntraID SSO configuration information in the Structural User Guide.

SQL Server - Fixed the copy of stored procedures that reference full-text indexes.

Structural now supports diagnostic logging for upsert data generation.

Snowflake - Fixed an issue where, when using the connection string and key-pair authentication options, the test connection button was disabled unless a password was entered.

Updated the legal text that is displayed for new account creation on self-hosted Structural instances.

For custom sensitivity rules, column matching rules are now always case insensitive. Previously, the column matching rules were always case sensitive.

SQL Server - Added support for:

• Full-Text Search catalogs and indexes
• Change tracking for database and table scopes

‍

Bug fixes and other internal updates.

Improved the free trial onboarding flow. Added to the available data connectors. Fixed an issue with creating a Google BigQuery workspace. Fixed a duplicate display issue.

File connector - Fixed an issue that a caused authorization failures when using Assume Role to authorize access to Amazon S3 from Structural Cloud.

Fixed an issue where after an import from a JSON file, Subsetting view did not immediately reflect the state of the workspace.

Spark - Removed support for Livy on Hive.

PostgreSQL - Fixed an issue where Structural failed to process tables that contained a generated column that referenced a user-defined type.

Scheduling data generation - From the Jobs view (renamed from Job History) for a workspace, you can now configure the data generation to run automatically on a schedule. The schedule consists of one or more cron expressions, along with the time zone to use for the schedule. The Structural API includes new endpoints to manage the job schedule.

MongoDB - Fixed an issue that caused connection errors for connections to a Mongo Atlas cluster from Tonic Cloud.

Bug fixes and other internal updates.

File connector - You can now specify a separate IAM role for the output location.

Create sensitivity rule from Database View - When you select the Database View bulk edit option for columns that have the same data type, do not have an assigned generator, and do not have a recommended generator, you now have the option to create a custom sensitivity rule. You can then immediately run a new sensitivity scan to catch matching columns.

Fixed an issue with the JSON Mask generator configuration panel where the example data did not update correctly.

Structural now displays a warning when the pre-job checks determine that the source database is on a newer major version than the destination database.

Salesforce - Rewrote the connector algorithm to avoid using sentinels, and to improve subset creation.

Snowflake - Fixed a regression introduced in v1213 that limited table parallelism for all data generations that use the V2 pipeline.

Oracle - Fixed an issue where for subsetting data generation, the Maximum Character Limit was not calculated properly.

Databricks - Structural now supports writing Identity columns to tables.File connector - You can now assign the Timestamp Shift and Date Truncation generators to Parquet date fields.

PostgresSQL - Removed the option to run PostgreSQL jobs using the older flow. All jobs now run with the Data Pipeline v2.

Snowflake

• Fixed an issue where after any row failed to parse, Structural did not process the rest of an unloaded table file.
• For AWS, added retry support to address some transient issues with Amazon S3.

Bug fixes and other internal updates.

The detector for city names now ignores misleading values that are not city names.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

Updated the application to reflect the rename to Tonic Structural. Includes renaming the Tonic Settings view to Structural Settings.

From the Access Management tab of Structural Settings, users with permission to manage Structural access can now restore deleted users.

For a column that is part of a unique compound index, Structural now only suggests generators that can be used for unique columns.

Structural now detects SWIFT codes based on the format of the data in addition to the column name.

Fixed an issue where all subsetting WHERE clauses failed internally.

Databricks

• On self-hosted instances, you can now configure whether Structural creates the destination database schema for Databricks tables. The environment setting TONIC_DATABRICKS_SKIP_CREATE_DB indicates whether to skip the schema creation. The default is false. The environment setting TONIC_DATABRICKS_ENABLE_WORKSPACE_SKIP_CREATE_DB indicates whether to include the option in the workspace configuration, and use TONIC_DATABRICKS_SKIP_CREATE_DB to determine the default. The default is true. You can add these settings to the Environment Settings list on Structural Settings.
• When a cluster is resizing, and the workspace is configured with Use Databricks Job Cluster turned off, Structural data generation now waits for the resizing to complete.

File connector

• Fixed an issue where a copied local files workspace that contained Avro or Parquet files could not be loaded.
• Structural now supports authentication to Amazon S3 buckets using assumed roles for cross-account access.
• Fixed an issue where file connector workspaces did not load when they used an Amazon S3 source with environment-based authentication.

Snowflake

• Fixed a rare subsetting issue where processing a large table could cause the job to hang.
• Fixed an issue where Structural did not retry transient failures when it read unloaded files from a cloud storage provider.

The scheduled sensitivity scans are now daily instead of weekly. By default, the scans run every day at midnight. Structural scans the 10 workspaces that have the most recent activity. Activity is defined as either a user-initiated workspace event that is added to the Protection Audit Trail, or a data generation job.

On the details view for custom sensitivity rules, fixed an issue where the Edit Current Preset button was always disabled.

When a generation to Ephemeral fails, Structural job logs now include the Ephemeral logs and destination database pod logs.

For users who do not have permission to manage sensitivity rules, the Sensitivity Rules option now displays in a disabled state.

When you configure a workspace to write to a self-hosted Ephemeral instance, or to write to Ephemeral Cloud from a self-hosted Structural instance, the workspace configuration now includes an option to test the Ephemeral connection.

Yugabyte data connector - Structural now allows you to connect to databases on Yugabyte version 2024.1 and above. The Yugabyte data connector is available with a Professional or Enterprise license. It only supports Yugabyte SQL (YSQL).

When you configure a custom security rule, you can now create or edit the assigned generator preset. You can also use a workspace to preview the security rule results. The preview displays the matching columns for the selected workspace.

Structural can now detect the following additional sensitivity types:

• US driver’s license number
• Passport number
• Marital status
• GPS coordinates
• Non-birthday dates: admission date, discharge date, date of death
• US license plate

MySQL

• When the SQL mode ALLOW_INVALID_DATES is set, Structural now allows Passthrough for columns that contain invalid dates.

Snowflake

• For the Data Pipeline V2 process, reduced the frequency of polling cloud platforms for new unloaded files.

Fixed an issue on the webhook configuration panel where users could not click Save when the Message Body tab contained large property values.

Fixed an issue that caused the Notifications service to stop processing webhooks.

Improved the detection of name values to identify more specific types of names.

Amazon EMR

• Fixed an issue with previewing data after applying the Timestamp Shift Generator or Date Truncation Generator to timestamp columns with timezones.

Self-hosted instances can now schedule sensitivity scans to run automatically on a weekly basis. By default, the weekly scans are enabled and run each Sunday at midnight.

Structural can now detect the following additional sensitivity types:

• Money Amount
• Usernames

File connector

• Fixed a regression that made Table View unusable.

Oracle

• Fixed an issue that caused data generation to fail when subsetting with a date-based upstream table filter.

Salesforce

• Resolved issues where Salesforce workspaces became unrecoverable.
• Fixed issues with the use of session refresh tokens.

PostgreSQL

• Preserve Destination and Incremental tables no longer drop columns that reference user-defined types (UDTs) or extensions.
• Added support for the pgvector vector data type as a non-replaceable type that is always passed through to the destination database.

Snowflake on AWS

• Improved performance when copying passthrough tables when a workspace uses separate source and destination S3 buckets.

Structural can now detect the following additional sensitivity types that are defined by the HIPAA Safe Harbor method:

• Medical record numbers
• Health plan beneficiary numbers
• Account numbers
• Certificate and license numbers
• Web URLs
• Full face photographic images and similar images
• Biometric identifiers, including finger and voice prints

Removed the environment setting TONIC_SUBSETTING_CYCLE_BREAK_GREEDY_ALGORITHM. The greedy algorithm to compute the required cycle breaks for subsetting is no longer available.

Snowflake

• A new environment setting allows you to control whether Structural creates the destination database schema before it populates the destination data. By default, TONIC_SNOWFLAKE_SKIP_CREATE_DB is false, meaning that Structural creates the destination database and schema. If you set this to true, then Structural does not create the schema. You must create the destination database with the full schema. You can add TONIC_SNOWFLAKE_SKIP_CREATE_DB to the Environment Settings list on Tonic Settings.
• Improved performance when writing to Snowflake for de-identified tables and some Passthrough tables.

For post-job webhook URLs, you cannot use URLs that resolve to a private IPv4 range.

To provide the column name matching criteria for custom sensitivity rules, you can now use a regular expression.

The Structural sensitivity scan can now detect UK and Canada postal codes.

You can now use the Structural API to manage custom sensitivity rules.

When you configure a PostgreSQL or MySQL workspace to write the destination data to a container repository, you can now specify the name of the database.

Fixed an issue where the column sensitivity type was not updated when a later sensitivity scan detected a different type. Columns that are manually marked as sensitive are not affected.

Increased the number of column names that Structural uses to detect sensitivity types.

Amazon EMR

• Iceberg now correctly handles schemas that contain capitalized column names.

Salesforce

• The Continuous generator is now available.
• The Algebraic generator is now available.
• You can now use WHERE clauses in subsetting target table configuration.

When Structural detects a state abbreviation, it no longer identifies it as a full state name.

During a sensitivity scan, the value finders now look more holistically at both the data and the column name instead of assessing them individually.

Custom sensitivity rules - On self-hosted Enterprise instances, you can now configure custom sensitivity rules, which allow you to create your own sensitivity types. For each rule, you configure the general data type, text matching rules for the column name, and the recommended generator. Structural uses these rules during the sensitivity scan. Matching columns are included on the Recommended Generators by Sensitivity Type panel.

Toleration configuration for output to container repositories - Self-hosted customers who write output to a container repository can now set pod tolerations to enable pods to be scheduled on nodes that have taints. The tolerations are configured in environment settings. You can add these settings to the Environment Settings list on Tonic Settings.

MySQL

• Fixed an issue with subsetting for instances of MySQL that were deployed using Amazon Aurora where downstream tables were not populated properly.

When configuring a workspace to write output to an Ephemeral snapshot, you can now optionally configure the compute resources. By default, the resources are based on the size of the source database.

Sensitivity scans now detect name values more accurately.

Fixed an issue with certificate uploads for database settings.

Fixed an issue where the Structural application would hang after you created a workspace.

Shared logs are now transferred to an HTTPS endpoint instead of an Amazon S3 endpoint.

Amazon EMR

• Fixed an issue where Table View filters and table filters reported all WHERE clauses as invalid.

Amazon Redshift

• You can now configure a workspace to include or exclude specific schemas.

PostgreSQL

• When you choose to write output to an Ephemeral snapshot, you can now provide a custom configuration file.

Fixed an issue where sensitivity scans suggested generators based on substrings within a column name.

HTML is now removed from text in comment fields.

Fixed an issue where the XML Path generator did not work correctly.

A new environment setting, TONIC_SUBSETTING_CYCLE_BREAK_GREEDY_ALGORITHM, indicates whether to use a new, faster greedy algorithm to compute the required cycle breaks for subsetting. By default, the setting is false.

File connector

• Fixed an issue with uploading .txt files for local file workspaces.
• For CSV file groups, added an option to specify the encoding format of all files. If not specified, Structural attempts to detect the encoding automatically. When encoding cannot be determined, the automatic encoding detection now defaults to UTF-8 instead of windows-1252.

MongoDB

• Added an API endpoint to retrieve all of the field paths in a database.

MySQL

• Writing output to a container repository now works with multiple database schemas.
• Improved resilience to transient issues when copying tables.
• Fixed an issue with delayed retries of failed file uploads during data generation.

Oracle

• You can now write output data from an Oracle workspace to a Tonic Ephemeral snapshot.

Salesforce

• You can now provide connected app credentials in the workspace configuration. These fields are only displayed if the credentials are not configured in the TONIC_SALESFORCE_CONSUMER_KEY and TONIC_SALESFORCE_CONSUMER_SECRET environment settings.

Snowflake

• Fixed a regression where ALTER statements were inappropriately run through the GetDdl flow.

Improved the accuracy of name detection.

MongoDB

• On Collection View, for hybrid view, added a Filters panel. For single view, you can now filter fields by value.

Oracle

• The Data Generation Pipeline v2 for Oracle now supports subsetting.

Salesforce data connector - The Salesforce data connector is now available for self-hosted instances that have a Professional or Enterprise license. It is currently only available by request. To request access to the Salesforce data connector, contact Tonic.ai support.

Linking address columns for recommended generators - The recommended generators panel in Privacy Hub now indicates when address columns should be linked. The columns are displayed in groups. You then apply the recommended generators to all of the columns in the group, and the columns are automatically linked.

Other updates

Fixed an issue with subsetting. When processing upstream tables with nullable foreign keys that had no referenced key values to process, upstream filters were not applied.

Improved performance of the Conditional generator when using the IS IN operator.

The upsert option for workspaces is now out of beta.

Fixed an issue where the number of generators that are slow to compute was calculated incorrectly, which affected how we parallelized the generator processing.

The default value for the environment setting TONIC_ORACLE_DBLINK_ENABLED is changed to false. The plan is to eventually remove the feature.

Fixed an issue where the TONIC_DISABLE_IPV6 setting did not completely prevent services from binding to ipv6 addresses.

When applied to a numeric type column, the SSN generator now by default generates values without hyphens.

Amazon EMR

• Added support for the Iceberg framework in Glue. If you use the Iceberg framework, then in the Spark Configuration section of the workspace details, make sure to add the following configurations: spark.sql.catalog.glue_catalog, .warehouse, .catalog-impl, and .io-impl.

File connector

• Improved resilience to missing cloud files when configuring file groups, previewing data, and running generation jobs.

MongoDB

• Fixed an issue with creating virtual foreign keys in MongoDB workspaces.

MySQL

• Improved the performance of the destination database teardown step for databases that have many partitions.

Oracle

• You can now use the Data Pipeline V2 process to run data generation for an Oracle workspace. The Confirm Generation panel includes a toggle to enable or disable the new process. Note that you cannot use the new process when subsetting is enabled.

Snowflake

• Fixed a subsetting issue where extra downstream rows were included when no primary keys existed in the table relationship.

Helm charts for deploying Structural to Kubernetes are now published at quay.io/tonicai/structural in addition to GitHub.

From the recommended generators panel on Privacy Hub, you can now enable or disable self-consistency for all columns within a sensitivity category.

Fixed an issue in Table View that sometimes caused the column order to be incorrect.

Added an environment variable TONIC_DISABLE_IPV6 to the PyML container. When set to true, the container no longer listens on IPv6 addresses.

File connector

• Added support for Avro files.

MySQL

• Fixed an issue with validating subsetting target table WHERE clauses when the table or schema name contained special characters.

PostgreSQL

• Added limited support for ltree columns on versions older than 1.2. For tables where all columns are assigned the Passthrough generator, Structural copies the ltree data from the source database to the destination database. In tables that are de-identified, ltree columns are nullified in the destination database. If an ltree column is not nullable, then all of the columns in the table must be assigned the Passthrough generator.

Snowflake

• Added support for date and time columns that have a seconds precision of 0.
• Fixed an issue where the incorrect credentials were used when using separate source and destination buckets and credentials for Amazon S3.

SQL Server

• Fixed an issue introduced in v1178 where XML columns might be persisted as nvarchar columns in the destination database.

From the recommended generators panel on Privacy Hub, you can now enable or disable self-consistency for each column.

When you create a virtual foreign key, the nullability of the foreign key is now determined by the nullability of the foreign key column.

On Privacy Hub and the job details view, changed the label on the Download option to Reports and Logs.

Db2 for LUW

• Subsetting is now supported.

MySQL

• Fixed an issue where data generation failed when object identifiers included symbol characters.

PostgreSQL

• Workspaces with PostgreSQL 16 databases can now write output to a container repository.

Sending telemetry to Tonic.ai is now required and enforced. After 5 days of failed telemetry connections, a warning displays. If the connection issue persists for 15 days, data generation is restricted.

Structural Cloud users can now reset their passwords from the login page. To reset your password, click Forgot your password?.

You can now set the TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS environment setting from the Environment Settings tab on Tonic Settings.

File connector

• Added support for AWS C2S Secret Regions.

SQL Server

• Improved the upsert logic to prevent jobs from hanging.

Added a new API endpoint to resolve all schema changes in a workspace. You can choose whether to resolve only conflicting changes, only notifications, or all of the schema changes.

Fixed an issue that caused the Tonic Structural PyML Service to be unreachable in IPV4-only containers.

Added a new conflicting schema change when a column that has an assigned generator becomes a foreign key. Foreign key columns must inherit the generator from the primary key.

Structural can now generate data with subsetting when a primary key table is truncated, as long as the foreign keys that reference the primary key are nullable.

Amazon Redshift

• Added support to pass through varbyte, geometry, and hllsketch types.

File connector

• For CSV, XML, and JSON files, fixed issues with the data preview in Database View and Table View. The preview no longer includes extra rows, and the preview now correctly reflects the Skip first N rows setting.
• Fixed the validation of file groups that only contained compressed files.

Snowflake

• For subsetting, added support for virtual foreign keys.

SQL Server

• Fixed an issue that caused data generations to fail for versions of SQL Server older than SQL Server 2016.

For the notifications image, replaced alpine with ubuntu.

File connector

• For a local files workspace, the job details view for a successful data generation now includes an option to download the transformed files that were produced by that job.
• For workspaces that use files from cloud storage, you can now include prefix patterns in the file group definition. You can also provide file extension filters. The file group then includes all of the files that match a prefix pattern and the file extension filters. The file group details now include the content type, file extension filter, and prefix patterns.
• For cloud storage workspaces, users who do not have permission to view all buckets can once again specify the bucket to view.
• For cloud storage workspaces, added a configuration option to only process new files during data generation. For existing file groups, the new configuration is off by default. For new file groups, the configuration is enabled by default.

MongoDB

• Fixed an issue where environment setting updates from the UI required a restart to take effect.

MySQL

• For a workspace that is configured to write output to Ephemeral, you can now provide a custom configuration file.

Snowflake

• Fixed an issue where the Subsetting tab disappeared after you edited a workspace.

SQL Server

• For versions 2017, 2019, and 2022, you can now configure workspaces to write output to a container repository.

During the free trial, Structural now displays next step highlights to indicate the next recommended action. When you hover over the recommended action, Structural displays an explanatory tooltip.

A new environment setting, TONIC_DB_MAX_POOL_SIZE, sets the connection pool size for the Structural application database. The default value is 3.

Fixed an issue where the preview data in the JSON Mask generator editor did not respect the applied table filter.

File connector

• For workspaces that connect to Amazon S3, you can now specify different credentials for the source and output locations.
• For cloud storage workspaces, fixed a regression in Tonic v1146 where the file explorer sometimes displayed incorrect results and files could not be added.
• You can now add Parquet files to a file connector file group, either from cloud storage or a local file system. You cannot select the following Parquet file types: HalfFloat, Struct, Union, Dictionary, Map, List, FixedSizeList, or arrays of any type. There also is no file preview for Parquet files.

MongoDB

• Fixed an issue that prevented the use of regex conditions in the conditional generator.
• You can now select the UUID Key generator as a sub-generator when configuring a composite generator.

PostgreSQL

• Fixed an issue where Preserve Destination tables sometimes were not preserved when data generation failed.

Fixed an issue in Table View where characters were sometimes represented inaccurately. For example, a lowercase x would become a multiplication symbol.

Fixed an issue where data generation to Tonic Ephemeral Cloud failed with the error "Ephemeral URL not found".

For a schema change that adds a new column, both the Schema Changes view and the API response now include the data type for the new column.

For the Structural free trial, Structural now displays a checklist for each workspace. There are slightly different checklists for database-based and file connector workspaces.

Added support to run Structural in dual-stack networks and IPv6-only network environments.

On the workspace details view, fixed an issue where an Ephemeral API key appeared to be populated when no value was provided.

Fixed an issue that caused data generation to Ephemeral to fail with "Ephemeral output must be configured".

Fixed an issue where data generation to Ephemeral failed after a first successful run.

Amazon Redshift

• You can now enable TLS for workspace data connections.

MySQL

• Fixed an authentication issue that prevented output from being written to an Ephemeral snapshot.

SQL Server

• You can now configure a SQL Server workspace to write output to an Ephemeral snapshot.

Writing output to a Tonic Ephemeral snapshot - For database types that Tonic Ephemeral supports (currently PostgreSQL and MySQL), you can now write the output to an Ephemeral user snapshot. This replaces the option to write the output to an Ephemeral database, except for workspaces in the Structural free trial. In Ephemeral, you can use the user snapshot to start new Ephemeral databases.

Other updates

For the UUID Key generator, added a new configuration option, Preserve Version and Variant. By default, the setting is turned off. When turned on, the version and variant bits from the source UUID are preserved in the output value. For the API, the new setting is preserveVersionAndVariant.

In the Tonic Structural free trial, the sample workspace now by default writes the output to a Tonic Ephemeral database.

Fixed an issue where vertical scrolling was sometimes blocked.

You can now configure the allowed SSL/TLS protocols and ciphers on the Tonic Web Server. To configure the protocols and ciphers, use the environment settings TONIC_WEB_SERVER_TLS_PROTOCOLS and TONIC_WEB_SERVER_TLS_CIPHERS.

File Connector

• Data generation jobs no longer fail when they encounter a configured file group file that no longer exists in the source cloud storage location.
• For local file workspaces, fixed a regression from Tonic v1136 where users could not upload additional local files into a file group.

MongoDB

• Fixed an issue that caused generations to fail when documents contained empty arrays or document fields.

MySQL

• Improved job warnings around missing tablespaces in the destination database.

The Structural API now includes endpoints to get and set the assigned table modes and table filters for a workspace.

Fixed an issue where the workspace audit trail displayed generator preset events that occurred before the workspace was created.

Fixed an issue where an error was returned when users tried to export selected files from a file group.

Improved error message when Structural cannot write output to Ephemeral because Ephemeral does not have a compatible base image for the database.

Databricks

• When a workspace is configured with an unsupported version of Databricks, the error message now suggests the supported Databricks versions.
• Updated the default spark_version to 14.3.

Google BigQuery

• Fixed an issue with applying some generators to numeric columns on large tables.

Oracle

• Fixed an issue where rows that contained NULL values in a VARCHAR2(1) column were dropped during data generation.

Output to a Tonic Ephemeral database - Tonic Ephemeral is a separate Tonic.ai product that allows you to create temporary databases. On Tonic Cloud, for data connectors that Ephemeral supports (currently PostgreSQL and MySQL), you can configure the workspace to write the destination data to an Ephemeral database. This is the default option for data connectors that Ephemeral supports.

The database belongs to your Ephemeral account. If you do not already have an Ephemeral account, then Tonic automatically creates a two-week Ephemeral free trial account for you. The Tonic data generation job details provide access to the database connection details.

Free trial checklist - During the free trial, the sample workspace now includes a checklist to help users get through the required steps to complete their first data generation.

Other updates

Free trial users can no longer use a public email address to create an account.

Fixed an issue where password reset links lead to a blank page.

Fixed an issue where pay-as-you-go users would see the countdown for a free trial.

In the sample workspace, fixed an issue where a faulty destination database template caused an error when a user tried to update it.

Google BigQuery

• Tonic can now de-identify snapshot tables.

Oracle

• For databases that use non-default database character sets, fixed an issue where rows that contained character data sometimes failed to be written to the destination database.
• For databases that use non-default database character sets, fixed an issue where passthrough CLOB data types had extra bytes inserted when they were copied to the destination database.

PostgreSQL

• If you do not use extensions, then the destination database no longer requires a super user.

Snowflake

• Subsetting is now supported on Snowflake workspaces. The table filtering option is still available.

You can now manually add selected environment settings to the Environment Settings list on Tonic Settings.

Improved the performance of data previews in the Tonic application.

For workspaces that write output data to a container repository, fixed an issue that prevented GAR credentials from being saved.

Google BigQuery

• Improved performance for data preview.

Snowflake

• The workspace configuration now includes a Trust Server Certificate option for the source and destination connections. When enabled, it indicates to bypass certificate revocation checks.
• Data generation jobs are now more resilient when they encounter views that have dependencies that Tonic does not have permissions for.
• Destination privileges can now exclude schema creation. If the user does not have schema creation permissions, then the schemas must already exist in the destination database.

SQL Server

• The upsert function now supports tables that have identity columns but do not have primary keys.

New Db2 for LUW data connector - Tonic now has a data connector for IBM Db2 for Linux, Unix, and Windows (Db2 for LUW). Tonic supports Db2 for LUW version 11.5.

Other updates

When the AI Synthesizer is used in a workspace, Tonic now verifies before data generation that the AI Synthesizer does not use more than the maximum allowed categories.

Amazon EMR

• Improved data generation performance when using Spark 2.4.0-3.2.x.Databricks
  
  
  • Fixed an issue in Databricks 11.3+ where a generator that was consistent on another column received modified input values if a generator was applied to the consistency column. Generators that are consistent on another column now correctly receive unmodified input values.
  • Improved data generation performance when using Spark 2.4.0-3.2.x.
  Google BigQuery
  
  
  • You can now de-identify views, except for views that are written in Google Legacy SQL. On the workspace settings view, to enable de-identification of views, toggle De-identify Views to the on position. You can then assign table modes and generators to the views. In the table lists, views are identified by (view) after the view name.
    
    For each view, Tonic:
    
    
    • Writes the de-identified data to a table called <view name>_tonic_table.
    • Creates a view that has the same name and metadata as the view in the source data, but is populated from the destination table.
  • Fixed an issue where Passthrough tables that had table constraints failed to copy to the destination database.
  • Fixed an issue where cloned tables did not appear in the Tonic application, but caused job failures. Cloned tables now display within Tonic and can be managed in the same way as any other table.
  Oracle
  
  
  • For Oracle 12, improved the schema remapping across multiple object types and configurations.
  • Fixed an issue where Tonic failed to copy database link objects that were configured using passwords. Tonic no longer processes database links. If required, users must manually create database links in the destination schema.
  • Fixed an issue where tables that contained computed columns dropped rows during data generation.
  Snowflake
  
  
  • You can now assign the JSON Mask generator to Snowflake variant columns.

Privacy Report PDF file

We added a new Privacy Report PDF that you can download from Privacy Hub and the job details view. The Privacy Report PDF contains a summary of the privacy ranking values, visualizations to summarize the workspace column privacy rankings based on the applied generators, and a summary table that contains the .csv Privacy Report data.

To accommodate the new file, on Privacy Hub and the job details page, the available downloads are combined into a Download menu.

Assigning recommended generators from Database View

On Database View, when an unprotected column has a recommended generator, the generator name tag now displays the type of sensitive data that was detected.

When you click the generator name tag, Tonic displays a panel that displays the sensitivity type, the recommended generator, and sample source and output data based on the recommended generator. The panel provides options to either apply or ignore the recommendation.

Other updates

Fixed an issue where changing the configuration of a generator preset did not accurately update the count of occurrences of the preset.

Oracle

• Fixed an issue with Oracle 12.1 and 12.2 where constraints that had an index creation statement failed to apply.
• Tonic can now de-identify external tables. External tables display in the Tonic application and can be assigned table modes and generators. During data generation, external tables are created as relational tables in the destination schema.

Fixed an issue with the Name generator where capitalization was not preserved if consistency was disabled.

For Table View, fixed an issue where the delete button to remove the generator assignment was sometimes hidden.

Oracle

• Tonic now supports Oracle Advanced Queues. Queue tables and queues are created in the destination database. Messages are not copied from the source database to the destination database. Queue subscribers are not currently supported. You must add them to the destination database manually.

PostgreSQL

• Tonic can now de-identify the ltree data type.

SQL Server

• Fixed an issue where indexes were not created on schema bound views.

Redesigned Database View

We redesigned Database View to improve the display and the filtering.

In the updated columns list, the Column column contains the schema, table, and column name, and the column data type. It provides access to the data preview option.

The Applied Generator column shows the applied generator. Applied Generator indicates when a column is unprotected, when the column is a primary or foreign key, and when the configuration overrides the parent workspace. If the table mode is not De-Identify, it shows the table mode. It provides access to the commenting option.

Filters other than the column name filter are moved under the Filters option. There are also new filters for the sensitivity type (the type of sensitive data that Tonic detected in the column) and whether the column has a recommended generator.

Privacy Report updates

In the Privacy Report, new column, Column Privacy Rank, indicates the privacy ranking for a column based on the assigned generator and generator configuration. The generator summary and generator reference include the possible privacy ranking values for each generator.

Added a new column, Tonic Detected Sensitivity, that indicates whether the Tonic sensitivity scan identified the column as sensitive. Renamed the Is Sensitive column to Current Sensitivity. Current Sensitivity indicates whether the column is currently marked as sensitive.

Also corrected an earlier issue with the order of the columns.

Other updates

Fixed an issue that caused all subset runs to record the percentage of rows in the subset as 100%. Subset runs that occur after updating to this version display the correct percentage.

The option to write output data to a container repository is out of beta.

Databricks

• For Databricks 13+, writing output to delta tables now uses the table’s liquid clustering configuration to cluster the insertion.
• Fixed an issue with loading tables into the Tonic application.

Google BigQuery

• Improved batch size calculations to reduce memory pressure during data generation jobs.
• Fixed an issue with applying the Timestamp Shift and Date Truncation generators on datetime or timestamp data when TONIC_GRPC_ENABLED was false.
• For the Timestamp Shift and Date Truncation generators, improved support for time values that include fractional seconds.
• Fixed an issue where the Character Substitution generator failed when assigned to a NUMERIC or BIGNUMERIC column.

Oracle

• Tonic now supports IDENTITY columns. Before this change, IDENTITYM columns caused errors during destination database creation.

For the Custom Categorical generator, you can now add a NULL value to the available custom category values. To indicate a NULL value, use the keyword {NULL}.

Made the following API updates to better accommodate users of the previous version of the API:

• jobs/{id}/workspace_snapshot now returns the WorkspaceDataModel object.
• Fixed an issue where workspace_snapshot could return empty replacements.
• Added a new endpoint, GET jobs/{id}/workspace_snapshot?api-version=v2023_07_00, that returns V17WorkspaceDataModel

Databricks

• When writing output to delta tables, the destination tables should now retain the TBLPROPERTIES from the source delta table, including 'delta.feature.allowColumnDefaults'.

Redesigned data model for generator assignments - The new version of the Tonic API includes a redesign of the data model for generator assignments. To use the previous version of the generator assignment data model, make sure that your API calls specify version 2023.07.0.

The generator assignment data model redesign includes the following changes:

• For all generators, moved the following fields to the metadata object in the link object:
  • presetId
  • generatorId
  • customValueProcessor
  • encryptionProcessor
• For the <value type>: Mask composite generators:
  • Moved pathExpression to the metadata object in the link object.
  • Removed the following fields from the link object:
    • subPresetId
    • subGeneratorId
    • customSubGeneratorValueProcessor
  • Added the following fields to the subGeneratorMetadata object under metadata:
    • presetId
    • generatorId
    • customValueProcessor
  Other updates
  
  Updated the sensitivity scan to better identify company and organization names and suggest the appropriate generator.
  
  Databricks
  
  
  • The Conditional Generator is now supported on version 11.3+.
  Google BigQuery
  
  
  • Fixed some cases where a copied View did not have its definition updated to reference resources within the destination dataset. This could result in failures when attempting to copy the View into the destination dataset.
  • Added support for Preserve Destination table mode.
  • Fixed an issue where sub-queries in table filter clauses did not work if TONIC_GRPC_ENABLED was set to true.
  MySQL
  
  
  • You can now enable diagnostic logging for a MySQL data generation job.
  PostgreSQL
  
  
  • Improved the destination database permissions check on Google Cloud SQL to handle additional superuser authentication setups.
  SQL Server
  
  
  • Data generation now preserves hidden properties of columns.

NOTE: v1074 was removed.

If your instance of Tonic is deployed on Docker, you can now use an external Kubernetes cluster to enable the option to write destination data to container artifacts.

You can now assign the Integer Key generator to a column with a decimal data type. The actual column values must still be integers.

Fixed an issue in Table View where an error displayed if you changed the selected table while the data was loading.

Databricks

• For cluster versions 11.3 and above, Tonic now displays accurate lists of available and suggested generators.

File connector

• Fixed an issue that prevented users from adding files to existing local file groups.

SQL Server

• Fixed an issue that caused errors when creating inline functions in the destination database.

The Enable Diagnostic Logging global permission is now granted to the built-in Account Admin permission set.

Databricks

• CREATE CATALOG or CREATE SCHEMA permissions are no longer required if the destination catalog or schema already exists.
• For Databricks 11.3 and higher, added support for the Random Double generator.
• For Databricks 11.3 and higher, added support for the IP Address generator.

Diagnostic logging for data generation - By default, Tonic now redacts sensitive data in data generation log files.

When users start a data generation or upsert job, if they have the new global permission Enable diagnostic logging, they can choose to enable diagnostic logging, which does not redact the logs. The Enable diagnostic logging permission is also required to download the diagnostic logs. By default, the permission is only granted to the Admin and Admin (Environment) global permission sets.

In addition to the option for individual jobs, there are environment settings that enable diagnostic logging for specific data connectors.

Other updates

In the Release Candidate version of the API, the response model for the GET /api/workspace/minimal endpoint has been updated for more straightforward de-serialization.

Fixed an issue where a non-unique composite primary key column could only be assigned unique generators.

Users can now press Enter to finish copying a workspace or a generator preset, instead of having to click Copy.

File connector

• Added support for the Conditional generator for file groups that contain CSV files.

Google BigQuery

• Post-job scripts now support DDL and EXPORT.

Oracle

• Fixed an issue with the permissions check that prevented connecting to Amazon RDS for Oracle databases.

SQL Server

• In Data Definition Language (DDL) that applies to the destination database, Tonic now strips WITH INLINE clauses from definitions of user-defined functions (UDF). Inlining does not require these clauses. WITH_INLINE clauses in UDF definitions that do not meet the requirements for inlining can prevent the UDF from being restored properly in the destination database.
• Fixed an issue where the order of XML columns was changed in the destination database.

For the OpenID Connect (OIDC) SSO integration, Tonic now supports authentication by client secret that uses HTTP basic authentication (client_secret_basic). To provide the client secret, configure the TONIC_SSO_CLIENT_SECRET environment setting.

SQL Server

• A new environment setting, TONIC_SQL_SERVER_SKIP_CREATE_DB, indicates whether to skip schema creation for the destination database. If true, then Tonic does not create the schema. It uses the existing schema to populate the destination database. The default is false. You can configure this environment setting from the Environment Settings tab on Tonic Settings.

NOTE: These releases were removed.

During free trial signup, the data connector options now include an option to use local files for the source data. This creates a file connector workspace for local files, and displays the File Groups view to allow the free trial user to start to add file groups to the workspace.

Added an environment setting, Tonic Test Connection Timeout In Seconds (TONIC_TEST_CONNECTION_TIMEOUT_IN_SECONDS), that you can set from the Environment Settings tab on Tonic Settings. This setting configures the timeout for testing a database connection. Previously, connection test attempts timed out after 5 seconds. The new default is 15 seconds.

When you configure a workspace to write the output to container artifacts, you can now specify custom resources for the Kubernetes pod, including the ephemeral storage, memory, and CPU millicores.

Improved performance when marking a large number of columns as not sensitive.

Fixed an issue that caused Tonic workers that are deployed on Docker to crash unexpectedly.

For numeric columns that support arbitrary precision and scale, when the scale is 0 (for example, NUMERIC(N,0)), or when the underlying values are all integers, these columns are now supported as primary keys for the purpose of subsetting.

Amazon EMR

• You can now use the Timestamp Shift generator as a sub-generator within the Struct Mask generator.

Amazon EMR and Databricks

• When files are saved to non-job-specific file destinations, the new environment setting TONIC_WORKSPACE_DEFAULT_SAVE_MODE indicates the mode to use. If set to a value other than null (Ignore, Append, Overwrite), this setting takes precedence over TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE.

Google BigQuery

• Fixed an issue introduced in v982 where data generation failed with an HTTP 404 Not Found error for the destination table when the source and destination are in different BigQuery projects.

MongoDB

• Added an environment setting, TONIC_DOCUMENT_MAX_DEPTH, to configure the maximum depth of JSON document that can be handled. The default value, which is also the recommended minimum value, is 32.
• For bulk apply, fields with the same selection path but different data types no longer share a selection state.

SQL Server

• Fixed an issue to properly restore schema-bound views in dependency order.

NOTE: v1051 through v1053 were removed.

Enable administration functions in Tonic Cloud - For Tonic Cloud customers, the new Account Admin permission set provides access to Tonic administration functions for their organization. The Account Admin can reset passwords, delete users, copy and share all workspaces, and download the usage report. The Account Admin permission set is initially granted to the first user in the organization.

Databricks

• For the Test Cluster Connection option, added an error message when the specified cluster cannot describe all the tables in the specified catalog + database.
• Improved error messaging for the Test Cluster Connection option.

File connector

• When you use the API to add files to a file group, Tonic now validates that the file exists and does not duplicate a file that is already in the group.

MySQL

• Added a new environment setting, TONIC_MYSQL_MAX_CONCURRENT_INDEX_CREATION, to limit the number of concurrent indexes that are created. The default value is 0, which indicates that there is no limit.

SQL Server

• External tables are no longer displayed on the application or carried over to the destination database.

When you select the option to write destination data to container artifacts, you can now use Google Artifact Registry (GAR) authorization using Google Cloud Platform (GCP) service account keys.

For the JSON Mask and XML Mask generators, fixed the data preview for JSON or XML field samples that are larger than 120MB by generating a smaller subset of the field.

The Name generator now supports consistency with other columns.

Added new API endpoints to retrieve and set table replacements. These new endpoints are compatible with workspaces for data connectors that do not have schemas, such as Spark-based databases and the file connector. The existing endpoints, which require you to provide a schema, eventually will be deprecated.

Amazon EMR

• You can now write data generation output either to a single, static location or to a job-specific folder. Previously, all data generation output was exported to a job-specific folder.

File connector

• You can now use the Continuous generator in file connector workspaces.

MySQL

• When writing destination data to a container repository, you can now provide the content of a my.cnf file that contains custom configuration.
• Fixed an issue where indexes that use expressions in their definition caused an error during data generation.

Oracle

• To fix an issue where turning off database links caused triggers to break, we now run the ALTER TRIGGER statements after the TRIGGERS are validated.

PostgreSQL

• For Azure PostgreSQL, fixed an issue with the destination database connection test.
• For Azure PostgreSQL, fixed the connection test check for schemas that are not admin owned.

SQL Server

• Data generation jobs that are canceled during the schema gathering phase now cancel more promptly.

Snowflake

• On Tonic Cloud, the Snowflake with Azure option is now enabled correctly.
• Free trial users can now use Snowflake on AWS and Snowflake on Azure.

Added an environment setting TONIC_DELETE_COLUMN_SCHEMA_ON_WORKSPACE_DELETE. If the setting is true, then when a workspace is deleted, Tonic also deletes the associated rows from the ColumnSchemas table in the Tonic application database.

The new environment setting TONIC_NOTIFICATION_SMTP_TRUST_CERTIFICATE indicates whether to allow the SMTP server certificate to be trusted.

Improved the performance of previewing data in Privacy Hub.

Fixed an issue where SSO groups were not removed when the value of TONIC_SSO_GROUP_FILTER_REGEX changed in a way that excluded previously imported groups. The removed groups are removed from any workspaces that they were granted access to.

For the Timestamp Shift Generator, added Month and Year as options for the date part to use to set the allowed range.

When writing data to container artifacts, Tonic now first shuts down the temporary database before it begins to write data to the container.

Amazon EMR

• Fixed an issue where, when the Athena workgroup used engine version 3, you could not preview data for tables that contained struct data types. Tonic now supports both Athena engine versions 2 and 3.
• Fixed an issue where Professional plan customers could not save workspaces.
• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.
• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

Databricks

• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.
• The FNR generator is now supported.
• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

MongoDB

• Fixed an issue where erroneous schema change items were reported when the frequency of a field changed.

Oracle

• Fixed an issue where when Oracle database links were not enabled (TONIC_ORACLE_DBLINK_ENABLED is false), privileges were not copied from the source to the destination.

Amazon EMR

• Fixed an issue where jobs failed if Tonic could not retrieve the tracking link for the EMR Step logs. To ensure maximum compatibility with all AWS regions, tracking links are now valid for 7 days instead of 14 days.
• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Databricks

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

MySQL

• On destination databases, Tonic no longer checks for REPLICATION CLIENT and REPLICATION SLAVE grants.

Spark SDK

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Spark with Livy

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

When an email notification contains a link to a comment, clicking the link now correctly navigates to and displays the comment.

Added a flag to mark log files that might contain sensitive information.

You can now assign the Timestamp Shift generator to primary key columns, including columns that are part of a composite primary key.

When writing destination data to a container repository, based on the size of the source database, Tonic attaches resource requests to the datapacker pod to verify that the cluster has sufficient resources for the data.

Tonic now supports writing destination data to container registries on Amazon Elastic Container Registry (Amazon ECR).

Fixed an issue where users could not assign access to permission sets when they did not have access to create and manage custom permission sets.

For foreign keys that have multiple primary keys, Tonic now prevents data generation if the assigned generation configuration for the primary key columns is inconsistent.

Standardized the display format of the heading area of the workspace management views.

File connector

• Fixed an issue with deleting multiple files from a file group.

MongoDB

• From Privacy Hub, you can now use the option to review and apply recommended generators to all of the detected sensitive fields.
• When there are schema changes that are not yet scanned, such as a new collection or new fields in a collection, they are now handled as Passthrough. The schema changes no longer cause data generation to fail.

MySQL

• You can now configure a MySQL workspace to write the destination data to a container repository.

PostgreSQL

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.
• Customers are now by default automatically enrolled in Data Pipeline V2 mode, and do not see the option to enable or disable it. The ability to enable or disable Data Pipeline V2 is granted to individual customers as needed.

Snowflake

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.

API endpoints for subset configuration - The Tonic API now includes endpoints for subsetting configuration. You can use the endpoints to retrieve the subsetting configurations for a workspace, update subsetting configuration, and remove subsetting configuration. A subsetting configuration identifies a table as either a target table (percentage or WHERE clause) or a lookup table.

Improved how Tonic identifies values as names, to reduce false positives.

For upsert data generation, fixed an issue that caused failures on tables that contain foreign keys but no primary keys.

File connector

• Sensitivity scans no longer discard the most recently generated set of downloadable generated files, which are generated from files uploaded from a local file system.

MongoDB

• Fixed an issue that caused duplicate or inaccurate schema issues to be displayed on Schema Changes view.

Oracle

• Tonic now validates destination database objects, such as views and packages, that were invalidated by the data generation. A warning is issued for objects that fail to validate. Validation failure can be caused by insufficient permissions for the Tonic user on the destination schema.

PostgreSQL

• For Data Pipeline V2 data generation, fixed a race condition that could cause data generation to fail.

Toggle between database server and container repository - On the workspace configuration view, if the data connector supports writing to a container repository, you can now switch between writing to a database server and writing to a container repository. Tonic saves the information you provide for each option.

Workspace override for statistics seed - From the workspace configuration view, you now can override the Tonic-wide statistics seed that is set as the value of the TONIC_STATISTICS_SEED environment setting. You can either provide a custom seed value for the workspace, or disable consistency across data generation runs for the workspace. Consistency also applies across workspaces that have the same custom seed value.

New telemetry URL - Tonic telemetry now routes through https://telemetry.tonic.ai/ instead of https://api2.amplitude.com/. The following IP addresses must be allowed:

• 75.2.74.76
• 99.83.246.105

The following IP addresses no longer need to be allowed: 52.43.241.47, 54.186.140.101, 54.203.75.164, 44.236.122.176, 34.215.78.194, 54.149.61.206, 54.191.147.220, 52.24.22.222, 52.37.168.36, 54.213.191.53, 54.68.108.104, 52.10.121.164, 52.27.184.186, 44.239.225.209, 54.148.216.233, 52.88.224.247Other updates

Fixed an issue in the Kubernetes client configuration that caused Tonic to reject the SSL certificate of a Kubernetes context.

Fixed an issue where, during subset configuration, an error was returned incorrectly if the user had permission to edit subsetting but not to view source data.

Improved the error message that displays when an uploaded virtual foreign key file is invalid.

Fixed an issue that prevented the Character Substitution generator from being used on primary and foreign key columns when subsetting.

When a workspace import encounters an unhandled exception, the error now displays correctly.

For the Address generator, the Country and Country Code options can now be linked. When linked, the country and country code are either “United States” or “US” to match the other linkable components, which are locations in the United States.

Fixed an issue where SSO login using Okta did not work when a custom authorization server is used.

Fixed an issue where, when using data science mode on Tonic Cloud, users could not download CSV files that contained synthetic data.

Data generation jobs that write to a container now include the datapacker logs, if the worker has permissions to read the pods and logs.

Fixed an issue with uploading container output to Harbor.

For string data type columns:

• Added yyyy/MM/dd as a valid format for the Timestamp Shift generator.
• Added yyyy/MM/dd and MMddyyyy as available output formats for the Random Timestamp generator.

Databricks

• For Databricks 11.3 and later, the Databricks data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark SDK

• The Spark SDK data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark with Livy

• For Spark 2.3.x and 2.4.2, the Spark with Livy data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

SQL Server

• Added a new worker environment setting, SQL_SERVER_SCRIPT_CROSS_DATABASE_REFERENCES. The default is true, which preserves the existing behavior. To prevent scripting of objects that are defined in other databases that the source database references, change the setting to false.
• Fixed issues where SQL Server connections did not honor the TONIC_SQL_COMMAND_TIMEOUT environment setting. The default is 0, which indicates an infinite timeout.

Apply recommended generators to multiple detected sensitive columns - On Privacy Hub, a new banner displays the number of detected (not manually designated) sensitive columns that are not protected. From the banner, you can display the list of columns, grouped by sensitivity type. For each sensitivity type, for the selected columns, you can:

There is also an option to apply the recommended generators to all of the selected columns across all of the sensitivity types.

Writing destination data to container registries (beta feature) - For data connectors that support it, you can now configure a workspace to write destination data to container artifacts on a container registry instead of to a database server. The Job History view provides access to the generated artifacts for each job.

Other updates

Fixed an issue where generated API documentation did not exclude endpoints and schemas from API versions other than the version being viewed.

Fixed an issue where the Protection Audit Trail displayed the incorrect override status of columns in child workspaces.

On the System Status tab of Tonic Settings, fixed an issue where the Data Sharing section incorrectly showed logging as unable to connect.

Databricks

• On the table mode selection panel, for workspaces that write to Databricks Delta tables, disabled the Error on Overwrite table configuration for tables. The setting is not used in this case.

File connector

• In the file group configuration, added options to skip lines from the beginning or end of CSV files.

MongoDB

• On Collection View, fixed an issue that prevented array fields from displaying in the Hybrid Document View of a collection.

Oracle

• For new workspaces, the Preserve source database file storage preferences setting is now off by default instead of on. Existing workspaces are not affected by this change.

Configure Tonic environment settings from Tonic Settings - On the Tonic Settings view, a new Environment Settings tab allows you to configure a subset of Tonic environment settings (previously referred to as Tonic environment variables). To use the Environment Settings tab to configure settings, you must have the new Manage environment settings global permission. When you change the values from Tonic Settings, you do not need to restart Tonic.

Other updates

For the Regex Mask generator, fixed an issue where quotes in regular expressions were malformed.

Fixed an issue where the Test Webhook option failed when it shouldn’t.

Fixed issue where the Confirm Data Generation panel erroneously indicated that Tonic could not connect to the destination database when it actually was only unable to connect to the source database.

For the Address generator, removed an inappropriate value from the possible street names.

Improved error messages when Tonic containers fail to start because of missing or incorrect environment variable values.

Databricks

• Fixed source catalog workspace handling when the Databricks Unity source catalog contains a table with a key constraint.
• For Databricks 10.4 and earlier, generations to output Databricks tables now correctly create a new destination Databricks database if the specified one is not found.

File connector

• For file connector workspaces, the post-job scripts option is now hidden. The file connector does not support post-job scripts.
• For Amazon S3, fixed an issue where Tonic did not delete a temporary file that was created to test permissions.

Google BigQuery

• External tables are now supported with some restrictions. Destination tables must be native BigQuery tables and cannot be external tables, whether masked or passthrough. Performance might be affected because of Google’s implementation of external tables.

MongoDB

• Fixed an issue where an error was thrown when generators were applied.

Snowflake on AWS and Azure

• Added an option to use key pair authentication to connect to the source and destination databases.

Logging and telemetry connection status - On the System Status tab of Tonic Settings, a new Data Sharing section provides a summary of the logging and telemetry connectivity to the Tonic backend. The new section indicates:

• Whether sending logs and telemetry to Tonic.ai is enabled
• If they are enabled, whether Tonic is able to connect in order to send the logs and telemetry

Other updates

Tonic now checks for invalid virtual foreign keys for all data generation. Previously it only ran the check for subsetting data generation.

Fixed an issue where the Continuous generator failed when a column contained only NULL values.

Improved performance for the Continuous Generator.

Fixed an issue where the Cell Count in the Usage Report could throw an integer overflow error.

Fixed a subsetting issue where Tonic displayed the error “Error fetching Subset preview” when users navigated to a workspace.

For subsetting, updated the Graph View display to make it easier to see the connections between the tables.

Improved messaging when Tonic cannot reach the database when you test a database connection.

Databricks

• On the workspace settings view, Test Cluster Connection no longer requires that you re-enter your API key.
• On the workspace settings view, the default job cluster specification now recommends a Databricks 14.0 Spark version.
• Databricks is now enabled for Professional and Enterprise users on Tonic Cloud, as well as for free trial users.
• Fixed source catalog workspace handling when the source catalog contains a table with a key constraint.
• Added a warning to the workspace settings view to prevent specifying the same source and destination locations.

File connector

• Fixed the validation used to prevent duplicate files in Amazon S3 file groups.

PostgreSQL

• Improved handling when the database does not contain a public schema.
• Improved subsetting performance for the Data Pipeline V2 data generation process.
• Improved performance for the Data Pipeline V2 data generation process.

SQL Server

• Fixed how Tonic handles system time periods.
• Improved messaging when Tonic cannot access SQL objects.

The new TONIC_ENABLE_SECURE_COOKIES setting indicates whether to enable the "Secure" attribute on Tonic authorization and analytics cookies. The default value is false. Do not set this to true if you access Tonic over an HTTP connection. When TONIC_HTTPS_ONLY is set to true, the “Secure’” attribute is always enabled on Tonic authorization and analytics cookies, and the value of TONIC_ENABLE_SECURE_COOKIES is ignored.

Updated to prevent simultaneous updates to the same workspace configuration.

For the Constant generator, fixed an issue for JSON columns where setting the constant to an empty string caused data generation to fail without setting the job status to failed.

The upsert pre-job check that validates the constraints on the intermediate and destination databases no longer fails when a database has constraints with duplicate signatures.

Fixed an issue where an empty upstream filter WHERE clause caused subsetting to fail if the schema changed so that the table was no longer upstream.

To use the API to obtain data encryption settings, the API user must now have the required global permission.

When users log out of Tonic, we now automatically invalidate any JSON Web Tokens (JWTs) that are not expired.

The API endpoint /api/permission-sets now requires the ManageUserAccess (Manage user access to Tonic and to any workspace) permission. Added a new endpoint /api/permission-sets/public, which returns the subset of the data needed by users who do not have that permission.

Amazon EMR

• Fixed an issue where when data generation was run from the SDK, some generators, including the Categorical generator, would not work.some text
  • You can now configure a Databricks workspace to write destination data to Databricks Delta tables.
  some text
  • On the file group details for CSV files, added configuration options to quote spaces and trim whitespace.
  • Fixed an issue where Tonic was not able to display a preview of extremely large files.
  some text
  • On the Collection View, the preview icon is now hidden for types that cannot be previewed.
  some text
  • For Data Pipeline V2 data generation, fixed an issue where we did not correctly truncate destination tables.
  • Fixed an upsert issue for tables that have generated identity columns.
  some text
  • For a Snowflake on AWS workspace, you can now provide specific AWS credentials for the file storage locations (S3 buckets or external stages).
  • The Snowflake data connectors are now available to Tonic Cloud users with a Professional or Enterprise license. They are not available to free trial users.
DatabricksFile connectorMongoDBPostgreSQLSnowflake

NOTE: v980 through v982 were removed.

New monthly pay-as-you-go plan on Tonic Cloud - Tonic now offers a pay-as-you-go subscription plan for Tonic Cloud. Free trial users are offered the option to use a credit card to purchase a pay-as-you-go license. The monthly subscription grants a Professional level license. With the pay-as-you-go plan, you can configure generators for up to 20 tables across all of your workspaces. Tonic bills you separately for each additional table that you configure. The license renews automatically each month. On Tonic Settings, a new Billing tab displays the next renewal date.

Data migration option for upsert - For upsert, Enterprise users can now connect a workspace to their own data migration script or tool to ensure that schema changes are automatically reflected in the intermediate database.

Other updates

Timestamp Shift is now the suggested generator for birthdate fields. Previously, the suggested generator was Random Timestamp.

Fixed an issue where editing workspace settings could cause you to be logged out.

File connector

• On Tonic Cloud, you can now use Amazon S3 as a source of files. Previously, Tonic Cloud only supported Google Cloud Storage and uploaded local files.
• For the Categorical generator, linked columns are now in the correct order.

Google BigQuery

• Tonic now supports arrays of a supported type. The STRUCT and INTERVAL types are still not supported.

MongoDB

• Fixed issue where a UI refresh was required in order to show automatic de-identification of foreign keys from de-identified primary keys.

Oracle

• The Download SqlLdr Files workspace permission is now assigned to the built-in Manager and Editor workspace permission sets.
• Downloaded sqlldr files no longer include .bad files.

PostgreSQL

• Fixed an upsert issue for tables that have generated identity columns.

SQL Server

• The Constant generator can now handle bit values.

New global permission to view organization users - A new global permission, View organization users, determines whether a user is able to see the lists of users and groups in the organization. This permission is required in order to use the Tonic application to grant access to and transfer ownership of a workspace, and to grant access to global permission sets. It is not required when you use the Tonic API to perform these tasks. The permission is granted to the built-in Admin, Admin (Environment), and General User permission sets. When you upgrade, Tonic automatically grants this permission to your custom global permission sets.

Other updates

On the workspace details view, added a new upsert processing option, Warn on Mismatched Constraints. When this is enabled, Tonic treats mismatched foreign key and unique constraints between the source and destination databases as warnings instead of errors, so that the upsert job does not fail.

Tonic now accepts all AWS RDS certificate authorities. Previously, we only accepted rds-ca-2019. The accepted certificates include:

• rds-ca-rsa2048-g1
• rds-ca-rsa4096-g1
• rds-ca-ecc384-g1

When job log recording (used to download job logs from the Tonic application) fails, it no longer creates a recording retry loop.

File connector

• Additional fixes to skip and log invalid rows instead of failing the data generation.
• Fixed an issue where when you added a file to an existing file group, and any column name contained a leading or trailing space, Tonic incorrectly displayed a schema mismatch error.
• You can now add .gzip files to a file group in a file connector workspace. The original file that was compressed must have the same format and structure as the other files in the file group. .gzip files are only supported for workspaces that use files from Amazon S3 or Google Cloud Storage. They are not supported in workspaces that use local files.

PostgreSQL

• During upsert, improved performance when de-conflicting unique constraints.

Create virtual foreign keys from Subsetting view - On Subsetting view, from a table details panel, you can now add a virtual foreign key to that table. To add a virtual foreign key, you select the foreign key column from the current table, then select the primary key column from the other table.

Other updates

Fixed an issue with TLSv1 and TLSv1.1 support in Tonic.

Improved performance of downstream processing during subsetting.

Fixed an issue where subsetting failed when a composite foreign key included a Boolean value.

On Table View:

• Added a column warning when the assigned generator fails to generate preview data.
• Increased the default width of the preview data column.

File connector

• Tonic now skips invalid CSV file rows and logs a warning instead of failing the entire file.
• Added a warning when the same source file is added to multiple file groups in the same workspace.
• Added JSON Mask and XML Mask as the recommended generators for JSON and XML files.

​​MongoDB

• You can now assign the Conditional and Null generators to binary fields.

MySQL

• Fixed an issue where Tonic did not clean up temporary file uploads after it wrote the data.

Oracle

• For a data generation job that ran SQL Loader (sqlldr), if sqlldr either failed or succeeded with errors, the job details include an option to download the sqlldr log files.

PostgreSQL

• Fixed an upsert issue where jobs failed with a unique constraint violation if the table contained a unique index but not a unique constraint.
• During upsert, improved performance when de-conflicting unique constraints.

Snowflake on Azure

• Fixed a bug that occurred when specifying the Azure Storage Account URL.

Removed the requirement that the authentication cookie goes over HTTPS (Secure Cookie). This fixed an issue where users could no longer log into Tonic over HTTP, but they could still log in over HTTPS.

Fixed an issue where users could not log out of Tonic from the email confirmation page.

Fixed an issue where upsert failed because of foreign key violations. Also improved upsert performance.

MongoDB

• When generator configurations are updated in single document view, Tonic now generates the preview data without re-fetching the data and refreshing the page.
• Fixed an issue that caused fields to disappear from hybrid view when the Null generator was applied.

In Table View, when a generator cannot be applied to a column in order to produce the preview data, the error message now includes the name of the column.

Expanded the table and collection dropdown lists to accommodate longer names.

The Privacy Report now marks a column as consistent when the generator is always consistent.

Fixed a migration issue with file connector files that were added before V920.

Fixed an issue where data generation could not run because of the permissions hierarchy.

Fixed a security issue related to JWT authentication.

Fix an issue where webhooks sometimes did not start when a job was canceled.

Improved error message when Tonic cannot display a date value.

PostgreSQL

• For the Tonic Data Pipeline V2 processing, Tonic now stops job execution after the initial error.
• Fixed an issue where check constraints failed to be applied in the destination database.
• Fixed an issue where views that depend on both a table and a view at the same time were not created in the destination database.
• Tonic no longer uses the TONIC_PAGE_PARALLELISM and TONIC_PARALLEL_READ_RANGES_TABLES environment variables for parallel processing.

SQL Server

• Fixed a caching issue that occurred when connecting to SQL Server.
• Improved error messaging when a view cannot be created.
• Improved the readability of SMO error messages.

You can now export individual topics from the Tonic documentation to PDF files. To export a topic to PDF, click the actions menu next to the topic title, then click Export as PDF.

Fixed an issue with removing unique constraint conflicts in upsert where rows that didn’t have a conflict were excluded from the upsert process.

File connector

• For Amazon S3 and Google Cloud Storage, the permission to list all buckets is no longer required. However, if that permission is not present, users must manually type in the bucket name where the file is located.
• When you copy a file connector workspace, Tonic now copies the file groups to the new workspace.

Upsert data generation (beta feature)

Previously, the data generation process always replaced the entire destination database. The new upsert data generation option (currently in beta) allows you to add new records and update existing records without touching any of the other records in the destination database. For example, you might have a regular set of records that you use for testing that you want to maintain.

Upsert requires a connection to an intermediate database. When you run data generation with upsert, the initial data generation writes the transformed data to the intermediate database. It replaces the intermediate database, similar to regular data generation. After the generation to the intermediate database, the upsert process identifies the records to add to or update in the destination database. It ignores other records in the destination database.

Upsert requires a Professional or Enterprise license, and is only supported for the following data connectors:

• MySQL
• Oracle
• PostgreSQL
• SQL Server

New AI-enhanced documentation search option

The Tonic documentation now provides access to Lens, an AI-based search option. Instead of searching for specific words, you can ask questions such as "How do I create a workspace?". Lens searches the documentation for the answer. It generates a response that includes links to the topics that contain the information it used.

To use the Lens search, click the search field. At the top right of the search panel, click Lens. Then ask your question.

Other updates

The Custom Categorical generator now supports consistency with other columns. Previously, the generator only supported self-consistency.

Tonic now prevents you from starting data generation for a workspace that does not have a destination database specified.

Fixed a display issue where the generator preset details panel briefly showed the occurrences for the previously selected generator preset.

Tonic now suggests the Name generator for columns that Tonic detects as containing names, when the detection is based on the sampled data in the column. By default, the Name generator uses the First Last format.

A new configuration option allows webhooks to bypass SSL certificate validation and trust the server certificate.

File connector

• You cannot start data generation for a file connector workspace that has no source files specified.
• The Table View data preview for file groups that contain JSON or XML files no longer displays above the real data records an extra row that contains the value 0.
• Fixed an issue where generators such as the Categorical generator unexpectedly could not be used as sub-generators.
• Improved error handling when a file group is incorrectly configured as having a header row.
• Fixed an issue that caused data generation to fail for XML files because of missing metadata.
• Fixed an issue where Database View displayed duplicate columns.
• Table View no longer displays an extra data row.

MongoDB

• Fixed an issue that caused unscanned collections to not display on Collection View.
• The sensitivity scan no longer marks Null fields as sensitive.

PostgreSQL

• Improved performance when refreshing materialized views in PostgreSQL v11, v12 and v13.

On the generator configuration panel, changed the label of the Save As menu to Preset Options. The menu contains options related to configuring generator presets.

Free trial users now have access to the file connector.

Tonic now displays the error that occurs when an Algebraic generator configuration does not include any floating-point values.

For composite generators, the generator preset details panel now provides a clearer explanation that presets for composite generators must be configured from within a workspace.

Improved performance for the Address generator and the HIPAA Address generator.

Amazon Redshift

• Fixed an issue with clearing temporary tables.

File connector

• Fixed how Tonic handles EOF characters.

MongoDB

• Corrected the order of the available generator presets for a field.

Snowflake

• Fixed an issue where data generation returned the error The specified bucket does not exist.

SQL Server

• Database connections can now use the MultiSubnetFailover option.
• Improved error messaging when a database cannot be created because of permissions issues.

Removed support for TIM - The Tonic Installation Manager (TIM) command-line tool to install and configure Tonic is no longer available.

Free trial users can now use a public email address to create the free trial account. Users with public email addresses cannot invite other users or share workspaces. Public accounts are only allowed for free trials.

Users on a Professional instance can now share the Manager workspace permission set with users and groups.

Improved error handling and validation messages for the foreign key file upload process.

Counts of generator preset occurrences no longer include occurrences in deleted workspaces.

On the bulk update panel in Database View, the consistency and differential privacy options now display correctly.

Fixed an issue where you could not select Passthrough as a sub-generator in a composite generator.

Fixed an issue where custom presets could not be deleted.

Fixed a display issue where long post-job action names overflowed into the next column.

Fixed an issue where you could not assign Random Timestamp as a sub-generator for the Conditional generator.

Fixed an issue where the generator configuration panel displayed the generator preset options when the user did not have the Manage generator presets global permission.

Fixed an issue where when a constraint failed to be applied, data generation failed.

Improved display when users who do not have the Manage generator presets permission try to display the Occurrences tab on the preset details panel.

Improved how we handle unavailable options for workspace actions in Workspaces view and in the Tonic navigation options.

For the Conditional generator, Tonic now correctly compares MySQL date values.

Databricks

• Tonic cluster initialization scripts are now uploaded as workspace files instead of DBFS files. The new, optional Workspace Path setting for Databricks workspaces controls the parent directory where Tonic uploads initialization scripts. The default value is /Shared.

File connector

• The file connector can now support .txt files that contain CSV, XML, or JSON content.
• Fixed an issue where Tonic incorrectly identified how a file connector file was encoded.
• Improved error messages when uploading files for the file connector.
• Fixed an issue when configuring a file group from Amazon S3 where users saw the error "Failed to fetch files from S3. The continuation token provided is incorrect." but could still see the list of files in the S3 bucket.
• Tonic now correctly updates the file configuration for file groups. Previously, users could not add files that did not match the default configuration.
• Tonic now displays an error when it is unable to read files from Amazon S3.
• The file explorer for Amazon S3 can now list the files in folders that have names that contain special characters.
• Improved encoding detection and file parsing.
• Tonic now correctly handles EOF characters in .csv files.
• Tonic now preserves the encoding of .csv files.

MongoDB

• Fixed an issue where the protection status information at the top of Privacy Hub did not update correctly after a new sensitivity scan.

Custom generator presets

Earlier this year, for Enterprise instances, we introduced the concept of generator presets. A generator preset is a saved configuration of a generator. You can assign generator presets to columns.

The initial release only included built-in generator presets, which allowed you to set the default configuration for Tonic generators.

This update in v924 introduces custom generator presets, which allow you to set up multiple configurations of the same generator. You can create custom generator presets from Generator Presets view. From a generator configuration panel, you can also save the current configuration as a new custom generator preset.

Generator preset occurrences

From Generator Presets view, you can see how often each preset was used in a workspace configuration.

The Occurrences column of the generator presets list shows:

• The number of times the baseline configuration was used
• The number of times the baseline configuration was overridden, meaning that a user selected the generator preset and then made a change to the generator configuration

On the generator preset details panel, the Occurrences tab displays both the number of occurrences and the specific workspaces and columns where the generator preset was used. You cannot see workspace and column details for workspaces that you do not have access to.

Other updates

Tonic can now integrate with GitHub for SSO authentication.

To manage generator presets, users must now have the Manage generator presets global permission. Previously, you could also manage generator presets if you had the Manager or Editor workspace permission set for any workspace.

Fixed an issue where the table data in Table View was not updated correctly when switching the table mode to or from Scale mode.

Improved performance for the Regex Mask and Conditional generators.

MongoDB

• Fixed an issue where the subsetting Graph View did not display virtual foreign key relationships.
• You can now add collections to a subsetting rule before the sensitivity scan completes.

PostgreSQL

• Fixed an issue where certain database constraints were not handled correctly, which resulted in job warnings about the failure to add those constraints.

Permissions and permission sets

As of v922, Tonic now uses permissions and permission sets to manage access to Tonic features and functions.

A permission controls access to a single feature or function. A permission set is a saved collection of permissions.Tonic provides built-in global and workspace permission sets. You cannot change the configuration of built-in permission sets. Enterprise instances can create custom permission sets.

Global permission sets contain global permissions, which control access to actions outside the context of a specific workspace. The built-in Admin global permission set grants access to all global permissions. Users and groups configured in the TONIC_ADMINISTRATORS environment variable are granted the Admin (Environment) global permission set, which also grants access to all global permissions. These global permission sets replace the previous admin user concept.

The built-in General User global permission set is assigned to all Tonic users, and grants access to create workspaces. You can also designate a different global permission set to assign to all Tonic users.

Workspace permission sets are assigned in the context of a specific workspace. They provide access to workspace permissions, which are associated with workspace management functions. The built-in workspace permission sets (Manager, Editor, Viewer Auditor) mirror the previous workspace roles. Similar to the previous Owner role, the Manager workspace permission set is granted access to all workspace permissions. However, unlike the Owner role, the Manager workspace permission set can be assigned to any user or group. You use the workspace sharing function to assign workspace permission sets within a workspace.

Each workspace has a single owner. The user who creates the workspace is the initial owner. All owners are by default granted the Manager workspace permission set. You can also designate a different workspace permission set to assign to workspace owners. You use the transfer ownership function to select a different owner for a workspace.

On Tonic Settings view, the User Management tab is replaced by the Access Management tab. From the Access Management tab, you can:

• View and manage the list of Tonic users
• If you use SSO, view a list of SSO groups
• View, configure, and manage access to global permission sets
• View and configure workspace permission sets

API endpoint to track user access and permissions

A new API endpoint to track the following events related to user access and permissions:

• A user account is created.
• A user account is removed.
• A user logs in to Tonic.
• A permission is added to or removed from a permission set.
• A permission set is assigned to or removed from a user. This might be a global permission set, or a workspace permission set in the context of a specific workspace.
• A generator preset is updated.

The endpoint is:

GET /api/audit-events/search

Other updates

You can now assign the Business Name generator as a sub-generator for the Regex Mask generator.

For subsetting, Graph View and the table details panel now display information about cycle breaks, when the subsetting process needs to break a circular dependency.

Databricks

• Fixed an issue that prevented the use of partition filters in Databricks Unity Catalog workspaces.

File connector

• When a file connector workspace is deleted, Tonic now deletes files that were uploaded from a local file system.

Spark

• Fixed an issue with data generation for workspaces that use Hive.

The Admin Panel is renamed to Tonic Settings.

Tonic now provides a more meaningful error when Preserve Destination mode is assigned to a table in a workspace that does not have a defined destination database.

Fixed an issue where Tonic opened too many connections to the application database.

Fixed an issue with timestamps in the Tonic API specification.

Fixed a Tonic Cloud issue where using a different email domain to update a Tonic license caused issues with Tonic logging.

Enhanced the performance of the HIPAA Address generator.

The Data Pipeline V2 data generation process now respects the TONIC_PROCESS_PARALLELISM environment variable.

Improved performance for subsetting, especially for data that contains a large number of foreign key relationships.Made a small performance improvement to primary key generators.

File connector

• Improved error messages when uploading files for the file connector.
• The file connector now supports extended ASCII-encoded files.
• Fixed an issue where the file preview omitted the first row of the file.

MongoDB

• On the Jobs view, you can now filter for the Collection Schema Scan job type.
• Fixed an issue where sensitivity scans failed on large collections.

Oracle

• Tonic now provides a more meaningful error when it loads Table View for a table that the database account does not have access to query data from.

PostgreSQL

• Corrected the job history entries for subsetting jobs that run using the Data Pipeline V2 process.

Spark

• Fixed an issue that caused jobs to fail when an invalid Repartition or Coalesce value was specified.

NOTE: Releases v899 through v901 were removed.

Tonic Data Pipeline V2 for PostgreSQL ends beta

During the first half of 2023, Tonic has run a beta program for PostgreSQL for our new Data Pipeline V2. The beta program is now ending. Thank you to all of those who provided feedback.

Starting with version V905, Tonic.ai will progressively enable Data Pipeline V2 for all customers. To ensure a smooth transition for all our PostgreSQL customers, Tonic.ai controls the rollout remotely.

The remote rollout mechanism is controlled by an HTTPS request from your instance to https://feature-flag.tonic.ai. A JSON payload with a unique identifier for your deployment is sent, and the status of Data Pipeline V2 is returned. This request happens at the start of each data generation. If your Tonic server cannot reach https://feature-flag.tonic.ai, then the check is skipped.

What to expect for the enrollment:

• Before your instance is enrolled in Data Pipeline V2, Tonic Customer Success will contact you to confirm your enrollment.
• After your instance is enrolled, the Data Pipeline V2 toggle on the Confirm Generation panel is removed. All PostgreSQL jobs run using Data Pipeline V2.

For jobs that run on V2:

• The job type is Data Pipeline Generation instead of Data Generation.
• Jobs should run faster. Data Pipeline V2 has better resource handling, and can provide more parallel execution, especially for large tables and to apply constraints. Not all jobs will be faster, but for some jobs there should be a significant improvement.

We will continue to improve Data Pipeline V2 as we expand coverage to other data connectors.

Expanded Graph View for subsetting

The subsetting Graph View is expanded to use more of the available screen space. The Configure Subset panel, which includes the Options and Latest Results tabs, no longer displays on Graph View. It only displays on Table View.

Other updates

Fixed an issue where when a data generation job failed, tables that used Preserve Destination table mode were not restored.

The generated Tonic API documentation now includes the endpoints for managing file groups for file connector workspaces.

Fixed an issue that caused jobs for some workspaces to fail with the exception "Cannot modify workspace whose schema is not the latest version.".

Fixed an issue where the job details view displayed incorrect information.

Updated the /api/DataSource endpoint to not contain secure data such as the API key.

Updated our OpenAPI documents to ensure that all values of operationId are unique.

Improved error messages for failed data generation.

Databricks

• The Test Connection button on the workspace details view now works correctly.

MongoDB

• Fixed an issue where schema scans time out even when a timeout is not configured.
• Fixed an issue where MongoDB workspaces did not support MongoDB versions below v4.4.

MySQL

• Tonic data generation now supports generated columns in de-identified tables.

Oracle

• To improve the resiliency of Oracle commands during data generation, fixed the retry logic.

PostgreSQL

• Tonic no longer waits to process a job cancellation until after it finishes the constraint application that was in progress when the job was canceled.
• Fixed an issue where Tonic returned an "insufficient space" exception when writing numeric values to a destination database.

Snowflake

• Improved performance for Snowflake on Azure.

Spark

• Job tracking URLs now display correctly on the job details page.

SQL Server

• Fixed an issue where Tonic did not retrieve all of the compound keys from the source data.

NOTE: Releases v897 and v898 were removed.

On Subsetting view, Graph View now displays a loading animation as new data is loaded.

Improved performance for the UUID Key and Integer Key generators.

File connector

• Fixed an issue where files sometimes did not upload completely.

Google BigQuery

• Improved performance of destination database writes during data generation.

MySQL

• Improved performance for destination database writes during data generation.

Oracle

• Improved error handling when the tablespace in the source database is missing from the destination database.

File connector

The new file connector data connector allows you to use files from either Amazon S3, Google Cloud Storage, or a local file system as the source data. The file connector supports .csv, .json, and .xml files. Within a file connector workspace, you create file groups. Each file group contains files that have an identical format and structure. A file group is treated as a table for the purposes of table mode and generator configuration. The file connector is available with the Professional and Enterprise license plans.

Generic O​​IDC SSO Support

Tonic now supports authentication using a generic OpenID connection.

Tonic API versioning

We have introduced a versioning scheme for the Tonic API. API versions are released more or less quarterly, with the version identifier in the format vYYYY.MM.P (Year.Month.Patch). The current release candidate (v.RC) contains API updates in progress.

You should now specify an API version in your API requests. The System Status tab of the Admin Panel lists the latest available version. You can also select the version to use when you do not provide a version in the request. If you do not provide an API version in a request or select a default API version, then until January 31, 2024, Tonic automatically uses the latest version. After January 31, 2024, Tonic will return an error from the request.

Other updates

Fixed an issue where Tonic incorrectly returned the error No Destination DB has been configured for this Workspace for workspaces that used Preserve Destination.

For subsetting Graph View, updated the default zoom level to allow users to see more of the graph.

The Keycloak SSO provider now supports PKCE challenge.

Fixed an issue where deep links did not work for SAML SSO.

MongoDB

• For Mongo queries, Tonic now can now use disk as well as memory.

Oracle

• Tonic now refreshes materialized views even when SKIP_CREATE_DB is set to true.

PostgreSQL

• In the job progress steps, fixed an issue that caused the number of rows in a table to display as -1.
• When the Data Pipeline V2 processing is enabled, tables are now processed by size, with larger tables processed first.

SQL Server

• Fixed support for Kerberos authentication.

Spark

• Added support for the UUID Key generator to Livy and Databricks workspaces that use Spark 2.3.x and above.
• Added support for the UUID Key generator to the Tonic Java SDK.

The ASCII Key generator now includes an Exclude Lowercase Alphabet option to exclude lowercase letters from the destination values.

Fixed an issue that prevented free trial signups.

Data generation no longer fails when Tonic is unable to retrieve the destination database size.

Updated the FNR generator to prevent a possible leakage of PII.

Added a Date column to the usage report. The date column provides the date and time when the data generation job was completed.

Fixed an issue in the JSON Mask generator where it incorrectly changed the format of timestamps.

Subsetting is no longer prevented when a table that is not in the subset is assigned Preserve Destination mode.

Databricks

• On the workspace details view, you can now optionally specify the catalog where the source database is located. If you do not specify a catalog, then the default catalog is used.

MongoDB

• Fixed an issue where the collection statistics failed because the statistics object became too large.
• Updated to write collection records in batches.
• Tonic now continues to retrieve documents after a failure.

Oracle

• Removed the uniqueness check for individual columns that are part of a composite index.

PostgreSQL

• Fixed an issue where there was duplicated data from parent tables of inherited tables.
• Improved performance for query to retrieve tables and columns.

Snowflake

• Improved performance for data generation.
• Updated how Tonic uploads files to Amazon S3 to reduce memory usage.

SQL Server

• Fixed DNS resolution for Kerberos.
• Fixed an issue where Kerberos authentication failed with an error that the destination array was not long enough.

The new usage report summarizes the data processed for each table for data generation jobs. The report is a .csv file that you download from Tonic. To download the report, on the Admin Panel, click Download Usage Report.

When certain sensitive loggers are enabled, Tonic now disables log collection.

Fixed an issue on Database View where a column configuration panel would close unexpectedly.

For the TONIC_ADMINISTRATORS environment variable, you can now specify the names of SSO groups to grant administrator privileges to. Previously you could only specify user email addresses.

The Tonic SDK Javadoc now displays correctly.

Restored the ability to import a workspace configuration from Workspaces view.

MongoDB

• Added support for the DBRef datatype.
• Improved performance for collection management.

PostgreSQL

• For the beta Data Pipeline V2 data generation, adjusted the logging level for telemetry-related log messages to DEBUG.
• For the beta Data Pipeline v2 processing, improved parallel processing for constraints that cross tables.
• For the beta Data Pipeline V2 process, reduced the default maximum number of source and destination connections to 8. These are set as the values of the TONIC_JOBFLOW_MAX_SOURCE_CONNECTIONS and TONIC_JOBFLOW_MAX_DESTINATION_CONNECTIONS environment variables. We recommend that you set each value to the number of CPUs on the corresponding database.

Snowflake

• Fixed an issue where preserve destination tables were removed during data generation.

The new FNR generator transforms Norwegian national identity numbers. The FNR generator was added in V857. It included options to specify a range of birthdates and preserve the indicated gender. In V866, removed the date range configuration options. The destination values are now always within the same date range as the source values. The FNR generator also now can be used for columns that have uniqueness constraints. The final digits in the destination value are not a valid checksum.

For the beta Data Pipeline V2 processing, fixed an issue where jobs would hang if they were canceled before the job started.

Fixed an issue where the Foreign Keys view would freeze.

Fixed an issue where when you typed @ to add a user mention to a comment, suggestions for the user did not display.Upgraded to use .NET 7.

"Data science modeling" is changed to "data science mode".

When you configure the SSH tunnel settings for a workspace, Tonic now obscures the SSH passphrase.

MongoDB

• Added a configuration to prevent Tonic from retrieving other information about collections when retrieving a collection list. Addresses an issue where retrieving the collection list took a very long time. To disable the retrieval of other collection information, set the environment variable TONIC_MONGO_DISABLE_COLLECTION_INFORMATION_FETCHING to false.
• Fixed an issue with collection scanning that caused application pages to not load.
• Collection scans are now able to continue when the scan for an individual collection fails. The job logs include warnings for each failed collection.
• For collection scans, for each schema Tonic now limits the number of documents to scan and the length of time for the scan.

PostgreSQL

• Tonic now uses the estimated row count from PostgreSQL statistics to determine the parallelism for a table. Customers should ensure that they have up-to-date statistics for their source tables, especially for large ones.

Snowflake

• For source tables that are assigned Preserve Destination mode, Tonic no longer attempts to add existing constraints to the destination tables.
• Fixed a syntax error in the post-generation processing.
• Fixed an issue where data generation failed with the error "Unable to determine AWS Region".
• Fixed an issue where preserve destination tables were removed during data generation.

SQL Server

• During data generation, Tonic now warns users when a filegroup that exists in the source database does not exist in the destination database.

NOTE: Releases v849 through v854 are removed.

Removed caching of AWS credentials.

For the beta Data Pipeline V2 job processing (available for PostgreSQL only):

• When a job fails, Tonic no longer tries to fall back to the current job processing. In most cases, jobs fail for reasons that are not connected to the processing type. Falling back to the current job processing is not effective.
• Improved performance for subsetting.
• Adjusted the logging level for telemetry-related log messages to DEBUG.

On the Foreign Keys view, when you filter the keys, click the select all option, and then clear the filter, only the matching keys are selected. Previously, the select all option always selected all of the keys.

Fixed an issue where CSV files could not be uploaded to data science mode workspaces.

You can now configure parallelism for sensitivity scans. For relational databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_RDMBS, and the default is 4. For document-based databases, you use the environment variable TONIC_PII_SCAN_PARALLELISM_DOCUMENTDB, and the default is 1.

On the table configuration panel for subsetting, Tonic no longer displays a count of post-subset rows before a subset is generated.

Fixed an issue where subsetting data from one workspace appeared in a different workspace.

You can now use UUID columns in the conditions for the Conditional generator.

Fixed an issue where when you deleted a linked column from a column configuration panel, the other linked columns were deleted.

The Timestamp Shift generator can now be assigned to columns where the values use the date format MMddyyyy.

Fixed a regression that caused NPGSQL logging to occur even when it was disabled.

Google BigQuery

• Improved error handling when rows contain invalid data. Tonic now provides a method to look at the data that caused the error. Fixed the handling of rows to prevent errors on certain data.
• Fixed how we create views in the destination database.

MongoDB

• Fixed an issue where Tonic was unable to get the schema for the source data.

Oracle

• Fixed an issue where retries of Oracle commands from transient errors failed.

Snowflake

• Improved resource handling during data generation in order to enable parallel processing.
• Snowflake on Azure no longer requires CREATE SCHEMA permissions in order to support Preserve Destination mode for tables. Snowflake on AWS with Lambda processing continues to require CREATE SCHEMA permissions in order to support Preserve Destination mode.
• When Tonic is unable to create a view in the destination database, it now returns a warning instead of an error.
• Implemented a more accurate method to detect hexadecimal values.

SQL Server

• Tonic now displays the correct generators for columns that are part of a composite unique index.

The new Business Name generator produces realistic names of businesses or companies. The Business Name generator can be consistent with itself or with other columns. It improves on and is intended to replace the Company Name generator, which is now deprecated.

Fixed an issue on Table View where users could not use the delete icon to remove a generator assignment for a linked column.

Fixed an issue where the job details view for a subsetting job did not always show all of the steps as completed.Updated the version of pytorch, which is used for data science mode. This new version addresses some security vulnerabilities.

Fixed an issue where when the Tonic server was air gapped, the Admin Panel did not correctly display the current Tonic version.

Fixed an issue where jobs took longer than expected to complete.

Fixed an issue where the subsetting Graph View did not show how table participation in the subset changed since the most recent subsetting data generation.

Fixed an issue where after a single failure to write logs, the Download Job Logs feature stopped refreshing the logs. Tonic now continues to try to upload logs.

Google BigQuery

• Fixed an issue with the test connection function for the destination database.
• Materialized views and routines from the source database are now copied to the destination database.

MongoDB

• Improved performance for sensitivity scans.

MySQL

• Fixed an issue where subsetting failed because of a data type mismatch between a primary key and a foreign key.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB = true, foreign keys are now correctly enabled on the destination database.

PostgreSQL

• Fixed an issue where the source database permissions check provided a false error about insufficient privileges for sequences.
• For the beta Data Pipeline V2 data generation process, fixed an issue where the data generation process continued even after the job failed.

Snowflake

• Fixed an issue where the presence of comments caused data generation to fail.
• Improved performance for sensitivity scans.

SQL Server

• Added support for user-defined types.

Generator presets are now supported for Enterprise licenses on Tonic Cloud.

For Tonic data encryption, fixed an issue where the previous encryption key environment variable value was saved in the application database, which caused Tonic to use those values even after they were removed.

The Tonic diagnostic logs now include the Tonic worker ID.

Fixed an issue where the Tonic web server would not launch unless the Tonic application database used PostgreSQL v13 or later.

For data connectors other than MongoDB, the sensitivity scan is now parallelized.

Google BigQuery

• Data generation now works correctly when the region that hosts Google BigQuery for the destination database is different from the region for the source database.
• Fixed an issue where failed data generation jobs were incorrectly reported as successful.
• Tonic now handles the TIME data type correctly.

MySQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.

Oracle

• When TONIC_ORACLE_SKIP_CREATE_DB=true, fixed an issue where the truncation of tables violated foreign key dependencies, which caused jobs to fail.
• Added an option to enable the TCPS protocol for Oracle database connections. Previously, only TCP was supported. If you enable TCPS, you must also provide a wallet file.
• Tonic now cleans up temporary destination database tables that were created during subsetting data generation.

PostgreSQL

• For Incremental mode, fixed an issue where the values of timestamp columns on modified rows were not updated in the destination from the source.
• During data generation, Tonic now warns users when an extension that the destination database needs is unavailable for installation.

Snowflake

• For both Snowflake on AWS and Snowflake on Azure, you can now configure workspaces to limit the schemas to include.

Fixed an issue where users could only select generators that supported uniqueness constraints for columns that were not unique.

Fixed an issue where admin users who did not have edit permissions on any workspaces could not edit presets from the Generator Presets view.

Improved data generation resiliency against transient failures.

Removed erroneous error messages.

To add AWS credentials to containers, you can now mount to ~/.aws/credentials.

Improved error messaging for Table View.

Fixed a display issue where the column configuration panel was too narrow and required horizontal scrolling.

Exporting or copying a workspace no longer requires the workspace to have a valid source database connection.

Reduced the amount of memory needed to run the Tonic web server.

MongoDB

• Better handling of errors that involve invalid UUIDs.

Oracle

• Updated the required permissions for destination database connections. If SELECT ANY DICTIONARY or SELECT_CATALOG_ROLE cannot be granted, then Tonic can use a selection of ALL_ views (not recommended).
• If TONIC_ORACLE_SKIP_CREATE_DB=true, then external tables are now excluded from the table list in Tonic. Tonic does not process those tables.

PostgreSQL

• Fixed an issue where the Data Pipeline V2 flow would hang.
• Fixed an issue where extensions such as pgcrypto were not transferred when data generation included schema filtering.
• Improved performance when handling constraints.

Snowflake on AWS

• As of V823, you can choose whether to use the Lambda process for data generation, which was previously the only option. By default, Snowflake on AWS uses a new, more resilient data generation process. You only need to use the Lambda data generation process for extremely large volumes of data (hundreds of gigabytes to terabytes).
  
  For existing workspaces, for versions before V826, the new default process is used. To use the Lambda data generation process, you must update your workspace configuration. As of V826, existing workspaces use the Lambda data generation process.
• For the temporary CSV files used to retrieve and write source and destination data, you can now specify to use an external stage instead of an S3 bucket. The option to use an external stage is not available when you use the Lambda data generation process.
• You can now specify different file storage locations for the temporary source and destination data files. In other words, you can have different S3 buckets or different external stages. Note that this option is not available when you use the Lambda data generation process.
• For the new data generation process, fixed an issue where data generation jobs would hang instead of failing.

Snowflake on Azure

• Before it runs a data generation, Tonic now verifies that there is a valid value for the Azure Blob Storage account key, which is set as the value of the environment variable TONIC_AZURE_BLOB_STORAGE_ACCOUNT_KEY.
• Fixed an issue where data generation jobs would hang instead of failing.

For Tonic data encryption, Tonic now only verifies the key for the enabled process. If you only enable decryption, then Tonic only verifies the value of TONIC_DATA_DECRYPTION_KEY. If you only enable encryption, then Tonic only verifies the value of TONIC_DATA_ENCRYPTION_KEY.

Upgraded our Docker images from Ubuntu 20 to Ubuntu 22.04.

Updated to ensure that the Tonic URL reflects the currently active workspace.

Recently started jobs no longer display a start time that occurred several years ago.

On the Data Encryption tab, the option to provide custom initialization vectors is now a toggle instead of radio buttons.

Resolved an issue where Tonic took an extremely long time to load.

Oracle

• Reduced the permissions required to test database connections.
• Changed the required permissions to better support when TONIC_ORACLE_DBLINK_ENABLED is false. For the source database user, you can either grant SELECT ANY DICTIONARY, grant SELECT_CATALOG_RULE, or (not recommended) grant access to the ALL_* views.

PostgreSQL

• Improved the error messaging when testing the connection to the destination database.

Snowflake

• You can now use a connection string to connect to the source and destination databases. Also added support for proxy connections.

NOTE: v805 and v806 are removed.

The new Snowflake on Azure data connector uses Azure Blob Storage to store interim uploaded and generated files.

A copied workspace now includes manual sensitivity designations. A manual sensitivity designation is when you change the sensitivity designation that was assigned by the sensitivity scan to either sensitive or not sensitive.

When the configured encryption or decryption key is not valid - for example, the key is not configured or uses the incorrect size - then Tonic does not allow you to configure Tonic data encryption.

Tonic Cloud now correctly enforces the supported data connectors.

Improved error messaging when subsetting data generation fails because the generator cannot be used with subsetting.When you change the type of Tonic data encryption (decryption, encryption, or both), Tonic no longer clears the decryption and encryption text fields.

MongoDB

• Fixed an issue where workspaces that contained views did not load.

Oracle

• Fixed an issue where the default Oracle NUMBER type was not compatible with the Integer Key Generator.
• Removed an invalid error that was returned when users tested data connections.

PostgreSQL

• For the beta Data Pipeline V2 generation process, improved the error logic to prevent jobs from hanging when errors occur.

SQL Server

• Improved the resilience of data generation to transient failures.