Sending telemetry to Tonic.ai is now required and enforced. After 5 days of failed telemetry connections, a warning displays. If the connection issue persists for 15 days, data generation is restricted.
Structural Cloud users can now reset their passwords from the login page. To reset your password, click Forgot your password?.
You can now set the TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS
environment setting from the Environment Settings tab on Tonic Settings.
File connector
SQL Server
From the recommended generators panel on Privacy Hub, you can now enable or disable self-consistency for each column.
When you create a virtual foreign key, the nullability of the foreign key is now determined by the nullability of the foreign key column.
On Privacy Hub and the job details view, changed the label on the Download option to Reports and Logs.
Db2 for LUW
MySQL
PostgreSQL
Added a new API endpoint to resolve all schema changes in a workspace. You can choose whether to resolve only conflicting changes, only notifications, or all of the schema changes.
Fixed an issue that caused the Tonic Structural PyML Service to be unreachable in IPV4-only containers.
Added a new conflicting schema change when a column that has an assigned generator becomes a foreign key. Foreign key columns must inherit the generator from the primary key.
Structural can now generate data with subsetting when a primary key table is truncated, as long as the foreign keys that reference the primary key are nullable.
Amazon Redshift
File connector
Snowflake
SQL Server
For the notifications image, replaced alpine with ubuntu.
File connector
MongoDB
MySQL
Snowflake
SQL Server
During the free trial, Structural now displays next step highlights to indicate the next recommended action. When you hover over the recommended action, Structural displays an explanatory tooltip.
A new environment setting, TONIC_DB_MAX_POOL_SIZE
, sets the connection pool size for the Structural application database. The default value is 3.
Fixed an issue where the preview data in the JSON Mask generator editor did not respect the applied table filter.
File connector
MongoDB
PostgreSQL
Writing output to a Tonic Ephemeral snapshot - For database types that Tonic Ephemeral supports (currently PostgreSQL and MySQL), you can now write the output to an Ephemeral user snapshot. This replaces the option to write the output to an Ephemeral database, except for workspaces in the Structural free trial. In Ephemeral, you can use the user snapshot to start new Ephemeral databases.
Other updates
For the UUID Key generator, added a new configuration option, Preserve Version and Variant. By default, the setting is turned off. When turned on, the version and variant bits from the source UUID are preserved in the output value. For the API, the new setting is preserveVersionAndVariant
.
In the Tonic Structural free trial, the sample workspace now by default writes the output to a Tonic Ephemeral database.
Fixed an issue where vertical scrolling was sometimes blocked.
You can now configure the allowed SSL/TLS protocols and ciphers on the Tonic Web Server. To configure the protocols and ciphers, use the environment settings TONIC_WEB_SERVER_TLS_PROTOCOLS
and TONIC_WEB_SERVER_TLS_CIPHERS
.
File Connector
MongoDB
MySQL
Fixed an issue in Table View where characters were sometimes represented inaccurately. For example, a lowercase x would become a multiplication symbol.
Fixed an issue where data generation to Tonic Ephemeral Cloud failed with the error "Ephemeral URL not found".
For a schema change that adds a new column, both the Schema Changes view and the API response now include the data type for the new column.
For the Structural free trial, Structural now displays a checklist for each workspace. There are slightly different checklists for database-based and file connector workspaces.
Added support to run Structural in dual-stack networks and IPv6-only network environments.
On the workspace details view, fixed an issue where an Ephemeral API key appeared to be populated when no value was provided.
Fixed an issue that caused data generation to Ephemeral to fail with "Ephemeral output must be configured".
Fixed an issue where data generation to Ephemeral failed after a first successful run.
Amazon Redshift
MySQL
SQL Server
The Structural API now includes endpoints to get and set the assigned table modes and table filters for a workspace.
Fixed an issue where the workspace audit trail displayed generator preset events that occurred before the workspace was created.
Fixed an issue where an error was returned when users tried to export selected files from a file group.
Improved error message when Structural cannot write output to Ephemeral because Ephemeral does not have a compatible base image for the database.
Databricks
Google BigQuery
Oracle
Output to a Tonic Ephemeral database - Tonic Ephemeral is a separate Tonic.ai product that allows you to create temporary databases. On Tonic Cloud, for data connectors that Ephemeral supports (currently PostgreSQL and MySQL), you can configure the workspace to write the destination data to an Ephemeral database. This is the default option for data connectors that Ephemeral supports.
The database belongs to your Ephemeral account. If you do not already have an Ephemeral account, then Tonic automatically creates a two-week Ephemeral free trial account for you. The Tonic data generation job details provide access to the database connection details.
Free trial checklist - During the free trial, the sample workspace now includes a checklist to help users get through the required steps to complete their first data generation.
Other updates
Free trial users can no longer use a public email address to create an account.
Fixed an issue where password reset links lead to a blank page.
Fixed an issue where pay-as-you-go users would see the countdown for a free trial.
In the sample workspace, fixed an issue where a faulty destination database template caused an error when a user tried to update it.
Google BigQuery
Oracle
PostgreSQL
Snowflake
You can now manually add selected environment settings to the Environment Settings list on Tonic Settings.
Improved the performance of data previews in the Tonic application.
For workspaces that write output data to a container repository, fixed an issue that prevented GAR credentials from being saved.
Google BigQuery
Snowflake
SQL Server
Privacy Report PDF file
We added a new Privacy Report PDF that you can download from Privacy Hub and the job details view. The Privacy Report PDF contains a summary of the privacy ranking values, visualizations to summarize the workspace column privacy rankings based on the applied generators, and a summary table that contains the .csv Privacy Report data.
To accommodate the new file, on Privacy Hub and the job details page, the available downloads are combined into a Download menu.
Assigning recommended generators from Database View
On Database View, when an unprotected column has a recommended generator, the generator name tag now displays the type of sensitive data that was detected.
When you click the generator name tag, Tonic displays a panel that displays the sensitivity type, the recommended generator, and sample source and output data based on the recommended generator. The panel provides options to either apply or ignore the recommendation.
Other updates
Fixed an issue where changing the configuration of a generator preset did not accurately update the count of occurrences of the preset.
Oracle
New Db2 for LUW data connector - Tonic now has a data connector for IBM Db2 for Linux, Unix, and Windows (Db2 for LUW). Tonic supports Db2 for LUW version 11.5.
Other updates
When the AI Synthesizer is used in a workspace, Tonic now verifies before data generation that the AI Synthesizer does not use more than the maximum allowed categories.
Amazon EMR
<view name>_tonic_table
.Fixed an issue with the Name generator where capitalization was not preserved if consistency was disabled.
For Table View, fixed an issue where the delete button to remove the generator assignment was sometimes hidden.
Oracle
PostgreSQL
SQL Server
Redesigned Database View
We redesigned Database View to improve the display and the filtering.
In the updated columns list, the Column column contains the schema, table, and column name, and the column data type. It provides access to the data preview option.
The Applied Generator column shows the applied generator. Applied Generator indicates when a column is unprotected, when the column is a primary or foreign key, and when the configuration overrides the parent workspace. If the table mode is not De-Identify, it shows the table mode. It provides access to the commenting option.
Filters other than the column name filter are moved under the Filters option. There are also new filters for the sensitivity type (the type of sensitive data that Tonic detected in the column) and whether the column has a recommended generator.
Privacy Report updates
In the Privacy Report, new column, Column Privacy Rank, indicates the privacy ranking for a column based on the assigned generator and generator configuration. The generator summary and generator reference include the possible privacy ranking values for each generator.
Added a new column, Tonic Detected Sensitivity, that indicates whether the Tonic sensitivity scan identified the column as sensitive. Renamed the Is Sensitive column to Current Sensitivity. Current Sensitivity indicates whether the column is currently marked as sensitive.
Also corrected an earlier issue with the order of the columns.
Other updates
Fixed an issue that caused all subset runs to record the percentage of rows in the subset as 100%. Subset runs that occur after updating to this version display the correct percentage.
The option to write output data to a container repository is out of beta.
Databricks
Google BigQuery
TONIC_GRPC_ENABLED
was false.NUMERIC
or BIGNUMERIC
column.Oracle
IDENTITY
columns. Before this change, IDENTITYM
columns caused errors during destination database creation.For the Custom Categorical generator, you can now add a NULL value to the available custom category values. To indicate a NULL value, use the keyword {NULL}
.
Made the following API updates to better accommodate users of the previous version of the API:
jobs/{id}/workspace_snapshot
now returns the WorkspaceDataModel object.GET jobs/{id}/workspace_snapshot?api-version=v2023_07_00
, that returns V17WorkspaceDataModel
Databricks
TBLPROPERTIES
from the source delta table, including 'delta.feature.allowColumnDefaults'
.Redesigned data model for generator assignments - The new version of the Tonic API includes a redesign of the data model for generator assignments. To use the previous version of the generator assignment data model, make sure that your API calls specify version 2023.07.0.
The generator assignment data model redesign includes the following changes:
metadata
object in the link
object:presetId
generatorId
customValueProcessor
encryptionProcessor
pathExpression
to the metadata
object in the link
object.link
object:subPresetId
subGeneratorId
customSubGeneratorValueProcessor
subGeneratorMetadata
object under metadata:presetId
generatorId
customValueProcessor
TONIC_GRPC_ENABLED
was set to true.NOTE: v1074 was removed.
If your instance of Tonic is deployed on Docker, you can now use an external Kubernetes cluster to enable the option to write destination data to container artifacts.
You can now assign the Integer Key generator to a column with a decimal data type. The actual column values must still be integers.
Fixed an issue in Table View where an error displayed if you changed the selected table while the data was loading.
Databricks
File connector
SQL Server
NOTE: v1051 through v1053 were removed.
Enable administration functions in Tonic Cloud - For Tonic Cloud customers, the new Account Admin permission set provides access to Tonic administration functions for their organization. The Account Admin can reset passwords, delete users, copy and share all workspaces, and download the usage report. The Account Admin permission set is initially granted to the first user in the organization.
Databricks
File connector
MySQL
TONIC_MYSQL_MAX_CONCURRENT_INDEX_CREATION
, to limit the number of concurrent indexes that are created. The default value is 0, which indicates that there is no limit.SQL Server
The Enable Diagnostic Logging global permission is now granted to the built-in Account Admin permission set.
Databricks
CREATE CATALOG
or CREATE SCHEMA
permissions are no longer required if the destination catalog or schema already exists.Diagnostic logging for data generation - By default, Tonic now redacts sensitive data in data generation log files.
When users start a data generation or upsert job, if they have the new global permission Enable diagnostic logging, they can choose to enable diagnostic logging, which does not redact the logs. The Enable diagnostic logging permission is also required to download the diagnostic logs. By default, the permission is only granted to the Admin and Admin (Environment) global permission sets.
In addition to the option for individual jobs, there are environment settings that enable diagnostic logging for specific data connectors.
Other updates
In the Release Candidate version of the API, the response model for the GET /api/workspace/minimal
endpoint has been updated for more straightforward de-serialization.
Fixed an issue where a non-unique composite primary key column could only be assigned unique generators.
Users can now press Enter to finish copying a workspace or a generator preset, instead of having to click Copy.
File connector
Google BigQuery
Oracle
SQL Server
WITH INLINE
clauses from definitions of user-defined functions (UDF). Inlining does not require these clauses. WITH_INLINE
clauses in UDF definitions that do not meet the requirements for inlining can prevent the UDF from being restored properly in the destination database.For the OpenID Connect (OIDC) SSO integration, Tonic now supports authentication by client secret that uses HTTP basic authentication (client_secret_basic
). To provide the client secret, configure the TONIC_SSO_CLIENT_SECRET
environment setting.
SQL Server
TONIC_SQL_SERVER_SKIP_CREATE_DB
, indicates whether to skip schema creation for the destination database. If true, then Tonic does not create the schema. It uses the existing schema to populate the destination database. The default is false. You can configure this environment setting from the Environment Settings tab on Tonic Settings.NOTE: These releases were removed.
During free trial signup, the data connector options now include an option to use local files for the source data. This creates a file connector workspace for local files, and displays the File Groups view to allow the free trial user to start to add file groups to the workspace.
Added an environment setting, Tonic Test Connection Timeout In Seconds (TONIC_TEST_CONNECTION_TIMEOUT_IN_SECONDS
), that you can set from the Environment Settings tab on Tonic Settings. This setting configures the timeout for testing a database connection. Previously, connection test attempts timed out after 5 seconds. The new default is 15 seconds.
When you configure a workspace to write the output to container artifacts, you can now specify custom resources for the Kubernetes pod, including the ephemeral storage, memory, and CPU millicores.
Improved performance when marking a large number of columns as not sensitive.
Fixed an issue that caused Tonic workers that are deployed on Docker to crash unexpectedly.
For numeric columns that support arbitrary precision and scale, when the scale is 0 (for example, NUMERIC(N,0)
), or when the underlying values are all integers, these columns are now supported as primary keys for the purpose of subsetting.
Amazon EMR
Amazon EMR and Databricks
TONIC_WORKSPACE_DEFAULT_SAVE_MODE
indicates the mode to use. If set to a value other than null (Ignore, Append, Overwrite), this setting takes precedence over TONIC_WORKSPACE_DEFAULT_ERROR_ON_OVERRIDE
.Google BigQuery
MongoDB
TONIC_DOCUMENT_MAX_DEPTH
, to configure the maximum depth of JSON document that can be handled. The default value, which is also the recommended minimum value, is 32.SQL Server
When you select the option to write destination data to container artifacts, you can now use Google Artifact Registry (GAR) authorization using Google Cloud Platform (GCP) service account keys.
For the JSON Mask and XML Mask generators, fixed the data preview for JSON or XML field samples that are larger than 120MB by generating a smaller subset of the field.
The Name generator now supports consistency with other columns.
Added new API endpoints to retrieve and set table replacements. These new endpoints are compatible with workspaces for data connectors that do not have schemas, such as Spark-based databases and the file connector. The existing endpoints, which require you to provide a schema, eventually will be deprecated.
Amazon EMR
File connector
MySQL
Oracle
PostgreSQL
SQL Server
Snowflake
Added an environment setting TONIC_DELETE_COLUMN_SCHEMA_ON_WORKSPACE_DELETE
. If the setting is true
, then when a workspace is deleted, Tonic also deletes the associated rows from the ColumnSchemas
table in the Tonic application database.
The new environment setting TONIC_NOTIFICATION_SMTP_TRUST_CERTIFICATE
indicates whether to allow the SMTP server certificate to be trusted.
Improved the performance of previewing data in Privacy Hub.
Fixed an issue where SSO groups were not removed when the value of TONIC_SSO_GROUP_FILTER_REGEX
changed in a way that excluded previously imported groups. The removed groups are removed from any workspaces that they were granted access to.
For the Timestamp Shift Generator, added Month and Year as options for the date part to use to set the allowed range.
When writing data to container artifacts, Tonic now first shuts down the temporary database before it begins to write data to the container.
Amazon EMR
Databricks
MongoDB
Oracle
TONIC_ORACLE_DBLINK_ENABLED
is false
), privileges were not copied from the source to the destination.Amazon EMR
Databricks
MySQL
REPLICATION CLIENT
and REPLICATION SLAVE
grants.Spark SDK
Spark with Livy