Features
Added support for user defined enum types as keys in subsetting
Simplify schema diff changes error toast on generation
Show more unprotected columns button MongoDB Privacy Hub
Added filtering on custom value generator dropdown
Add ConsistentOn to Additive Noise generator
Enabling Timestamp Shift generator on compound primary keys
Regex Mask generator supports matching empty capture groups
Accommodate OFFSET in where clauses in SQL Server subsetting
Miscellaneous workspace configuration improvements
Add support Azure SQL with SqlServerDatabaseCreator
Add progress tracking for mongo schema scans
Add syntax highlighting for where clauses in subsetting
Add support for integer primary keys on smallint
Bugs
Fixed error message for MySQL data type not found
Fix PostgreSQL Test Connection bug
Fix Workspace patches after rename or modify tags
Fix Unix Timestamp empty column bug
Throw OperationCanceledException instead of CancelTonicJobException on cancellation request
Fix Amazon Redshift view replication missing schema
Features
Better error exception handling for number conversions in MongoDB
Added loading indicator when viewing single MongoDB document
Improved MongoDB performance and logging
Added table filter validation for Amazon EMR
Handling numbers as doubles instead of integers in Snowflake
Added aiven.io system schema to PostgreSQL
Added the Geo generator for latitude and longitude data
Added support for materialized view in Amazon Redshift
Bug fixes
Fixed Array Regex Mask generator with correct sub-generator options
Fixed Conditional generator by adding binding
Features
On Amazon EMR and Databricks, write JSON files to JSON instead of Parquet
Improved handling of Latitude/Longitude with Smart Linking generator
Add support for enum arrays columns in PostgreSQL
Bugs
Fix issue with Amazon EMR cross account data generation failing to resolve the Glue database
Features
Admin users can now view all workspaces in Tonic. They can copy, share, and transfer any workspace, regardless of whether they have a role in the workspace.
Multiple enhancements for workspaces. Users can now leave a workspace. Improved the workspace share and transfer functions. Added the ability to filter workspaces by owner.
Parse as bigint if decimal fails
Oracle Whitelist support in BuildDb
Initialize after log sink reset
Replace slashes with underscores in build versioning
Add 'No generators' filter for bulk table view
Add progress tracker for SQL Server post-data script generation
Allow support for views without fully qualified tables in Amazon Redshift
Improve performance of listing workspaces
Improve performance of the Constant generator on Amazon EMR and Databricks
Sensitive columns now appear in the privacy hub for MongoDB workspaces
Allow processing of decimals with a scale of 1 up to 9223372036854775807 in Postgres
Check for cloudsqlsuperuser permission on GCP Cloud SQL destination PostgreSQL databases
Bugs
Address Lambda function setup bug
Improve Regex Mask display of empty matches and values with new lines
Do not fail completely when SSO groups do not resolve
Miscellaneous Amazon Redshift View fixes
Fix issue where column can't be in schema changes and show as sensitive
Fix issue where some downstream tables are missing their foreign keys in the destination database when using subsetting
Fix issue with shifting table size on data preview in database view
Prevent error from occurring when MongoDB collection size is too large
Features
Added the ability to configure Tonic administrators. Admin users have access to the new Admin Panel. The Admin Panel contains Tonic usage metrics, a table of all Tonic users, and the ability to manage other admin users.
For Kubernetes, admin users can now use the Admin Panel to update Tonic
Oracle performance improvements
Add new range options for Integer Key generator
CNPJ and CPF generator performance improvements
Updates Regex Mask generator to support replacing all matches
Start the progress token for passthrough tables when there are no passthrough tables with references
Add support for Timestamp Shift for Spark
Add warning system to privacy scan
Regex mask generator with Custom Value Processor
Check for cancellation and report progress on text copy in Postgresql
Miscellaneous performance improvements
Bugs
Start the progress token for passthrough tables when there are no passthrough tables with references
Treat non-standard Oracle error code 5 not as an error
Fix grabbing workspace users for comments
Dispose the source connection instead of just closing in PostgreSQL
Fix for test discovery exception
Fix PostgreSQL sequences behavior
Object mask bulk view fix
Miscellaneous typos fixed
Fixed database type for BigQuery test connection
Features
Show tooltip for sidebar menu items when collapsed
Allow clicking of Queue Status and Download Job logs buttons on Jobs page
Added privacy scans to Jobs page
Added quick navigation to Jobs page from Workspaces tables
Added support for random timestamp generator for Amazon EMR
Improved Hstore generation performance
Added job-specific URLs
Webhooks are triggered when jobs are manually cancelled
Added warning for missing use statements in MySQL post-job scripts
Make JSON Mask generator respect ordering with multiple sub-generators
Bug fixes
Fix PostgreSQL binary timeout issues
Improved error handling & logging when applying Foreign Keys on subsetted SQL Server
Filter out constraints with same name as foreign key
Features
Updated to dotnet 6
Moved FK import to FK page
Added support for tinyint in MySQL as a primary key
Added table partition filtering for Amazon EMR
Enabled Regex Mask generator on Amazon EMR
Added support decimals with precision and scale for Amazon EMR
Moved Tonic version number to hamburger menu
Added job sorting by start time
Ability to retrieve Kubernetes logs via API
Improved behavior of upstream reference tables for subsetting
Improved logging for subset filtering
Updated sensitivity scan to better reflect job status
Improved privacy scan performance for MongoDB
Introduced automated schema checks (not for MongodD)
Added HTML Mask generator
Added Additive Noise generator
Added CNPJ generator
Added Array Regex Mask generator
Deprecated endpoints
Updates to the foreign key feature impact the API endpoints used to fetch, upload, and delete the Foreign Key file. Two previous API endpoints are now marked as deprecated and will be removed after 2022-06-01. The main change is the switch to using workspaceId instead of datasourceId to fetch the Foreign Key file. The deprecated endpoints are:
DELETE /api/datasource/delete_fkupload has been marked as deprecated and replaced by DELETE /api/workspace/{workspaceId}/foreign-key/GET /api/datasource/download_fkupload/ has been marked as deprecated and replaced by GET /api/workspace/{workspaceId}/foreign-key/
Bug fixes
Fix number parsing issues in JSON Mask generator
Fix MySQL parsing of auto_increment columns
Eliminated re-querying of data type on Amazon EMR Spark
Fixed issue viewing Mongo workspace when there's an empty collection
Fixed for Truncated Date Generator when used as a sub-generator
Fixed data passed to sub-generator in Regex Mask generator
Features
Ability to specify a fallback generator on XML Mask and JSON Mask generators
Ability to exclude multiple email domains in the email generator
Update column schema to preserve dropped-date and isDropped status
Change job details checkmarks to green
Prevent idle session timeout in PostgreSQL
Drop rows if decimal parse fails in PostgreSQL
Add pre-job check logging
Ability to connect to MongoDB via connection string
Allow incremental mode on tables where the PK is also an FK
Retry polling job updates and cancel job checks when there are connectivity issues with the Tonic database
Add warning message when fails to restore sequence on destination data
Always show job status warning icon when applicable
Bug Fixes
Fixed importing workspace json caused schema change alerts
Fixed Timestamp Shift for DateTimeOffset
Fixed for how we count the number of partitions on tables in SQL Server
Properly escaping MySQL Username/Password encoding
Features
Added Privacy Report CSV download for Preview (to Privacy Hub) and Generation Report (to Job Details).
Added support for preserve destination in Oracle.
Added support for temporal tables on SQL Server.
Added support for read parallelism via paginated queries.
Updated fallback generators to filter by column details and to remove text-mask generator.
Disable page parallelism when pageSize too small.
Add styles for signup divider.
Bug Fixes
Fix AuroraMySql hang.
Clarify subsetting UI.
SQL Server constraint performance improvements.
Fixes for PII scan.
Fix jobs foreign key migration issues.
Log missing environment variable once per process.
Improvements to JSON Mask performance using JToken creation to improve data type handling.
Text mask performance improvements.
Fix for polling TS error.
Features
Update workspace users/shares via web sockets
Provide job summary updates via web sockets
Make jump to page in Workspaces table more discoverable
Display which Tonic tier the customer is on
Wrap DNS resolution in try-catch block to avoid errors upstream
Expose ports 443 and 80 in Dockerfiles for applicable containers
Update session variables to enforce autocommit for MySQL
Parallelization of row processing for improved job performance
Bug Fixes
Removed random sampling in ConsistentOn generators
Fixed newly edited tags of active workspace not being exported
Updated Oracle writer path for job improvement
Improved error message when test connection fails
Made error message more clear for generators that perform frequency samples
Fixed Db2 connection spinning indefinitely via timeout
Features
Added incremental mode support for MySQL
Added workspace filter/sort improvements
Added Recovery Mode check for SQL Server destination databases
Added support for connecting to mongoDB via connection string
PII report jobs are now canceled when zombied/abandoned
Added button to resolve all schema changes
Added the ability to generate full US address
Bug Fixes
Faster sampling for large tables when running privacy scan
Improved handling of null exceptions when subsetting
Made performance improvements to get all workspaces
Made performance improvements to only job summaries when needed
Improved data generation for latitude/longitude with Smart Linking
Improved formatting of error messages on job details
Features
Extend timestamp format to microseconds
Restrict workspace generation details to Owners
Allow webhooks on all database types
Added Struct support on Spark
Bug Fixes
Fixed lingering column on bulk view when configuring generators
State abbreviation privacy check requires some variability
Added handling of when custom categories are null in the metadata for Custom Categorical generator
Fixed date parsing for event generator
Smart linking bulk edit + metadata
Do not add AND clause if there are no non-filtering columns (subsetting)
Improved Sidebar indicators
Fixed inability to generate over tables which contain invisible tables in Oracle
Added null check for error message in subsetting
Hide Subsetting/Post-Job Actions in Workspaces view when not applicable
Added used packaged explicitly and removed unused package dependencies
Primary key columns in synthesized mode should not have generator piсker
Features
Random Timestamp output format selection
Redshift allow users to preserve source database owners
Support binary types as primary keys in subsetting
Log and skip over unresolved URNs instead of failing the job
Ignore VIEWS for Databricks and Amazon EMR
Released Additive Noise generator
Added generator for creating CPF numbers
Bypassing XML validation on SQL Server XML columns is now optional
Added support for Windows authentication on SQL Server
Bug Fixes
Editors should have read only on workspace edit dialog
If no user has have been created, show create account page
Increased dialog width to prevent overflow when EMR spark is selected.
Added leading zero to 4 digit zip codes
Removed unavailable features from UI for Spark
Features
Improved user experience for Workspaces view
Custom value processors can now be applied to not-null non-replaceable data types.
Update subsetting toggle on/off via web sockets
Bulk store column info for performance improvements
Disable generators for generated columns in Oracle
Added support for DML table-level triggers
Preserve file preferences in Oracle
Bug Fixes
Login page no longer displays error on page load when SSO is not configured.
SQL Server now preserve their decimal places on numeric column types when using the continuous generator.
Continuous generator no longer fails on missing fields in MongoDB.
Fixed issue where Duo login button was disabled when REQUIRE_SSO_AUTH is true.
Connection test for Snowflake and Amazon Redshift no longer reports incorrect permission error.
Fixed issue where you could not edit a workspace if you lacked permissions on the active workspace.
Generated columns in PostgreSQL are no longer attempted to be written to the destination database.
Fixed Spark generations by no longer attempting to instantiate a SparkSession in the UDF
Fixed parsing issue in Databricks Jobs API where we were treating an Int64 value as an Int32
Fixed issue where continuous generator would fail when the precision of a numeric column is unavailable
Removed security vulnerabilities from Notifications container
Added signed tokens to Amazon Redshift load and unload statements
Features
Improved message when database job finishes
Added dependency sort on tables set to Incremental mode
Added HStoreMaskGenerator specific to PostgreSQL hstore types
Adding conditional generator support to MongoDB.
Added support for AWS SSO
Better error messages on the front-end when out-of-range IntPkGenerator
Bug Fixes
Removed linking from Random Boolean generator
EMR should show more than 100 tables when appropriate
Preserve Destination disabled in bulk on EMR and Databricks
Upgraded Redis to resolve CVE-2021-32762
Database type now shows properly for newly created workspaces in the workspace view
Changed the download logs button to be more intuitive for jobs that don't use SmartLinking
Consolidate Copy vs Duplicate terminology
Job end times now show properly when jobs terminate unexpectedly
Break long workspace names to avoid overflow
Fix Enable Log Collection for Lambda Functions
Fix scaled uuid foreign key generation for unique columns
Fixed data type for numeric in SQL Server
Fix varbinary truncation in MySQL
Features
Added UUID primary key support for synthesized mode.
Added support for Duo 2FA SSO.
Updates for Oracle XML, RAW types.
Handle triggers and grants when creating databases for Oracle.
Null generator now supported as Spark UDF.
Handle spatial (geometry/geography) types in SQL Server.
Bug Fixes
Update tracking URL in Databricks.
Leave temp schemas alone.
Fix ConsistentOn in Regex Mask generator.
Security updates for Linux.
Features
Update Database view filter behavior
Added configuration option to disable account creation.
Regex mask generator is now supported on unique columns.
Warnings and errors for Redshift and Snowflake are now displayed on the job details page.
Updated job queued status.
Add CloudWatch Log Filtering by Log Level.
Bug Fixes
Performance improvement for hybrid document creation for MongoDB.
Fixed issue where constraints could be duplicated when using the same Oracle server for source and destination.
Enhanced Oracle log messages when copying data.
Features
PII Scan will continue on error and log all issues.
Added Unique Phone Number generator.
Added support for Integer Key generator on decimal columns where precision is zero.
Rename "output" database to "destination".
Added Table Mode descriptions on hover
Added TLS support for Db2 iSeries connections.
Bug FixesTONIC_MYSQL_USE_COMPRESSION environment variable was removed.
Slow API responses that are still in process when a workspace switch occurs will no longer cause stale data to show in the application.
Properly set last visited collection for MongoDB workspaces.
Fix workspace row alignment.
Disallow applying generators on PostgreSQL generated columns.
Fixed logging issues with PII scan errors.
Leave destination schema alone on Db2.
Fix subsetting of FKs with same name.
Features
JSON Mask for Spark.
Make bulk applying generators more friendly on columns with disparate types.
Generated columns are now supported for PostgreSQL.
EMR will output CSV files if the input file is a CSV.
Added support for unsigned integer columns to the Integer Key generator.
Performance improvements for MySQL.
Bug Fixes
Improved iSeries UI response time.
Fixed copy workspace issue where some fields were empty until a page refresh was performed.
EMR Hive DDL fix for CSV.
Fixed issue with droping indexes on auto-incrementing composite keys in MySQL
Improve Db2 post data order.
Improved MySQL performance for CopyTable.
Features
Jobs now auto-stop if the worker crashes
Support for ENUM Arrays in PostgreSQL.
Categorical generator can now be used on non-string types in MongoDB.
Allow Conditional Generator on JSON, JSONB, and Boolean columns.
Custom Categorical and Categorical generator support for MongoDB arrays.
Add optional 'use compression' flag for MySQL.
Bug Fixes
Fixed issue when swapping between MongoDB workspaces
Fix Timestamp Shift generator for MongoDB.
Subsetting enhancements
EMR partitioned table UI bug fix.
Fixed an issue where metadata for the conditional generator did not always save correctly
Fix EMR Test Connection issue.
Features
Add pagination to ECR Image search.
Workspace tags.
Optimized the privacy scan for Oracle DBs.
Add Custom Categorical generator to Mongo DB.
Support for EMR cross account.
Bug Fixes
Fix issues with resizable panes in Database View and Table View.
Allow non-null self references in a table in subsetting only if the table is a reference table.
Security enhancements
Fixed issues in MongoDB workspaces with dismissing all schema diffs.
Better handling of Check constraints in MySQL
Features
Added support for Db2 iSeries
Added support for Delta Table on Databricks
Added support for reading IAM roles off Databricks machines instead of providing IAM credentials
Added a UI notification when your Tonic version is 10+ versions behind
Added support for MongoDB 2.4 and 3.4
Added the ability to automatically skip tables that match a regular expression via an environment variable
Added the ability to obfuscate values inside an array in Mongo
Added the ability for the JSON Mask generator to parse JSON objects containing escaped characters and surrounded by quotes
Added the ability to automatically cast for mismatched types during downstream subsetting
Added a Regex generator
Added a Generative Adversarial Network Generator
Bug Fixes
Refreshes subset preview when table mode is changed
Improved handling of IAM credentials on Databricks when an IAM instance profile is present
Hide Other in generator dropdown when there are no additional generators
Features
Added TLS support for iSeries Db2.
CloudWatch logs are available inside Tonic for AWS Lambda based jobs.
Support for 'consistent on column' in Regex Mask generator.
Bug Fixes
Fix SIN generator being applied to non-sin columns.
Optimized schema construction for destination database in DB2 LUW.
Truncated tables now handled properly for AWS Lambda based jobs.
Fix rendering of tables with double digit replacements on bulk view.
Consistent On dropdown is now alphabetized.
Features
Added destination database connection summary to job start confirmation.
Subsetting and post-job action indicators have been added to the workspaces view.
Added workspace name as a webhook content option.
Bug Fixes
Fixed issue with PostgreSQL index restoration and error handling.
Fixes for Tonic UI when connected to MongoDB
