Features
Ability to add generators to primary keys
Support hex strings for inserting into binary columns in SQL Server
Bug Fixes
Replaced browser alert with a toast notification when a job fails to be scheduled
Oracle now returns columns in the order of their ordinal position
Adjusted sizing on subset dropdown
Fixed issue where the progress tracker could fail to write events near the end of a generation.
Features
New UI and Schema View
Ability to generate in place (MySQL only)
Support consistency across databases and runs via the new TONIC_STATISTICS_SEED environment variable
Performance
Many MySQL performance improvements
Bug Fixes
Fixed potential deadlock issue in MySQL
No longer re-loads session when fetching a new refresh token
Better handling of comments inside JSON blobs that are being masked
Features
Bulk operations on multiple columns in the Database View:
- Add and remove generators
- Change privacy status
UI now distinguishes between primary (gold) and foreign keyed (black) columns
Advanced search for columns in the Database View.
Generator select popover on database view is now at feature parity with same popover in Table View.
Ability to manually mark columns as private
Columns with private data that are not protected are flagged as red (Database View only) and have a warning in the generator selector
tonic_worker now has a /health endpoint
Phone Number generator supports integer columns
Differential Privacy warnings now show for Categorical generator when being used on JSON and XML columns
Ability for user to provide their own secret to be used for encrypting both JWTs and database connection info
Ability to change password (under User Settings in the hamburger menu located on the right of the menu bar)
Performance
Preserved tables in MySQL are no longer copied into the preserve schema via SELECT INTO. Instead they are RENAMEd
No longer drop and restore primary keys and indexes in MySQL
Bug Fixes
Better support for 3 byte unicode characters when writing data to masked tables in PostgreSQL
Fixed rare concurrency exception being thrown in progress tracker when jobs are running
Fixed malformed query in SQL Server which is used when subsetting
MySQL handling of character
Additional logging of MySQL index handling
Features
Bulk operations on multiple tables in the Database View
Where clause in subset - in addition to a target %, you can now specify a custom where clause as the target for subsetting
URL Generator is more private
PII type is shown in Privacy Hub
Bugs
Styling issues for Chrome and Firefox
Event Ordering defaults corrected br>
Columns with nothing put nulls gave bad data previews
Features
Health Endpoints /health added for web server and PII detection server
Non-null values are no longer shown during data preview
Sort by column name in Privacy Hub
Ability to delete a workspace
Columns part of truncated tables are now labelled as such in the Database View
PII Detection now uses the Character Scramble on suspected phone numbers
Support for Noda time in PostgresSQL and throughout the product
Other columns no longer shows in the database view
Email generator now has an excluded domain filter
Subset with Foreign Key Upload File
Performance
Database View UI rendering improvements
UI improvements for MySQL databases with 1000s of tables
Ability to process tables in parallel across all databases. This setting is controlled by the TONIC_TABLE_PARALLELISM environment variable
Faster reads from source database. This affects SQL Server, MySQL, and PostgreSQL and should see improvements for reads from the source database across the board
Optimization for SQL Server to improve writes
Better distribution of load to workers
Bug Fixes
Fixed bug in XML generation for Postgres
Reduce rate of false positive when detecting zip codes
Subsetting now properly supports self-referential foreign keys
SQL Server unique identifier columns now properly work with the UUID generator
Privacy hub has better styling for long column names
Fixed issues related to change of focus on Target % input on subset configuration page
Features
Support consistency for the email generator on a unique column
Rename and consolidation of docker images. Please contact support@tonic.ai before upgrading for a new docker-compose file
Massive refactor of subsetting, mostly behind the scenes. User facing improvements:
- Support for all databases (not just Postgres and MySQL)
- Heroku no longer requires a temporary database
- Performance boost
Categorical generator now supports differential privacy (https://www.tonic.ai/post/differential-privacy-comes-to-tonic/)
Bug Fixes
Fix for certain unicode characters in table/schema names
Fix for character scramble on numerical columns
Several fixes for air gapped deployments
Significant performance improvements for the privacy scan
Features
Generator improvements:
- Email Generator can be used on columns with a unique constraint
- Character scramble now supports consistency
- Filename and Email generators make use of character scramble as opposed to character substitution (more secure). They both also now support consistency.
More improvements to PII Detection, specifically speed, resilience, and better name detection
Ability to duplicate workspaces
Show generators applied on column in bulk view when table is truncated
Support for SSH Bastion when connecting to source and output database
Added new environment variable for controlling parallelism for MySQL generation
Bug Fixes
Fixed bad description on FK Relationships upload help text
Fixed layout issue in Replacement Panel
Fixed issue with ICD10 PII detection
Fixed routing issues on Heroku
Features
Support for JSON datatype in MySQL
More easily remove a generator from a column in the bulk view
Improved PII detection
Ability to change table modifiers on Database View
Subsetting preview of included and excluded tables
Performance
Increase speed of restoring indexes for MySQL
Bug Fixes
Fixed bad timestamp for jobs when still in queued state
Fixed bug found when using some versions of MySQL where dropping an index on a FK column results in an error
Fixed description for algebraic generator
Resolves an issue that was causing the job details page to fail to load.
Fixed MySql failing to create foreign keys because the database/schema wasn't specified in all cases
More logging
Features
XML generator - use XPath to target one or more values for a generator. This implementation is very similar to the current JSON generator
Partitioning is out of beta. The Continuous and Event generators can be partitioned by either a Categorical or Passthrough generator
Support for computed columns in SQL Server
Re-sizing of autodetect and bulk view table trees, for long table names
Schema change notifications now use small icons
Active page is highlighted on sidebar and remember sidebar state
Check for correct user permissions on source database for SQL Server when first connecting (MySQL and PostgreSQL already have this)
Gzip or br compress API responses
Performance
Faster schema generation for SQL Server
Only SELECT required columns when gathering rows for data generation
Improved UI performance for SQL Server databases with a large number of objects
Bug Fixes
Properly send ALL logs to the log viewer in the jobs UI
Continue when (most) errors occur while running pre-data script on SQL Server
Fixed issue blocking successful CSV data generation
No longer incorrectly filter out tables in the DBO schema in SQL server
Improved logging around some SMO tasks for SQL Server
Features
Auto-detect now supports booleans
Users can now click on data cells to see large text strings
PII reports
Performance
Additional multi-threading added in various places for data generation
Process generated tables before passthrough to encourage fast failure
New environment variables so users can change the connection timeout, and min and max pool size
Bug Fixes
Support for Constant Generator on MySQL Blob columns
Support MySQL set datatype
Improved logging for PII detection
Multiple fixes involving switching between synthesize, excluded, and masked
Progress tracking added to BigQuery
POST to /api/autodetect/config will generate a default config first if one does not exist.
Multiple other small fixes
Features
PII Detection is now available locally
Column widths on the table are now saved between sessions
Truncating tables is now prevented when it will lead to FK violations when generating data
Ability to more quickly add JSONPaths
Schema Diff now auto-fixes issues with model due to schema changes
Performance improvements to data generation, and JSON Generator in particular, which is now multi-threaded
Bug Fixes
Ability to handle NULL database values in the JSON Mask Generator
JSON Paths that start with the same prefix. For example, $.value and $.value1 no longer cause an error
No longer cutting off list of tables in Auto Detect dialog
Better cleanup of excluded_tables and scaled_tables in Fingerprint when tables are dropped from DB
Fixed issue preventing data generation on PostgreSQL Standby databases (in other words, Read replicas in RDS nomenclature)
Features
JSON generator now supports consistency
Better JSON Mask UI
Name generator now has consistency option
Can run auto detect without sending logs
API documentation
Better tracking of Allos Console logs
Better progress tracking
Bug Fixes
Intermittent failures of data table are fixedGetAllTablesAndColumns no longer fails when table has columns with multiple constraints on them
No longer front end crash when subset target table no longer exists
Use Custom Data button location has been fixed
Performance improvements
Fixed a myriad of issues in the First Connection Experience when using a custom data source
Additional fixes for the JSON Mask Generator
Fixed issue created in v16 where changing a table's mode to 'TRUNCATE' wasn't being saved properly
Features
Can now press Enter on any input in the Database connection form and it will submit the form
Bug Fixes
Better layout for JSON Paths with long names
Fixed a 'missing key' prop issue in front end
Adding too many escapes to MySQL data when writing to CSV (prior to upload).
We now automatically truncate MySQL tables before data generation begins
Features
Insight into whether database generation job is running or queued (was not distinguished in prior versions)
Ability to cancel a currently running database generation job
Ability to assign generators by jsonpath for the JSON Mask Generator
Tonic now prevents users from entering identical source and destination database information
When connecting to a database Tonic now defaults the Port to the standard port for the database selected.
Tonic now checks if the database account has necessary permissions and warns if it does not.
Subset button in header now reflects the state of your subset configuration
Google BigQuery now allows user to specify an input and output Dataset for generation.
Bug Fixes
Better escaping of schema and table names in mysql
Workspace edit dialog file upload inputs (Foreign Key upload and BigQuery Service Account upload) were out of sync
No longer show stale data in the table UI when switching between tables quickly
We were not properly handling queries to pg_catalog tables where columns had recently been dropped from tables
Fix to allow synthesizing on MySQL tables that have weird characters in table name
Fix error message that occurs when user checks permissions on a PostgreSQL database with 0 tables
Fix to properly keep in sync the source and destination database names in SQL Server
Constant no longer appears twice in the generator list for JSON columns in PostgreSQL
Table search dropdown now clears search query once user presses Enter or closes dialog
CreatedDate in allos_db tables now has correct values by default
Ability to Filter data table based on SQL WHERE clauses
Support for Google Big Query
Continuous generator now supports nullable columns
Remember last visited Workspace and Table and navigate directly to them when reloading page
Added on-premises installation option of one Docker image
FK Columns are greyed out in UI
Several additional bug fixes
Note: We've transitioned from major.minor.hotfix to simple integer-based versioning
Features
Autodetect Generators (beta) - this feature that scans the source dataset and by analyzing both the structural properties (data types, column names, foreign key constraints) and the content of each data field it takes a first pass at picking generators and linking generators.
Ability to define custom foreign key relationships, this addresses the issue where a database doesn't have any FK constraints or it's missing some
Algebraic Generator - when you link 3 or more columns (A, B, C, ...) with the algebraic generator applied it searches the space of functions (A + B = C, A / B = C, ... ) to find the function that best describes the algebraic relationship between the linked columns.
We now support the ability to specify via an environment variable which schemas to include
Email generator now supports a custom email domain
Renamed Gaussian generator to Continuous
Added TIN generator
- First two digits are always 00
Upgraded pg_dump to v11
Added walkthrough tutorial and demo dataset for new users
IP Generator now uses 100% IPv4 by default
Fixed bad UX with Min/Max in Random Integer generator
Improved Gaussian Generator performance
Added user menu
The distribution of nulls in the source dataset is now persisted in transformed columns in the generated dataset
Added SSN Generator
Improved Address Generator (Zip-> City->State hierarchy preserved when columns using the address generator are linked)
Features
A handful of generators now support a notion of consistency across the database. In short, when consistency is turned on for a specific generator, the same input column will map to the same output column across an entire database (where it's turned on). Consistency can also be used to preserve the cardinality of the source dataset in the generated data.
We now use row count instead of scale factor for synthesizing data
Synthesize mode now supports starting with a table that's empty
Better handling of DateTimeOffset data type in SQL Server
Added hostname generator
Re-factored of how generators are executed during data generation
We now show a tutorial video the first time a user logs in
Renamed table mode Excluded to Truncated
Build scripts now build win10-x64 assets
Changed the (!) failure icon in the jobs dropdown to a button so it's more obvious
UI and server versions now show when you click the Tonic logo
CSVs with headers now map to column names
Bug Fixes
No long require a file to exist for the google_application_credentials env variable
Couldn't generate data if we used the MAC address generator with colons
Support for MySQL datetimes with 0000-00-00 00:00:00
Support for CSV files as a datasource
Support for arrays in Postgres
Improved PII detection
Support for Postgres databases that don’t have passwords
Renamed several generators
- Categorical —> Shuffle
- Random String —> Hash
- String Mask —> Character Substitution
- Text Mask —> Text Scramble
Added generator description callouts
