Features
Invalidate refresh tokens on logout, log in or out all open sessions (tabs) on login or logout, cross-tab inactivity timeout support
Improve load times on workspace pages
Bugs
Fix linking of non-consistent generators in Spark
API update
Deprecated the GET /api/workspace endpoint in favor of GET /api/workspace/search
Features
Include workspace ID in URLs
Parallelize writes on MySQL passthrough tables
Allow decimals in subsetting percentages
Bugs
Don't allow duplicate entries of tables in table mode settings for workspace
Handle null values with Copy generator on MongoDB
Database query performance improvements
Features
On Amazon EMR and Databricks, write JSON files to JSON instead of Parquet
Improved handling of Latitude/Longitude with Smart Linking generator
Add support for enum arrays columns in PostgreSQL
Bugs
Fix issue with Amazon EMR cross account data generation failing to resolve the Glue database
Features
Admin users can now view all workspaces in Tonic. They can copy, share, and transfer any workspace, regardless of whether they have a role in the workspace.
Multiple enhancements for workspaces. Users can now leave a workspace. Improved the workspace share and transfer functions. Added the ability to filter workspaces by owner.
Parse as bigint if decimal fails
Oracle Whitelist support in BuildDb
Initialize after log sink reset
Replace slashes with underscores in build versioning
Add 'No generators' filter for bulk table view
Add progress tracker for SQL Server post-data script generation
Allow support for views without fully qualified tables in Amazon Redshift
Improve performance of listing workspaces
Improve performance of the Constant generator on Amazon EMR and Databricks
Sensitive columns now appear in the privacy hub for MongoDB workspaces
Allow processing of decimals with a scale of 1 up to 9223372036854775807 in Postgres
Check for cloudsqlsuperuser permission on GCP Cloud SQL destination PostgreSQL databases
Bugs
Address Lambda function setup bug
Improve Regex Mask display of empty matches and values with new lines
Do not fail completely when SSO groups do not resolve
Miscellaneous Amazon Redshift View fixes
Fix issue where column can't be in schema changes and show as sensitive
Fix issue where some downstream tables are missing their foreign keys in the destination database when using subsetting
Fix issue with shifting table size on data preview in database view
Prevent error from occurring when MongoDB collection size is too large
Features
Added support for user defined enum types as keys in subsetting
Simplify schema diff changes error toast on generation
Show more unprotected columns button MongoDB Privacy Hub
Added filtering on custom value generator dropdown
Add ConsistentOn to Additive Noise generator
Enabling Timestamp Shift generator on compound primary keys
Regex Mask generator supports matching empty capture groups
Accommodate OFFSET in where clauses in SQL Server subsetting
Miscellaneous workspace configuration improvements
Add support Azure SQL with SqlServerDatabaseCreator
Add progress tracking for mongo schema scans
Add syntax highlighting for where clauses in subsetting
Add support for integer primary keys on smallint
Bugs
Fixed error message for MySQL data type not found
Fix PostgreSQL Test Connection bug
Fix Workspace patches after rename or modify tags
Fix Unix Timestamp empty column bug
Throw OperationCanceledException instead of CancelTonicJobException on cancellation request
Fix Amazon Redshift view replication missing schema
Features
Added the ability to configure Tonic administrators. Admin users have access to the new Admin Panel. The Admin Panel contains Tonic usage metrics, a table of all Tonic users, and the ability to manage other admin users.
For Kubernetes, admin users can now use the Admin Panel to update Tonic
Oracle performance improvements
Add new range options for Integer Key generator
CNPJ and CPF generator performance improvements
Updates Regex Mask generator to support replacing all matches
Start the progress token for passthrough tables when there are no passthrough tables with references
Add support for Timestamp Shift for Spark
Add warning system to privacy scan
Regex mask generator with Custom Value Processor
Check for cancellation and report progress on text copy in Postgresql
Miscellaneous performance improvements
Bugs
Start the progress token for passthrough tables when there are no passthrough tables with references
Treat non-standard Oracle error code 5 not as an error
Fix grabbing workspace users for comments
Dispose the source connection instead of just closing in PostgreSQL
Fix for test discovery exception
Fix PostgreSQL sequences behavior
Object mask bulk view fix
Miscellaneous typos fixed
Fixed database type for BigQuery test connection
Features
Relax unique index constraint on username
Added tonic version in containers as an environment variable
Adding logic to bypass fallback generators for Spark databases
Support case insensitive name consistency
Disable dropping replacements on missing columns for MongoDB
Update Oracle driver
Enable additive noise generation for strings in MongoDB
Add support for unsigned 64-bit integer columns in integer key generator
Add random timestamp and timestamp shift generators as options for integer columns (for unix timestamp columns)
Add checks for min/maxTime on timestamp and min/maxDate on dates
Adding null ref check for fallback generation option check during preview
Revamped workspace configuration screen
Support for Decimal on Spark (when using C# UDFs)
Allow Regex generator on PKs
Added new environment variable to filter tables ingested by UI for Oracle
Miscellaneous perf improvements
Bug fixes
Fix mongo view more values issues
Disable dropping replacements on missing columns for MongoDB
Fix workspace specified via API without databases can be modified in UI
Fixing Random Timestamp generator on Java
Date truncation generator bug fixes
Features
Better error exception handling for number conversions in MongoDB
Added loading indicator when viewing single MongoDB document
Improved MongoDB performance and logging
Added table filter validation for Amazon EMR
Handling numbers as doubles instead of integers in Snowflake
Added aiven.io system schema to PostgreSQL
Added the Geo generator for latitude and longitude data
Added support for materialized view in Amazon Redshift
Bug fixes
Fixed Array Regex Mask generator with correct sub-generator options
Fixed Conditional generator by adding binding
Features
Updated to dotnet 6
Moved FK import to FK page
Added support for tinyint in MySQL as a primary key
Added table partition filtering for Amazon EMR
Enabled Regex Mask generator on Amazon EMR
Added support decimals with precision and scale for Amazon EMR
Moved Tonic version number to hamburger menu
Added job sorting by start time
Ability to retrieve Kubernetes logs via API
Improved behavior of upstream reference tables for subsetting
Improved logging for subset filtering
Updated sensitivity scan to better reflect job status
Improved privacy scan performance for MongoDB
Introduced automated schema checks (not for MongodD)
Added HTML Mask generator
Added Additive Noise generator
Added CNPJ generator
Added Array Regex Mask generator
Deprecated endpoints
Updates to the foreign key feature impact the API endpoints used to fetch, upload, and delete the Foreign Key file. Two previous API endpoints are now marked as deprecated and will be removed after 2022-06-01. The main change is the switch to using workspaceId instead of datasourceId to fetch the Foreign Key file. The deprecated endpoints are:
DELETE /api/datasource/delete_fkupload has been marked as deprecated and replaced by DELETE /api/workspace/{workspaceId}/foreign-key/GET /api/datasource/download_fkupload/ has been marked as deprecated and replaced by GET /api/workspace/{workspaceId}/foreign-key/
Bug fixes
Fix number parsing issues in JSON Mask generator
Fix MySQL parsing of auto_increment columns
Eliminated re-querying of data type on Amazon EMR Spark
Fixed issue viewing Mongo workspace when there's an empty collection
Fixed for Truncated Date Generator when used as a sub-generator
Fixed data passed to sub-generator in Regex Mask generator
Features
Show tooltip for sidebar menu items when collapsed
Allow clicking of Queue Status and Download Job logs buttons on Jobs page
Added privacy scans to Jobs page
Added quick navigation to Jobs page from Workspaces tables
Added support for random timestamp generator for Amazon EMR
Improved Hstore generation performance
Added job-specific URLs
Webhooks are triggered when jobs are manually cancelled
Added warning for missing use statements in MySQL post-job scripts
Make JSON Mask generator respect ordering with multiple sub-generators
Bug fixes
Fix PostgreSQL binary timeout issues
Improved error handling & logging when applying Foreign Keys on subsetted SQL Server
Filter out constraints with same name as foreign key
Features
Added Privacy Report CSV download for Preview (to Privacy Hub) and Generation Report (to Job Details).
Added support for preserve destination in Oracle.
Added support for temporal tables on SQL Server.
Added support for read parallelism via paginated queries.
Updated fallback generators to filter by column details and to remove text-mask generator.
Disable page parallelism when pageSize too small.
Add styles for signup divider.
Bug Fixes
Fix AuroraMySql hang.
Clarify subsetting UI.
SQL Server constraint performance improvements.
Fixes for PII scan.
Fix jobs foreign key migration issues.
Log missing environment variable once per process.
Improvements to JSON Mask performance using JToken creation to improve data type handling.
Text mask performance improvements.
Fix for polling TS error.
Features
Ability to specify a fallback generator on XML Mask and JSON Mask generators
Ability to exclude multiple email domains in the email generator
Update column schema to preserve dropped-date and isDropped status
Change job details checkmarks to green
Prevent idle session timeout in PostgreSQL
Drop rows if decimal parse fails in PostgreSQL
Add pre-job check logging
Ability to connect to MongoDB via connection string
Allow incremental mode on tables where the PK is also an FK
Retry polling job updates and cancel job checks when there are connectivity issues with the Tonic database
Add warning message when fails to restore sequence on destination data
Always show job status warning icon when applicable
Bug Fixes
Fixed importing workspace json caused schema change alerts
Fixed Timestamp Shift for DateTimeOffset
Fixed for how we count the number of partitions on tables in SQL Server
Properly escaping MySQL Username/Password encoding
Features
Update workspace users/shares via web sockets
Provide job summary updates via web sockets
Make jump to page in Workspaces table more discoverable
Display which Tonic tier the customer is on
Wrap DNS resolution in try-catch block to avoid errors upstream
Expose ports 443 and 80 in Dockerfiles for applicable containers
Update session variables to enforce autocommit for MySQL
Parallelization of row processing for improved job performance
Bug Fixes
Removed random sampling in ConsistentOn generators
Fixed newly edited tags of active workspace not being exported
Updated Oracle writer path for job improvement
Improved error message when test connection fails
Made error message more clear for generators that perform frequency samples
Fixed Db2 connection spinning indefinitely via timeout
Features
Added incremental mode support for MySQL
Added workspace filter/sort improvements
Added Recovery Mode check for SQL Server destination databases
Added support for connecting to mongoDB via connection string
PII report jobs are now canceled when zombied/abandoned
Added button to resolve all schema changes
Added the ability to generate full US address
Bug Fixes
Faster sampling for large tables when running privacy scan
Improved handling of null exceptions when subsetting
Made performance improvements to get all workspaces
Made performance improvements to only job summaries when needed
Improved data generation for latitude/longitude with Smart Linking
Improved formatting of error messages on job details
Features
Extend timestamp format to microseconds
Restrict workspace generation details to Owners
Allow webhooks on all database types
Added Struct support on Spark
Bug Fixes
Fixed lingering column on bulk view when configuring generators
State abbreviation privacy check requires some variability
Added handling of when custom categories are null in the metadata for Custom Categorical generator
Fixed date parsing for event generator
Smart linking bulk edit + metadata
Do not add AND clause if there are no non-filtering columns (subsetting)
Improved Sidebar indicators
Fixed inability to generate over tables which contain invisible tables in Oracle
Added null check for error message in subsetting
Hide Subsetting/Post-Job Actions in Workspaces view when not applicable
Added used packaged explicitly and removed unused package dependencies
Primary key columns in synthesized mode should not have generator piсker
Features
Random Timestamp output format selection
Redshift allow users to preserve source database owners
Support binary types as primary keys in subsetting
Log and skip over unresolved URNs instead of failing the job
Ignore VIEWS for Databricks and Amazon EMR
Released Additive Noise generator
Added generator for creating CPF numbers
Bypassing XML validation on SQL Server XML columns is now optional
Added support for Windows authentication on SQL Server
Bug Fixes
Editors should have read only on workspace edit dialog
If no user has have been created, show create account page
Increased dialog width to prevent overflow when EMR spark is selected.
Added leading zero to 4 digit zip codes
Removed unavailable features from UI for Spark
Features
Improved user experience for Workspaces view
Custom value processors can now be applied to not-null non-replaceable data types.
Update subsetting toggle on/off via web sockets
Bulk store column info for performance improvements
Disable generators for generated columns in Oracle
Added support for DML table-level triggers
Preserve file preferences in Oracle
Bug Fixes
Login page no longer displays error on page load when SSO is not configured.
SQL Server now preserve their decimal places on numeric column types when using the continuous generator.
Continuous generator no longer fails on missing fields in MongoDB.
Fixed issue where Duo login button was disabled when REQUIRE_SSO_AUTH is true.
Connection test for Snowflake and Amazon Redshift no longer reports incorrect permission error.
Fixed issue where you could not edit a workspace if you lacked permissions on the active workspace.
Generated columns in PostgreSQL are no longer attempted to be written to the destination database.
Fixed Spark generations by no longer attempting to instantiate a SparkSession in the UDF
Fixed parsing issue in Databricks Jobs API where we were treating an Int64 value as an Int32
Fixed issue where continuous generator would fail when the precision of a numeric column is unavailable
Removed security vulnerabilities from Notifications container
Added signed tokens to Amazon Redshift load and unload statements
Features
Improved message when database job finishes
Added dependency sort on tables set to Incremental mode
Added HStoreMaskGenerator specific to PostgreSQL hstore types
Adding conditional generator support to MongoDB.
Added support for AWS SSO
Better error messages on the front-end when out-of-range IntPkGenerator
Bug Fixes
Removed linking from Random Boolean generator
EMR should show more than 100 tables when appropriate
Preserve Destination disabled in bulk on EMR and Databricks
Upgraded Redis to resolve CVE-2021-32762
Database type now shows properly for newly created workspaces in the workspace view
Changed the download logs button to be more intuitive for jobs that don't use SmartLinking
Consolidate Copy vs Duplicate terminology
Job end times now show properly when jobs terminate unexpectedly
Break long workspace names to avoid overflow
Fix Enable Log Collection for Lambda Functions
Fix scaled uuid foreign key generation for unique columns
Fixed data type for numeric in SQL Server
Fix varbinary truncation in MySQL
Features
Added UUID primary key support for synthesized mode.
Added support for Duo 2FA SSO.
Updates for Oracle XML, RAW types.
Handle triggers and grants when creating databases for Oracle.
Null generator now supported as Spark UDF.
Handle spatial (geometry/geography) types in SQL Server.
Bug Fixes
Update tracking URL in Databricks.
Leave temp schemas alone.
Fix ConsistentOn in Regex Mask generator.
Security updates for Linux.
Features
Update Database view filter behavior
Added configuration option to disable account creation.
Regex mask generator is now supported on unique columns.
Warnings and errors for Redshift and Snowflake are now displayed on the job details page.
Updated job queued status.
Add CloudWatch Log Filtering by Log Level.
Bug Fixes
Performance improvement for hybrid document creation for MongoDB.
Fixed issue where constraints could be duplicated when using the same Oracle server for source and destination.
Enhanced Oracle log messages when copying data.
Features
PII Scan will continue on error and log all issues.
Added Unique Phone Number generator.
Added support for Integer Key generator on decimal columns where precision is zero.
Rename "output" database to "destination".
Added Table Mode descriptions on hover
Added TLS support for Db2 iSeries connections.
Bug FixesTONIC_MYSQL_USE_COMPRESSION environment variable was removed.
Slow API responses that are still in process when a workspace switch occurs will no longer cause stale data to show in the application.
Properly set last visited collection for MongoDB workspaces.
Fix workspace row alignment.
Disallow applying generators on PostgreSQL generated columns.
Fixed logging issues with PII scan errors.
Leave destination schema alone on Db2.
Fix subsetting of FKs with same name.
Features
Jobs now auto-stop if the worker crashes
Support for ENUM Arrays in PostgreSQL.
Categorical generator can now be used on non-string types in MongoDB.
Allow Conditional Generator on JSON, JSONB, and Boolean columns.
Custom Categorical and Categorical generator support for MongoDB arrays.
Add optional 'use compression' flag for MySQL.
Bug Fixes
Fixed issue when swapping between MongoDB workspaces
Fix Timestamp Shift generator for MongoDB.
Subsetting enhancements
EMR partitioned table UI bug fix.
Fixed an issue where metadata for the conditional generator did not always save correctly
Fix EMR Test Connection issue.
Features
JSON Mask for Spark.
Make bulk applying generators more friendly on columns with disparate types.
Generated columns are now supported for PostgreSQL.
EMR will output CSV files if the input file is a CSV.
Added support for unsigned integer columns to the Integer Key generator.
Performance improvements for MySQL.
Bug Fixes
Improved iSeries UI response time.
Fixed copy workspace issue where some fields were empty until a page refresh was performed.
EMR Hive DDL fix for CSV.
Fixed issue with droping indexes on auto-incrementing composite keys in MySQL
Improve Db2 post data order.
Improved MySQL performance for CopyTable.
Features
Add pagination to ECR Image search.
Workspace tags.
Optimized the privacy scan for Oracle DBs.
Add Custom Categorical generator to Mongo DB.
Support for EMR cross account.
Bug Fixes
Fix issues with resizable panes in Database View and Table View.
Allow non-null self references in a table in subsetting only if the table is a reference table.
Security enhancements
Fixed issues in MongoDB workspaces with dismissing all schema diffs.
Better handling of Check constraints in MySQL
Features
Added TLS support for iSeries Db2.
CloudWatch logs are available inside Tonic for AWS Lambda based jobs.
Support for 'consistent on column' in Regex Mask generator.
Bug Fixes
Fix SIN generator being applied to non-sin columns.
Optimized schema construction for destination database in DB2 LUW.
Truncated tables now handled properly for AWS Lambda based jobs.
Fix rendering of tables with double digit replacements on bulk view.
Consistent On dropdown is now alphabetized.
Features
Added destination database connection summary to job start confirmation.
Subsetting and post-job action indicators have been added to the workspaces view.
Added workspace name as a webhook content option.
Bug Fixes
Fixed issue with PostgreSQL index restoration and error handling.
Fixes for Tonic UI when connected to MongoDB
Features
Added support for incremental table mode on PostgreSQL.
Enable TimeStamp Shift generator for text columns.
Updates to the Privacy Hub for MongoDB
Add support for object, variant types in Snowflake.
HIPAA Address generator now handles extended zip codes that don't contain a hyphen.
Inactivity timeout and authorization/refresh token timeout can now be configured.
MySQL now copies over routines and events (if permissions are set) to the destination database.
Bug Fixes
Empty schemas in Db2 output database are now handled properly.
Http now forwards to https on the health check endpoint of the PII Scanner container.
Resolved issue in the subset preview when the estimated row count of the source is unavailable.
Features
New UI to view all workspaces.
Updates to Privacy Hub to track progress in protecting sensitive fields.
Db2 support for triggers and stored procedures.
Prevent Db2 summary tables and views from showing in the UI.
Support for Regex Mask, JSON Mask, and XML Mask generators on Snowflake and Amazon Redshift.
Allow Content-Type Header in webhook requests.
Ability to cancel data generation jobs from the job details page.
Schema differences can now be detected on a MongoDB collection.
Bug Fixes
Memory usage reduction on MySQL and Oracle.
ErrorOnOverwrite for Databricks table mode.
Usability improvements with MongoDB workspaces.
Features
Users can now download job logs directly through the Tonic application.
Integer primary key generator allows more control.
Add index restoration parallelism for PostgreSQL.
Add name case consistency.
Bug Fixes
Prevent Safari from crashing.
No longer print environment variables when launching the PII detection or machine learning containers.
Improve API performance for endpoint api/schemadiff.
Mongo RegexMaskEditor now gets column values.
Features
Smart Linking generators added - train neural networks to mimic the implicit relationships across columns.
Webhooks now support sending JSON object literals.
MongoDB expanded generators on allowed data types and consistency for multiple types of paths.
Clarified error messages in the UI.
Constant generator will now show a true/false drop down when added to a boolean field.
Bug Fixes
Improved performance of schema change detection on large workspaces.
Handle long generator names in the UI.
Allow notifications container to start up without TONIC_URL env variable.
Updated TLS cipher negotiation between Tonic and AWS Aurora MysQL.
Full name will now be consistent with first and last name.
Null values in SQL Server xml fields are now handled properly.
Optimized memory usage on very large SQL Server rows.
Reduced the number of times statistics have to be calculated for JSON Mask generator, XML Mask generator, and Regex Mask generator.
Fixed issue where Amazon S3 + Spark jobs would not start.
Features
Preserve N bytes for MAC address generation
Modeling panel in table view can now be resized
Repartition and coalescing added for Databricks
Subset preview on small screen improved
Speed up PII scan for MySQL
Updated subsetting logs
Increment default fingerprint schema version
Bug Fixes
Disable Generate Data during page load
Fix constant generator timestamp issue
Handle case when Spark not installed on EMR cluster
Increase BigQuery limit of 10GB
Memory optimized tables fixed when tearing down SQL Server
Shift + Select in Bulk in MySQL fixed
Features
Webhooks can now be created to alert external systems when a job has finished.
Magnifying glass now appears in Collection View in Mongo to show additional values for a given path.
Post-job scripts can now be put in a specific order by user.
MongoDB expanded to support comments, notifications, current date generator, collection search.
Generators can be found by metadata now as well as name.
Copy workspace action limited to owner only.
Remove tutorial video.
Show UI warnings for Oracle when rows are rejected by sqlldr.
Bug Fixes
Data in identity columns is now properly inserted into Amazon Redshift databases.
User defined types are now dropped in the correct order when tearing down a SQL Server database.
Job status API no longer returns information about other jobs.
Make the polling for jobs resilient to failed checks.
Even better upstream null handling.
Features
Added support for Db2 iSeries
Added support for Delta Table on Databricks
Added support for reading IAM roles off Databricks machines instead of providing IAM credentials
Added a UI notification when your Tonic version is 10+ versions behind
Added support for MongoDB 2.4 and 3.4
Added the ability to automatically skip tables that match a regular expression via an environment variable
Added the ability to obfuscate values inside an array in Mongo
Added the ability for the JSON Mask generator to parse JSON objects containing escaped characters and surrounded by quotes
Added the ability to automatically cast for mismatched types during downstream subsetting
Added a Regex generator
Added a Generative Adversarial Network Generator
Bug Fixes
Refreshes subset preview when table mode is changed
Improved handling of IAM credentials on Databricks when an IAM instance profile is present
Hide Other in generator dropdown when there are no additional generators
Features
Added support for MongoDB
Added support for Amazon Redshift
Delta Table support on Databricks
Upgrades the subset preview with UI improvements
Bug Fixes
Better handling of adding foreign keys pointing to non-primary keys
Checks for PostgreSQL version mismatch
Remove generator button now works in synthesis mode and on key columns
Added support to break subsetting cycle when both Foreign Key and Primary Key are nullable
Features
Subset Preview
Support for Google SSO, including ability to read group membership
Java UDFs on Spark for Character Scramble and MAC Address generators
Added custom value processor extension framework
Bug Fixes
Auto-increment on MySQL bug fix
No longer remote auto-increment on primary keys or on preserved tables
Fixes composite foreign key issue with one nullable column in key
Incremental mode now works when a rowversion or timestamp column is on the table
Fixed collation issue for MySQL 8
Features
Add environment variable for altering MySQL batch size during CopyRows
Ability to search label on Foreign Key viewer
CSV generator support for tab/multi delimiters
Better performance tracking in the UI for upstream and downstream tables
Sunset Classic Subsetting in favor of the now default Full algorithm
Bug Fixes
Fix auto_increment issue on bigint columns
Don't fail if foreign key column has been removed
Fix Auto Increment on MySQL when dropping indexes
Better support for JSON during MySQL generation
Features
Support added for non-primary key auto-incrementing columns
MySQL no longer requires locks on the source database during generations
Generators are now easier to remove in the UI
Reference tables can now be defined in subsetting configurations
Enhancements to the Foreign Key UI
Added a clickable link to the Job Start notification
New Character Separated Value generator added
Bug Fixes
Performance optimizations for the Address generator
Improved UI experience for workspaces with thousands of tables
Optimized memory usage for large workspaces during data generation jobs
Features
Subsetting upstream has better handling for table relations with multiple constraint groups where one of the constraint groups is often null
Differential Privacy is now available for Continuous generator
Added new Tonic logos
Generators are not allowed on foreign key columns that are also primary keys
Subsetting is now functional when Primary Key Generators are applied to primary keys
Bug Fixes
Resolved data upload for MySQL
Oracle 19 helper improvements
HIPAA address generator fix for zip codes
Refresh data table after Foreign Keys update
Features
Password reset functionality added
Search and Sort columns when adding Foreign keys in Tonic
Nullable Foreign keys are now an option when adding them in Tonic
New Date Shift generator added
Foreign Keys are now sorted alphabetically by default
Ability to set a starting point in the sequential integer generator
Bug Fixes
Fix in how we handle multi-column primary keys in subsetting
@ mentioning style improvements for commenting
UI fixes for Foreign Keys section
Enhancements to statistics generation with conditional + categorical generators
Copying Databricks workspace works as expected
Features
Support for Db2 LUW added
New option to set strictness for schema changes
Added support for pg_repack extension
Added support for more Key generators in Spark
Added support for tinyint and smallint data types in subsetting
Bug Fixes
Better cache handling for subsetting
Improvements for NULL checking in Consistency On
Fixed issues in subsetting preview
Reduced collisions in Unique Email generator
Workers no longer crash when unable to obtain a queued job
Empty post-job scripts now throw a warning rather than failing the job
Better log messages with primary key generators fail
Fixed arithmetic overflow error when calculating SQL Server database size
Fixed upstream exhaustion in subsetting
Features
Protection Audit Trail now logs enable/disable of Differential Privacy toggle
Support for Memory Optimized tables in SQL Server
Ability to add new relationships in the Foreign Keys section
NULL generator can now be used on columns with uniqueness constraints
New subsetting option: Full Algorithm (default is still Classic Algorithm)
Bug Fixes
Fixed issue with retrieving column names when adding FK in Foreign Keys section
resolved conflict when setting generator to consistent on a column with a Constant generator
Switching between email generator and unique email generator now clears state as expected
MySQL point columns no longer halt generation
Features
New UUID Key generator
Support for PostgreSQL Client keys
Options to preserve OUIs and Uniqueness in MAC address generator
New HIPAA Address generator
Email generator can be used on synthesized tables
Custom Categorical generators can now be linked
Workspace ID copy button added
Column output data can now be made consistent on other column's data
Spark filtering of tables in Databricks
Improved logging
Bug Fixes
Better handling of times without timezone values in PostgreSQL
UI fixes
Subscriptions and Publications no longer copied in PostgreSQL
Features
API endpoint test_destination_db_connection added
User can now filter tables by schema in the Database View
Support for Hive + Spark datasource added
Support for synthesis on datetime primary keys
Improvements/Bug Fixes
Improved error messages displayed in the UI
Logging improvements
Synthesis mode improvements
Fixed UI issue with First connection wizard
Features
Added support for Spark/Amazon S3 as a data source and destination
Improved Company Name generator
Added support for text and ntext types in SQL Server
Added support for Google Big Query as a data source and destination
Workspace ownership can now be transferred from user to user
Bug Fixes
AWS Commons extension no longer breaks data generations
JSON and XML Mask generator configuration fixes
Minor display fixes
Features
Added the ability to create post job scripts that will be executed against the destination server at the end of the generation
Added Cross Table Sum generator. Allows summing of rows from another table by partition
Added single sign-on support
Added Enterprise license key support
Added a check to validate the foreign key file before a generation executes
Allow maximum for Integer Key generator
Removed dependency on RabbitMQ
Added ability to have SQL Server and PostgreSQL trust server certificates in workspace editor
Additional logging was added around relationship integrity during subsetting
Added adjustable batch sizes for Oracle
Bug Fixes
Fixed several rendering issues in the table view
Tries multiple connection methods for Oracle
Supports Extended VarChar2s for Oracle
No longer hides foreign key column headings for synthesize mode
Job descriptions for jobs cancelled before they started running are more accurate
Testing the source database connection now works for viewers and auditors
Workspaces are now sorted alphabetically in the workspace drop down
Changing a linked generator now properly breaks the link
Foreign key columns now show up properly without needing a refresh after setting the primary key table to synthesized mode
Expand long column names on hover in the Privacy Hubs audit trail
Workspaces with broken connections to their database can once again be edited
Views no longer show in the list of tables for Oracle databases.
Features
Workspace sharing - Allows sharing workspaces between multiple users with different access controls (This is an Enterprise Plan feature)
Added 'In' operator to the Conditional Generator.
Added environment variable to bypass certificate validation when connecting to databases with self-signed certificates.
Schema change detection will now flag columns with null generators that have been made not-null.
Performance
Significantly improved the speed that information is gathered about the source database affecting overall UI performance and schema change detection.
Added additional diagnostic information around query execution times to the logs.
Bug Fixes
Fixed issue where switching table mode to synthesized mode in Database View would not work.
Increase accuracy of error messages on SQL Server when tables fail to get created.
Integer Key generator in Oracle for Number columns now base off of the precision, stopping overflow
Resolved name collision when subsetting with a table that has more than one foreign key to the same primary key.
Fixed preview on tables that use the sequential integer generator.
Changing synthesized row counts from database view now saves properly.
Popover of column name on table view no longer flashes.
Large constant values no longer cause display issues in Database View.
Fixed issue with connection test sometimes failing when using MySQL.
Character Scramble now preserves null values when operation on JSON.
Fixed OID collision issue on Oracle.
Preview no longer fails when using complicated partitioning strategies.
Constant generator no longer loses preview when you click away.
Features
ASCII Character Primary Key generator that supports a wider range of characters in the column.
New API Endpoint for getting information on a single data generation job
Key generators now work on unique columns
Improvements to PII Detection:
- Added detection for columns containing passwords, postal codes, and VINs.
- Reduced false positives for SWIFT codes, ICD9, ICD10, US Cities
Switched to a new MySQL driver
Make XML Mask generator more human readable
Allow synthesize mode on Conditional generator and Unique Email generator
Add event generator in Oracle
Performance Improvements
UI performance for resolving schema difference
Bug Fixes
Better checking of uniqueness requirements for columns
Fixed constraints on large tables timing out in SQL Server
Fixed conversion issue with XML columns outside of subsets during subsetting
Block subsetting with synthesized tables
Do not allow source and destination to be the same in MySQL
Make preview stay up to date when generator is removed from different view
Fixed vulnerability with lodash
Features
Tables on SQL Server can now be updated in an incremental fashion where only the changes since the last generation are processed. More details here.
Added support for customer categorical generator in synthesize mode.
Date truncation generator has been added so you can truncate dates to a specified date part.
Added support for timestamp ranges in PostgreSQL.
Phone number generator now supports multinational phone numbers. The output phone number will match the country/region of the input phone number.
Integer key generator can now operate on an Int64/Long column.
Bugs
Fixed issue where a specifically sized payload could cause the API request to fail.
Geo generator properly passes through null values.
Added back remove table API.
Tables containing unicode characters now work in preserve destination mode on Sql Server.
Improved handling of transaction scopes for SQL Server.
Fixed issue where generators applied through the conditional generator sometimes used their default options during generation.
Generators added through autodetect are now able to be marked consistent/deferentially private.
Resolved multithreading issue using the key generators on tables with linked foreign keys.
SQL Server date columns no longer show time in the table view.
Features
Added ability for full name generators to be consistent with partial name generators.
Generator popovers now scroll after reaching a certain height.
Add Random Double generator.
Add ability to delete Tonic account.
Improved job progress tracking for Oracle.
Allow workspaces to be imported and exported.
Added support for conditional generator on SQL Server image columns.
Negative numbers can now be used in the constant generator with numeric columns.
Added support for timezone arrays in PostgreSQL.
Performance Improvements
Significant performance increase when using the continuous generator.
Bug Fixes
No longer displaying out of date conditional generator configuration in the collapsed view.
Fixed issue with not being able to backspace the constant value on a constant generator applied to a numeric column.
Users can no longer duplicate subset targets on the subset configuration page.
Removed conditional generator as a sub-generator choice when using the XML or JSON Mask generators.
Fixed issue where consistency could not be applied to a generator inside the conditional generator.
Loading indicators on the table view are now more consistent.
Continuous generator now works with smaller partitions when generating statistics.
Job details page now scrolls when number of steps is too large for the screen area.
Fixed issue where adding a custom categorical generator inside the JSON Mask generator could cause an error.
Removing a table from the Schema Changes page now properly removes it from the subsetting configuration.
Disabled including tables out of subset was enabled when subsetting was not.
Fixed broken link for API documents on the Job Details page.
Fixed issue where sequential integer generator didn't reset itself between generations.
Forced browser to get new version of assets after each release.
Add SQL injection safety to generators that partition.
Fixed issue with switching between workspaces on the jobs view.
Moved popovers in database view to not be blocking other columns.
Fix issue with attribute info generator failing on empty attributes.
Stopped job progress from updating after job is complete.
Fixed issue with canceled jobs still running if they were cancelled before they could run.
Features
Added the ability to specify a domain and enable consistency on the unique email generator
Conditional generator can now be used with the Unique Email generator
Added confirmation step for canceling a generation.
Added support for JSON and JSONB arrays in Postgres.
Performance Improvements
Updates to your workspace are now done through JSON Patch.
Bug Fixes
Fixed issue where a table was unable to be switched to synthesized mode.
Fixed issue where min and max on the random integer generator were not editable.
Fixed issue where multiple changes might not be saved properly if executed within a few milliseconds of each other.
Fixed issue where constraints failed to apply to a large Sql server database.
Fixed display issue where the password input appeared to be filled in on the destination database connection screen.
Swagger docs no longer report enum fields as integers.
Fixed issue where edit, copy, and delete workspace buttons where still clickable even when disabled.
Decreased clickable area on consistency and differential privacy switches.
Fixed issue with NaN values in PostgreSQL double fields.
Features
A new subsetting option allows you to process tables that are not included in your subset.
Several Conditional Generator improvements:
- Conditional Generator can now filter rows by a regular expression.
- Conditional Generator now supports the following additional generators: Categorical, Custom Categorical, Alphanumeric Key, Numeric String Key, Integer Key.
- 'Is null' and 'is not null' operators added to conditional generator
Custom Categorical Generator now supports numeric types.
Random timestamp generator can now be added to text columns.
Password managers are now prevented from interfering with the database connection form.
Improved logging for constraint application on SQL server.
Renamed 'private' to 'sensitive' when referring to a column with data that needs to be protected.
Column headers are now red when a column is sensitive but not protected.
Performance Improvements
Workspace updates no longer happen on keystroke and will now wait until you exit the field or popup.
Changed the way the application loads workspaces to increase performance for large workspaces.
Bug Fixes
Fix display issue with Custom Categorical Generator when there were no categories.
Fixed error message when applying random timestamp generator.
Fixed issue with column editor size changing when marking a column as sensitive.
Fixed an issue with xml columns on SQL server
Features
Tables can now be filtered by their current mode in the database view.
Added support for the 'contains' operator on the conditional generator.
Company name generator now supports consistency.
Added version check when editing a workspace to better support multitab/multibrowser use.
Multiple users can now edit the same workspace without worrying about race conditions. Note: this is a first step, more multiuser features are in development.
Performance Improvements
Improved speed of tables in mask mode by fixing issue introduced in v76.
Bug Fixes
Fixed issue with applying a large number of constraints with SQL Server.
Random boolean, IP address, and random integer generators now work correctly with the conditional generator
Tasks that complete immediately now display correctly in the job details page.
Fixed display issue with data preview button on Privacy Hub.
Features
Added a generator for shipping container codes.
Custom Categorical Generator now works on json fields.
Destination database names can now differ from the source for SQL Server.
Job completion time has been added to the Job Details page.
Additional diagnostic tools added to docker containers.
Performance
Column search performance in bulk editor significantly improved.
Removed Intercom
Bug Fixes
Preserving a partitioned table on SQL Server no longer causes an error.
Fixed issue with the Categorical generator when the table is empty.
Fixed issue with the last line break sometimes being removed from the category list on the Custom Category generator.
Fix preserve tables for non-Oracle databases.
Styling fixes for Conditional generator when used in Firefox.
Fixed issue where insert into an XML column can fail on SQL Server when the payload has a large attribute, too many nodes, or too much nesting.
Features
You can now apply generators conditionally based on the values within the column.
Added support for array types in PostgreSQL
Added a Custom Categorical generator, which allows you to specify a list of categories for a given column
Starting a data generation via the API will now return the ID of the job
Primary key generators can now be applied in the bulk edit view.
Added additional information on the job details page and in the logging, especially around subsetting.
Return existing refresh tokens if they exist to better enable multiple open sessions for the same account.
Performance
Improved the speed of the integer primary key generator
Improved subsetting performance when subset targets are referenced by large tables
Bug Fixes
Tonic will now ignore tables created by unsupported database add-ons that previously would require the user to truncate.
You may see these tables show up as a schema change alert. Most common ones are spatial_ref_sys and sysdiagrams.
Skip some the relationship checks needed for synthesis when table mode is not set to synthesize.
Fixed display issue where primary key would show as needing a generator for synthesis mode.
Fixed issue where the table drop down on the subset configuration view sometimes rendered in the wrong location.
Json path generators will no longer overwrite objects or arrays.
Fixed issue where identity sequence was not being set correctly on the destination table.
Fix issue when empty strings are present in a column with the numeric string or alphanumeric string primary key generators attached
Features
Ability to add generators to primary keys
Support hex strings for inserting into binary columns in SQL Server
Bug Fixes
Replaced browser alert with a toast notification when a job fails to be scheduled
Oracle now returns columns in the order of their ordinal position
Adjusted sizing on subset dropdown
Fixed issue where the progress tracker could fail to write events near the end of a generation.
Features
Full Oracle support is out of beta
Users can now generate API tokens to more easily authenticate with Tonic's APIs.
Completion times now show on job details.
Subset configuration area now scrollable.
Added a check to ensure destination version is not older than source version for MS SQL Server.
Security
Upgraded Java version on docker images to resolve security issues.
Bug Fixes
Fixed issue where consistency did not work across multiple workers.
Ensure that subset configuration is validated when the subset configuration page has not been loaded yet.
Fixed issue where Privacy Hub could continue to refresh even after a scan was complete.
Incorrect generators are no longer showing in the MS SQL Server image data type.
Fixed magnifying glass placement on Privacy Hub on field hover.
Fixed various issues with toast notifications. Reset scrollbar position on table selection change in bulk editor.
Features
Improved logging when testing a connection
Added MySql 8 compatibility for our collated queries.
Complete overhaul of progress tracking and the job details page. Job Details now has progress bars, spinners, and various icons to present the current status of a job.
Performance
Index restoration parallelized on MySQL.
Foreign keys added to a MySQL table are now added with 'foreign_key_checks=0'.
Tables marked as Preserve Destination now complete significantly faster on SQL Server.
Security
Added TLS support allowing you to use a custom certificate on the Tonic web server.
Encrypted all traffic between containers.
Update third party packages to fix various security vulnerabilities.
Bug Fixes
Fixed issue in MySQL Restore Foreign Key query
Fixed issue when user changes a JSON Mask or XML Mask generator to other generator types
Tiny Int not treated as Boolean in MySQL
Fixed MySql issue with preserved tables having the same name in different schemas
PII Scanner now picks up on zip code data by column name.
PII detection using lookup tables now removes leading and trailing whitespaces and converts all values to lowercase. This fixes our PII detection for cities and states.
Test Connection button no longer holds on to table locks on MySQL databases.
Workspace Name field now receives focus when workspace editor is opened.
Enter key can now be used to save workspace editor.
Allow UUID primary keys as downstream type in subsetting mode.
Fix issue with GUID primary keys when subsetting is enabled.
Long columns names now render properly on Privacy Hub.
Fixed issue with bulk adding consistent generators.
Account creation no longer hangs when server has no internet connection.
Fixed spacing issue with JSON Mask generator interface.
Features
Bulk operations on multiple columns in the Database View:
- Add and remove generators
- Change privacy status
UI now distinguishes between primary (gold) and foreign keyed (black) columns
Advanced search for columns in the Database View.
Generator select popover on database view is now at feature parity with same popover in Table View.
Ability to manually mark columns as private
Columns with private data that are not protected are flagged as red (Database View only) and have a warning in the generator selector
tonic_worker now has a /health endpoint
Phone Number generator supports integer columns
Differential Privacy warnings now show for Categorical generator when being used on JSON and XML columns
Ability for user to provide their own secret to be used for encrypting both JWTs and database connection info
Ability to change password (under User Settings in the hamburger menu located on the right of the menu bar)
Performance
Preserved tables in MySQL are no longer copied into the preserve schema via SELECT INTO. Instead they are RENAMEd
No longer drop and restore primary keys and indexes in MySQL
Bug Fixes
Better support for 3 byte unicode characters when writing data to masked tables in PostgreSQL
Fixed rare concurrency exception being thrown in progress tracker when jobs are running
Fixed malformed query in SQL Server which is used when subsetting
MySQL handling of character
Additional logging of MySQL index handling
Features
Bulk operations on multiple tables in the Database View
Where clause in subset - in addition to a target %, you can now specify a custom where clause as the target for subsetting
URL Generator is more private
PII type is shown in Privacy Hub
Bugs
Styling issues for Chrome and Firefox
Event Ordering defaults corrected br>
Columns with nothing put nulls gave bad data previews
Features
Health Endpoints /health added for web server and PII detection server
Non-null values are no longer shown during data preview
Sort by column name in Privacy Hub
Ability to delete a workspace
Columns part of truncated tables are now labelled as such in the Database View
PII Detection now uses the Character Scramble on suspected phone numbers
Support for Noda time in PostgresSQL and throughout the product
Other columns no longer shows in the database view
Email generator now has an excluded domain filter
Subset with Foreign Key Upload File
Performance
Database View UI rendering improvements
UI improvements for MySQL databases with 1000s of tables
Ability to process tables in parallel across all databases. This setting is controlled by the TONIC_TABLE_PARALLELISM environment variable
Faster reads from source database. This affects SQL Server, MySQL, and PostgreSQL and should see improvements for reads from the source database across the board
Optimization for SQL Server to improve writes
Better distribution of load to workers
Bug Fixes
Fixed bug in XML generation for Postgres
Reduce rate of false positive when detecting zip codes
Subsetting now properly supports self-referential foreign keys
SQL Server unique identifier columns now properly work with the UUID generator
Privacy hub has better styling for long column names
Fixed issues related to change of focus on Target % input on subset configuration page
Features
Support consistency for the email generator on a unique column
Rename and consolidation of docker images. Please contact support@tonic.ai before upgrading for a new docker-compose file
Massive refactor of subsetting, mostly behind the scenes. User facing improvements:
- Support for all databases (not just Postgres and MySQL)
- Heroku no longer requires a temporary database
- Performance boost
Categorical generator now supports differential privacy (https://www.tonic.ai/post/differential-privacy-comes-to-tonic/)
Bug Fixes
Fix for certain unicode characters in table/schema names
Fix for character scramble on numerical columns
Several fixes for air gapped deployments
Significant performance improvements for the privacy scan
Features
Generator improvements:
- Email Generator can be used on columns with a unique constraint
- Character scramble now supports consistency
- Filename and Email generators make use of character scramble as opposed to character substitution (more secure). They both also now support consistency.
More improvements to PII Detection, specifically speed, resilience, and better name detection
Ability to duplicate workspaces
Show generators applied on column in bulk view when table is truncated
Support for SSH Bastion when connecting to source and output database
Added new environment variable for controlling parallelism for MySQL generation
Bug Fixes
Fixed bad description on FK Relationships upload help text
Fixed layout issue in Replacement Panel
Fixed issue with ICD10 PII detection
Fixed routing issues on Heroku
Features
Support for JSON datatype in MySQL
More easily remove a generator from a column in the bulk view
Improved PII detection
Ability to change table modifiers on Database View
Subsetting preview of included and excluded tables
Performance
Increase speed of restoring indexes for MySQL
Bug Fixes
Fixed bad timestamp for jobs when still in queued state
Fixed bug found when using some versions of MySQL where dropping an index on a FK column results in an error
Fixed description for algebraic generator
Resolves an issue that was causing the job details page to fail to load.
Fixed MySql failing to create foreign keys because the database/schema wasn't specified in all cases
More logging
Features
XML generator - use XPath to target one or more values for a generator. This implementation is very similar to the current JSON generator
Partitioning is out of beta. The Continuous and Event generators can be partitioned by either a Categorical or Passthrough generator
Support for computed columns in SQL Server
Re-sizing of autodetect and bulk view table trees, for long table names
Schema change notifications now use small icons
Active page is highlighted on sidebar and remember sidebar state
Check for correct user permissions on source database for SQL Server when first connecting (MySQL and PostgreSQL already have this)
Gzip or br compress API responses
Performance
Faster schema generation for SQL Server
Only SELECT required columns when gathering rows for data generation
Improved UI performance for SQL Server databases with a large number of objects
Bug Fixes
Properly send ALL logs to the log viewer in the jobs UI
Continue when (most) errors occur while running pre-data script on SQL Server
Fixed issue blocking successful CSV data generation
No longer incorrectly filter out tables in the DBO schema in SQL server
Improved logging around some SMO tasks for SQL Server
Features
New UI and Schema View
Ability to generate in place (MySQL only)
Support consistency across databases and runs via the new TONIC_STATISTICS_SEED environment variable
Performance
Many MySQL performance improvements
Bug Fixes
Fixed potential deadlock issue in MySQL
No longer re-loads session when fetching a new refresh token
Better handling of comments inside JSON blobs that are being masked
Features
Auto-detect now supports booleans
Users can now click on data cells to see large text strings
PII reports
Performance
Additional multi-threading added in various places for data generation
Process generated tables before passthrough to encourage fast failure
New environment variables so users can change the connection timeout, and min and max pool size
Bug Fixes
Support for Constant Generator on MySQL Blob columns
Support MySQL set datatype
Improved logging for PII detection
Multiple fixes involving switching between synthesize, excluded, and masked
Progress tracking added to BigQuery
POST to /api/autodetect/config will generate a default config first if one does not exist.
Multiple other small fixes
Features
PII Detection is now available locally
Column widths on the table are now saved between sessions
Truncating tables is now prevented when it will lead to FK violations when generating data
Ability to more quickly add JSONPaths
Schema Diff now auto-fixes issues with model due to schema changes
Performance improvements to data generation, and JSON Generator in particular, which is now multi-threaded
Bug Fixes
Ability to handle NULL database values in the JSON Mask Generator
JSON Paths that start with the same prefix. For example, $.value and $.value1 no longer cause an error
No longer cutting off list of tables in Auto Detect dialog
Better cleanup of excluded_tables and scaled_tables in Fingerprint when tables are dropped from DB
Fixed issue preventing data generation on PostgreSQL Standby databases (in other words, Read replicas in RDS nomenclature)
Features
Can now press Enter on any input in the Database connection form and it will submit the form
Bug Fixes
Better layout for JSON Paths with long names
Fixed a 'missing key' prop issue in front end
Adding too many escapes to MySQL data when writing to CSV (prior to upload).
We now automatically truncate MySQL tables before data generation begins
Features
JSON generator now supports consistency
Better JSON Mask UI
Name generator now has consistency option
Can run auto detect without sending logs
API documentation
Better tracking of Allos Console logs
Better progress tracking
Bug Fixes
Intermittent failures of data table are fixedGetAllTablesAndColumns no longer fails when table has columns with multiple constraints on them
No longer front end crash when subset target table no longer exists
Use Custom Data button location has been fixed
Performance improvements
Fixed a myriad of issues in the First Connection Experience when using a custom data source
Additional fixes for the JSON Mask Generator
Fixed issue created in v16 where changing a table's mode to 'TRUNCATE' wasn't being saved properly
Features
Insight into whether database generation job is running or queued (was not distinguished in prior versions)
Ability to cancel a currently running database generation job
Ability to assign generators by jsonpath for the JSON Mask Generator
Tonic now prevents users from entering identical source and destination database information
When connecting to a database Tonic now defaults the Port to the standard port for the database selected.
Tonic now checks if the database account has necessary permissions and warns if it does not.
Subset button in header now reflects the state of your subset configuration
Google BigQuery now allows user to specify an input and output Dataset for generation.
Bug Fixes
Better escaping of schema and table names in mysql
Workspace edit dialog file upload inputs (Foreign Key upload and BigQuery Service Account upload) were out of sync
No longer show stale data in the table UI when switching between tables quickly
We were not properly handling queries to pg_catalog tables where columns had recently been dropped from tables
Fix to allow synthesizing on MySQL tables that have weird characters in table name
Fix error message that occurs when user checks permissions on a PostgreSQL database with 0 tables
Fix to properly keep in sync the source and destination database names in SQL Server
Constant no longer appears twice in the generator list for JSON columns in PostgreSQL
Table search dropdown now clears search query once user presses Enter or closes dialog
CreatedDate in allos_db tables now has correct values by default
Ability to Filter data table based on SQL WHERE clauses
Support for Google Big Query
Continuous generator now supports nullable columns
Remember last visited Workspace and Table and navigate directly to them when reloading page
Added on-premises installation option of one Docker image
FK Columns are greyed out in UI
Several additional bug fixes
Note: We've transitioned from major.minor.hotfix to simple integer-based versioning
Features
Autodetect Generators (beta) - this feature that scans the source dataset and by analyzing both the structural properties (data types, column names, foreign key constraints) and the content of each data field it takes a first pass at picking generators and linking generators.
Ability to define custom foreign key relationships, this addresses the issue where a database doesn't have any FK constraints or it's missing some
Algebraic Generator - when you link 3 or more columns (A, B, C, ...) with the algebraic generator applied it searches the space of functions (A + B = C, A / B = C, ... ) to find the function that best describes the algebraic relationship between the linked columns.
We now support the ability to specify via an environment variable which schemas to include
Email generator now supports a custom email domain
Renamed Gaussian generator to Continuous
Added TIN generator
- First two digits are always 00
Upgraded pg_dump to v11
Features
A handful of generators now support a notion of consistency across the database. In short, when consistency is turned on for a specific generator, the same input column will map to the same output column across an entire database (where it's turned on). Consistency can also be used to preserve the cardinality of the source dataset in the generated data.
We now use row count instead of scale factor for synthesizing data
Synthesize mode now supports starting with a table that's empty
Better handling of DateTimeOffset data type in SQL Server
Added hostname generator
Re-factored of how generators are executed during data generation
We now show a tutorial video the first time a user logs in
Renamed table mode Excluded to Truncated
Build scripts now build win10-x64 assets
Changed the (!) failure icon in the jobs dropdown to a button so it's more obvious
UI and server versions now show when you click the Tonic logo
CSVs with headers now map to column names
Bug Fixes
No long require a file to exist for the google_application_credentials env variable
Couldn't generate data if we used the MAC address generator with colons
Support for MySQL datetimes with 0000-00-00 00:00:00
Added walkthrough tutorial and demo dataset for new users
IP Generator now uses 100% IPv4 by default
Fixed bad UX with Min/Max in Random Integer generator
Improved Gaussian Generator performance
Added user menu
The distribution of nulls in the source dataset is now persisted in transformed columns in the generated dataset
Added SSN Generator
Improved Address Generator (Zip-> City->State hierarchy preserved when columns using the address generator are linked)
Support for CSV files as a datasource
Support for arrays in Postgres
Improved PII detection
Support for Postgres databases that don’t have passwords
Renamed several generators
- Categorical —> Shuffle
- Random String —> Hash
- String Mask —> Character Substitution
- Text Mask —> Text Scramble
Added generator description callouts
