Mockaroo and Tonic - Mocking versus Mimicking Data

Two data generators, both alike in dignity, on a zoom webinar where we lay our scene…

Psych. No ancient grudges here. Mockaroo and Tonic go way back, and we’ve been friends all along.

Join us for a rare opportunity to get an insiders’ look at two of the leading data generation platforms available today.

In this roundtable discussion, Mark Brocato, Founder and CEO of mock data generator Mockaroo, and Adam Kamor, Co-founder and Head of Engineering of data mimicking platform Tonic, will share insights into their data synthesis tools and where both products are headed.

• What challenges led to their creation?
• Which use cases are driving their development?
• What features are on the roadmap ahead?
  

You’ll learn about the use cases best suited to each platform and have the opportunity to ask your own use case questions directly to the founders themselves.

Chiara Colombi 

Hello everyone, and thank you for joining us today for our conversation on data synthesis with the founders of Mockaroo and Tonic. My name is Chiara Colombi. I'm on the marketing team at Tonic. And I'm very excited to introduce you to our speakers today. Mark Brocato is the founder and CEO of Mockaroo. The very powerful very popular online mock data generator founded in 2014. And Adam Kamor is the co-founder and head of engineering Tonic, the data mimicking and synthesis platform founded in 2018. Each of the speakers will take some time to introduce their platforms and key use cases, and also talk about some of the latest features and capabilities that they've released. And then we're going to turn the floor over to you. We're very curious to hear your questions. And to learn more about what your needs are in this space. Please feel free to drop your questions in the q&a at any time, and we will get to them at the end of the presentation. To help set the stage, here's a very high level overview of the key difference between Mockaroo and Tonic, and how that difference defines the use cases for each tool. Mockaroo is a data generator that allows you to create mock data from scratch. So it's very useful if you're building a new application and don't have any data to work with. Or if you need to spin up massive data sets for load testing, or maybe create realistic mock data sets for sales demos. Tonic on the other hand, is a platform that enables you to mimic data in an existing environment. So its use cases include getting production data out of your lower environments to better equip developers with safe data, protecting sensitive data for compliance, data security and risk mitigation, or maybe creating targeted subsets of massive data sets for easier debugging and QA. Now that you've got that bird's eye view of each technology, I'll let the founders go into more detail and show the tools to you in action. Mark, the screenshare is all yours.

Mark Brocato 

Thank you very much Chiara for that intro, you can see my slides, right?

Chiara Colombi 

Yes.

Mark Brocato 

Thank you. I'm very excited to meet some of my users tonight. Hopefully, there are some out there in the crowd. If you don't know, Mockaroo is a website that allows you to generate realistic but entirely fake data in APIs. So what is Mockaroo? It allows you to generate a wide variety of fake data from many different domains at large volumes. And you can also use it to mock your APIs, if you're, like, a frontend developer, while you wait for real APIs to be finished. It's simple to use. And in fact, you don't even have to sign up to use it. This was one of the key design principles when I made Mockaroo; the main screen is not an advertisement. It's just the app. So, many users every day—probably 80% of the users sign up, or don't even sign up—they just use it anonymously. So it's meant to hop on, create data in a couple minutes, and then get out. It's affordable. So if you need to scale up, there are very, very low cost plans for generating pretty massive amounts of data. And it's surprisingly powerful. 

I'm going to go over some of the basic features and then some of the more advanced features to show you exactly how powerful Mockaroo really is. And also it's enterprise ready. I'll get into that a little bit at the end.

If you haven't used Mockaroo, this is the basic experience. You come to the site mockaroo.com, and you start designing what's called a schema which is basically all the fields that you want to generate, in a fake data set that you're going to output to something like CSV or JSON. So you go through and you pick the different types for your different fields. There's 145 different types, all sorts of different data domains, from very simple to very powerful. So once you're done creating your fields and picking your types, then you can preview a small subset of the data that it's about to generate. And so that can be displayed as a table, or if it's JSON, as an array. And then if you're happy with that data, then you can download the full set. And that's the basic experience that probably 80% of mockery users use every day.

As I mentioned, 145 different data types, 10 output formats, most popular being CSV, and JSON, but there's quite a few others as well. There's up to 100, or up to 1000 records per file on the free tier, and there's no limit of the number of files that you can generate on the feature. That's one of the most popular questions, because the limits are so high. So you can generate, you know, 100 files in a day if you want, each of them 1000 records, and not have to pay anything. If you want to scale up, you can generate up to 100,000 records per file, again, with no limit on the number of files for just 50 bucks a year. And then to get really big, you can generate up to 10 million records per file for $500 a year.

Everything I've shown you so far is what's above the water in the iceberg, but Mockaroo is actually surprisingly powerful. And there's a lot of rich enterprise features that are available even to free users. So let's dive into what's below the surface. So one of the powerful features is the ability to mock APIs. So like if you're a front end developer, and you want to actually start working on like your network code refreshes, but you don't have a real API that you can mock that out using the schemas that you create in Mockaroo. So essentially, you create a schema, and then you use, you create an API endpoint to return random data from that schema. And you can also compute responses based on query parameters, and the request body as well. So you can make your API endpoint responsive to the input. And there are limits on that as well, you can get 1 million API calls per day for just 50 bucks a year. So that's the silver tier, and 10 million API calls a day for just $500 a year.

Before I get into some of the more powerful features, I want to talk about the origins of Mockaroo, and why I created it to give you kind of perspective on it. So it started many years ago, I think, was probably 2013, or 2014. And I was leading a development team. At a small Life Sciences startup, I probably had eight or 10 developers and three or four QA people. And the problem with that company was that we built apps for very complex workflows in like clinical trials, specimen management. So the domain was pretty complicated. The testers didn't really know the domain that well, they were pretty Junior employees. And so a lot of times they would fill the test application was just a lot of like garbage data. And it made testing intricate workflows very difficult. And so I built Mockaroo as a tool to help us do better software testing. And today, probably a big chunk of the users of Mockaroo are either QA or developers doing testing of their own work. It started out as just an internal tool, I thought it might be useful for others. So I just put it up on the internet. It was free, it had very few features. But almost immediately, it started getting used by quite a lot of people because it was better than the other simple apps that were out there at the time.

It didn't really dawn on me to create a business out of it until opportunity knocked or rather it actually called while I was on my honeymoon with my wife. I got a call, I think I was in Italy at the time. And it was from somebody very high up at Salesforce. And they were working on a kind of a stealth mode big project that actually turned out to be Salesforce wave, which was their new analytics platform. And that was a big deal for them. They actually bought a Superbowl ad that year; it was a huge launch. And they basically needed to simulate trends in fake data so that the analytics platform could find those trends when they were demoing it to prospects. So the problem for me at the time was that Mockaroo was built to generate hundreds of records, and they needed to generate hundreds of millions of records in a single file. So like any good startup engineer, I told him, oh, yeah, we can totally do that. That'll be no problem. And then I scrambled for the next few days after I got back from our honeymoon, trying to figure out exactly how that might be possible.

Eventually, what wound up happening is, you know, I made it work. Salesforce became the first customer of what would become Mockaroo Enterprise. mocker was kind of off and running as a business. And I started adding a lot of really powerful features. So I'll go over those.

Now one of them is the ability to ensure data integrity when you're creating mock data. So you can upload your own data, whether it's real or fake as a data set. And then you can reference that data when you create your schema. So here I'm uploading a list of contacts. And I'm pulling in the email field from the contact list as one of the fields in some schema that I'm creating. Another way that you can ensure data integrity is to derive values from other fields in your schema using formulas. And formulas, if you had to learn one advanced thing about Mockaroo, formulas is the best thing to learn. Because basically, you can calculate data based on other columns. And you can basically create any type of data that you need if one of the 145 data types doesn't suit your needs. Here's an example where let's say we're generating event data, and you have a start time and an end time for the event. You could generate the start time randomly using the date time field, but then use a formula to generate the end time, which would be some random number of minutes after the start time.

Formulas can also be used in line, as well. A lot of times you'll find a data type that almost matches what you need, but you need to tweak it a little bit. And so that same formula syntax can be used on each field to just change the value in some way, or maybe add some value from another field alongside it. So in this case, I'm generating company names, but for some reason, I want the suffix Inc, on the end of every company name. So you can do that kind of a thing with inline formulas.

Another thing you can do with Mockaroo is simulate trends. This goes back to the original Salesforce use case, but it's one that's become very popular. If you upload a dataset, or you create just a simple custom list, you can control the distribution of how values get generated from that list or data set using a custom distribution function. So here, I created a simple list of colors. And I'm making a list of basically favorite colors for people. And so assuming I have like gender field, and I want to say, maybe men have a stronger preference for red, and women have a stronger preference for blue, you could create this kind of truth table and control the distribution that way. So that's how you create trends to mimic reality for discrete data.

Then if you want to do the same thing for numeric data, you can do that using scenarios. So here's an example where I'm modeling the changes in gas price through the seasons, the different months, so more expensive in the winter, less expensive in the summer. Basically, I'm tweaking the mean of the normal distribution ever so slightly for each month to better simulate the real world fluctuations in price 

Mockaroo, when you first come as a new user, you'll see it export data as CSV, but it also handles JSON data really well, as well. You can generate arrays and objects, and objects with arrays and nested objects, and basically anything that you want to generate in JSON, you can do that in Mockaroo. And that's probably the number two most popular data type. You can use that ability to generate JSON to create mock APIs, as I touched on before.

Here's a really simple example of a mock API. Let's say I create a schema called recipes. And I want to create an API for my Index page. You set the method, the URL here, and then you have a little handler script, where you basically pick the schema, and you generate a number of records from that schema. And then you can also parameterize that schema, as shown in the example here. So if you want it based on like the input query parameter, if you wanted to slightly tweak some of the ranges that are generated from the data, and that schema, you can do that as well. And that creates an API endpoint, with a specific route on my.API.mockaroo.com. So basically, it's meant to model your own API, but with the hostname just changed.

There's even more than that I won't go into in full detail. But some of the highlights are many different fields in Mockaroo are automatically related. There's a bunch of different location fields, city, state, zip code, country. When you generate records from those fields, they'll all match up. The city will be in the state, and the state will be in the country, etc. You can share your work with teammates using the projects feature. And there's a pretty highly-frequented community forum where people ask questions and get answers all the time. And the API not only allows you to create these mock endpoints, but there's a second API that allows you to automate data generation. So anything that you can do through the frontend, you can also do through the API, as well if you want to do it in a completely headless manner.

Last I'll touch on Mockaroo enterprise. The idea behind Mockaroo Enterprise is to remove all the limits by allowing you to host Mockaroo in your own private cloud. There's two reasons why you might want to use Mockaroo Enterprise. One is you just want to generate massive amounts of data that's more than I can give you on the SaaS version of Mockaroo. Or if you have particular security concerns where you don't want anything to be available on the public Internet. 

Mockaroo Enterprise allows you to run Mockaroo as a Docker image in your own private cloud, like AWS, or Azure Google Cloud Platform, it has no limits of any kind on users, on data, or anything else. So it's basically, you pay one flat fee, and everybody in your company can use it to their heart's content. That flat fee is $5,000 per year. And it scales horizontally to generate massive data sets. So the more CPU instances you throw at it, the more virtual machines you throw at it, the more it can generate really huge datasets. People are out there literally generating terabytes of data with Mockaroo right now. And it writes directly to AWS S3. So if you're using, you know, one of their scalable platforms to ingest and do things with data, Redshift, or Spark, or something like that, AWS S3 is a very convenient place for that data to be dumped for you to work on it.

Lastly, what's coming next with Mockaroo. Specifically, with Mockaroo Enterprise, there's going to be a an often requested feature to basically be able to generate a reference dataset directly from a schema at scale without having to download it and re upload it. And this will be done without any limits on dataset size. So right now, I think the data set size is something like 30 megabytes, the maximum size that you can upload as a reference dataset. But at least starting with enterprise customers, you'll be able to generate an unlimited size of data as a reference data set, and then be able to randomly pick from that in future schemas. So this will allow you to simulate key relationships between multiple huge datasets. Look, my Twitter handle didn't show up. There it is. This is my contact info. If you want, you can follow me on twitter @mockaroodev. There's the website, and there's the email. I'll give you a second to note that down. But with that, I will turn it over to Adam at Tonic. Thank you very much.

Adam Kamor 

Hi everyone, my name is Adam. I'm the head of engineering and also a co-founder at Tonic. While I'm getting my screen shared, Mark, let me just thank you for giving that presentation. It's really impressive to see everything you've done with Mockaroo. And it's really clear why you have so many fans in the audience today.

Mark Brocato 

Awesome, thank you very much.

Adam Kamor 

Certainly. Alright, everyone. So like I said, my name is Adam. I work at Tonic. Tonic is the fake data company. We generate fake or synthetic data sets for our customers. A key difference though, between Tonic and Mockaroo is that Mockaroo is more suited for generating data from nothing. So you have a schema or a structure to the data that you think you need, but you have no data. You go and use Mockaroo to generate some data. Tonic works a little bit differently.

With Tonic, we take an already existing data source, for example, a copy of your production database, and we generate a new clone of that database that is devoid of any sensitive information or PII. The way we do that is by generating a new set of PII that look and feel exactly like the original data, but it's all fake. With a Tonic generated database, you'll be using that database and you won't even know it's fake data. It'll look, it'll feel exactly like the original, it'll preserve all of the same statistical properties that you care about. And it'll have all the same schema and structure, primary keys, foreign keys, unique constraints, indexes, etc, as that actual original production database.

Once you've generated your data with Tonic, you can go and directly connect your application to Tonic's output database and get to work, you know, doing your development and testing cycles. A little bit of history on Tonic: we've been around since 2018. We're based in Atlanta and San Francisco. On the left hand side, you can see some of the customers that we work with. We work with a lot of companies across a wide range of industries, from technology to healthcare, to finance to education. We're backed by a great group of investors such as GGV, Bloomberg Beta, SVCI, and Xfund. And our mission is to protect the privacy of individuals while enabling companies to get things done. So with that being said, let me give you a quick demo of tonic. Excellent.

We provide the connection details to the database similar to connecting to any SQL client. The one difference in Tonic, however, is that we also need to specify an output database. Once we're done applying the transformations to our data set, this is where tonic is going to put the clone of that production database. So wherever this database lives, that's where our synthetic version of the database is going to live. And this is what we would connect our application or staging environment or development environment to when it's time to do our job. In addition to Postgres, we actually support a variety of other databases. We support all of the popular relational database systems such as Postgres, MySQL, SQL Server, Oracle and DB two, we support a variety of data warehousing technologies, like BigQuery, and Snowflake. Amazon Redshift support is also coming soon. And we also support operating on a large number of flat files sitting on file systems like s3, or Hadoop file system or Databricks file system, via connecting to Spark clusters directly, either through Amazon EMR or through Databricks.

Is everyone able to see the Tonic application? Oh, Mark, are you able to see it? Great. I had to ask. Okay, so this is what it looks like when you first log into Tonic as everyone can see. I'm actually logged into Tonic's hosted environment right now. But almost all of our customers actually run Tonic in their own public or private clouds. Installation is really easy and straightforward. And we typically have our customers up and generating data within a few days. So I've connected today to a Postgres database sitting in Amazon RDS. This is our real production database. Of course, for the demo, it's not actually real data. But this for the demo purposes is where the real data exists.

So for today, we're connected to this Postgres database. Like I said, when you first log into Tonic and connect to your database, the first thing we'll do is run a privacy privacy scan of your data. And that's where we find all of the sensitive information that exists in the database today. And that's really important for our customers, because most of our customers are connecting to Tonic with databases that have hundreds of 1000s of tables, or 1000s, or 10s of 1000s of columns. So they're really massive, complex schemas. And it's not really possible to know where all the sensitive information is. So as a way to bootstrap our customers, and to make the experience more enjoyable, we scan the database first and tell you where all the sensitive information is. And this is where you'll spend most of your time. Since this is where the sensitive data is, this is where you'll spend most of your focus.

So we've identified things like name, gender, marital status, your occupation, your address, etc, as things that are sensitive. So let's take a look at this first name. These are actually values coming from the database. Clearly, those are first names. Tonic suggests we use a name transformation or a name generator. So let's apply that. Great. Once it's applied, it disappears from the list, because we can really think of this list as our to do list. These are the columns that are sensitive that we haven't yet protected. You know, I see marital status here, I'm actually not sure if that sensitive or not. So let me reach out to someone else in my organization. Um, oh, sorry. It's always hard to type when people are watching you.

So I'm going to ask Craig, who's a member of my organization is this column actually sensitive? Craig's in the security org, he knows you know, the requirements of what can and cannot go into our staging environment. So when he gets back to me, I can act on that column accordingly. Let's pretend Craig did get back to me already. And he said, "You know what, marital status isn't sensitive." In that case, I'm just going to remove it from my list and not do anything. Additionally, we have other columns that we flagged as sensitive, but we don't show them all in this list. So let's go look at all the other columns that have also been flagged as sensitive in our database view.

But before I do that, I'll quickly point out that as I'm making changes to this workspace, and as I am applying transformations to columns, we're generating an audit trail that tells you who has modified what when they modified it, and exactly what the modification was. This is really useful for reporting purposes down the road.

Let's look at all of the columns that contain sensitive information, we do that by going to the database field. In the database view here, you can actually see your entire database schema. On the left hand side, we have all of our tables. On the right hand side, we have all of the columns in the database. Of course, for a demo, we're using a fairly simple database with only 11 or 12 tables in it. But like I said, most of our customers come to us with hundreds or 1000s of tables. And it can often be across many different databases. So this, this view can get, you know, much more complex. But because of the utility, or because of the user experience we provide here, it's actually really easy to apply to transformations still.

So for example, let's remove these filters now, which are filtering just to my sensitive, not protected columns. So as soon as I removed the 'not protected' filter, we get back the name column, the first name column that we had protected earlier. And now I'll remove this filter as well. And we can just see all of our columns. So here's that name column. Let's actually look for more first names. Okay, so a lot of our other columns also have first names in them. I'm actually just going to bulk select all of them. And I'm going to apply the name generator to all of them all at once. So just like that all of our first name columns are protected. Let's take a gander.

Yep, you can see in the left hand side is the original set of names and on the right hand side is the synthetic or fake set of names. Notice that there's a problem, Melissa appears twice in the left hand side, but each time is mapped to something different: Kristen, and then Lena. So, in some situations, that's what you want. In other situations, though, consistency is crucial. Whenever you come across the name, Melissa, you want it to be mapped to the same fake entity. So for example, let's say you have some data sitting in a Postgres database, additional data sitting in your BigQuery, data warehouse, and even more data sitting in flat files in Amazon s3, you want all of that data, once it's been synthesized, to still link up together so that all of the properties and relationships remain right? Well, using data consistency is really important there. So you want to ensure that Melissa always gets mapped to the same big entity.

Well, that's really easy to do, I'm just going to turn on consistency. Let me go back here, you'll see Melissa gets mapped to Rosella, consistently now. And that happens within a column, across 10 columns in the same table, across tables, and even across databases, even when they're different types of databases. We've found that our customers really enjoy using the consistency feature, and it's one of the really important features within Tonic. Great. 

We can continue operating on columns in this bulk manner in the database view. But let's actually go down a level deeper. We're now going to look at just the data in one table. In this case, it's the Customers table. We've already applied that first transformation, it's there. And as we go through this table, we can apply additional transformations to further protect the data in the table. And remember, the ultimate goal is to end up with a new database devoid of sensitive information, but it looks and feels real, so we can still use it in our development and staging environments. So of course, I should protect my last name and gender column. So let's do that quickly.

We'll apply a last name generator, and we'll also make it consistent. And now we get to gender. Gender has been flagged as sensitive. And we know that because there's a red column header right here. So what I'm going to do on the gender column is apply a categorical generator. What the categorical generator is going to do is it's going to look at the percent of all values that are m, f, and null, and then it's going to generate a new column of values that match those percents. So if the original source column is 40%, male, then the output column will also be 40% male, because the goal here is to generate a new column that's fake, but it looks and feels real.

Additionally, this transformation supports differential privacy. Differential privacy is a mathematical framework that gives you certain guarantees around mathematical transformations meant to preserve privacy. In this case, it helps you as the user control the level of privacy and utility trade off that you're choosing when you're generating data. So let's keep differential privacy on as we go through Okay, you know, I see marital status, I actually think that is a sensitive column for our use case. So I'm going to flag it as sensitive, so we make sure that we're tracking it properly. And I'm going to apply that same categorical generator where we're shuffling values around.

But it turns out in this data, marital status and gender are related. Men are more likely to be separated than women in this specific data set. So to preserve that relationship, I'm simply going to link the columns together. When you link columns together in Tonic, all you're saying is, hey, Tonic, these columns are related to each other. I'm not going to tell you what the relationship is, but Tonic will figure it out and then preserve that relationship in the output database.

Moving on, we come across a few additional columns that were flagged as sensitive: income and occupation. Let's see what we can do about that. Well, for income, I'm going to use a continuous generator. And I'm going to turn differential privacy on for this generator as well. A continuous generator looks at the distribution of values in the column and generates a synthetic column that has the same distribution. However, every new value in each row is randomly drawn from that distribution. So if you look at a synthetic value, you won't be able to tie it back to the original value.

It turns out, though, and in the real world, this is also true, that your occupation affects your income, right? Doctors, on average, make more than hairdressers, right? So we want to preserve that relationship. I'm going to partition my income by my occupation, and what that's going to do is it's going to generate an income distribution per occupation. And when Tonic goes to generate data, it's going to first generate the occupation value, it's going to look at it, and then it's going to generate an income value based off that occupation. Great, but occupation is also sensitive. So we do need to protect that as well. Let's apply a categorical generator to occupation, which is going to sample those values around. And we can even link it back to our gender and marital status linkage that we made earlier, because both of those columns also affect your occupation in this dataset. So let's do that. And just like that, we've generated a fairly complex model involving several categorical and continuous columns. This synthetic table that we generate from this is going to strongly resemble the original data, but it's protected and it's safe and you can use it in your lower and staging environments, without fear.

Tonic has a lot of transformations to choose from, in varying orders of complexity. The continuous generator, clearly very complicated. The first name generator is less complicated. But when you throw in consistency, it's actually fairly complex as well. So we have transformations for many different use cases, I don't have time to show them all to you today. And in fact, I'm going to go on to a few other areas of the product.

Let's pretend like we finished protecting this database, it's now time to generate data. When I generate data, Tonic is going to take the transformations I've selected, it's going to copy the schema from the source database, apply it to the output, and then it's going to move data from source to output while applying the transformations that we've selected. The output results, or rather, the end result is going to be a new database that we can immediately start using in our lower environments without privacy fears, and our developers can use these databases to get their jobs done.

This job is running right now. I can go here and actually watch the status of things that are happening. And the job actually already completed. This job completed relatively quickly, and the database is actually now ready to go, we can connect to it. We're not going to do that today, though, instead, we're going to look at a few other features too.

One really popular feature in Tonic is subsetting. With subsetting, you can actually generate a smaller version of your production database. This is a really popular feature in Tonic simply because a lot of our customers come to us with incredibly large databases. You know, it could be 100 gigabytes, on the medium size, to multiple terabytes or even more on the large size. And sometimes you don't need all that data in your lower environments. Or perhaps you're trying to generate a database that can be used for local development purposes. It's going to be running on people's laptops; you can't put a five terabyte database on someone's laptop, that's simply not going to work. But with subsetting, you can generate a statistically good and referentially intact version of your production database. And all of the transformation rules that we applied earlier will be applied on top of your subset of database. So let's see how subsetting works.

Well, first, we enable it, and I need to select first a target table. In this case, I'm going to use the Customers table because we're already familiar with it. So I need to choose either a where clause to filter this table down, or just a random a target percentage. Today, I'm going to use a target percentage. So that's telling Tonic to sample from 10% of all users or from 10% of all customers, excuse me. And that part's easy. But where it gets complicated is Tonic then looks through the dependency graph in your database, following all of the foreign key constraints from the Customers table to all of the other tables either directly or indirectly. And it opportunistically is deleting rows from those other tables, such that we don't break our foreign key constraints. And then the output all of this is a smaller database, that still satisfies all the constraints of the system. It has all the transformation rules applied on top.

So we're wrapping up. But the last thing I want to show you is some of the enterprise features that we have that make using Tonic a joyful experience in large organizations. So everyone saw earlier that I messaged Craig, a question about the sensitivity of that column. Well, how did I do that? How does tonic know that Craig exists? Well, Tonic knows, because Craig is in my SSO workgroup. Tonic can integrate with any of your SSO providers to allow for easy sign on and group control access. And additionally, I can share my workspace with others in my organization. So Craig is actually an editor of this workspace as well, meaning he has the same permission levels that I have. However, if I needed to, I could add additional people here, or I could change Craig's permission level to auditor or viewer depending on Craig's role within the organization and what he needs to do. Today, though, I'm just going to remove Craig's access. Sorry, Craig.

But of course, adding foreign keys is crucial for the subsetting algorithm to work well. Not all of our customers come to us with a database that has all of the foreign keys in the database system. Sometimes they only have foreign keys in their application layer. So in that situation, they can actually in Tonic tell us where additional foreign keys lie. And we will treat those foreign keys as real for the purposes of subsetting and for transformations as well.

So here we can see all of the foreign keys currently existing in the system; these are in the database itself. And then we can add additional virtual foreign keys via this UI as well. Today, we don't need to actually do that because our database does have foreign keys. But oftentimes, your database won't have foreign keys either because it wasn't a good move from the design or architecture decision, or you could be working, for example with a data warehousing database that just doesn't support the notion of foreign keys at all.

Excellent. Well, there's a lot more we could show about Tonic today. But I'd be really excited now, and I'm sure Mark is as well, to hear from the audience about what questions you have for us. So with that being said, I'll hand it back to Chiara.

Chiara Colombi 

Thank you, Adam. And thank you, Mark. We do have a lot of questions. So I'm going to jump right to them. Questions for Mockaroo, questions for Tonic, questions that could apply to both. The first one that came through is does Mockaroo offer real estate data slash addresses per specific state.

Mark Brocato 

Yes. So if you pick the state type, you can actually filter the states down to whatever list you want. And then all of the other location fields will be generated just for that state. And the same goes for country as well. So if you want to just pick a few countries, you can do that too.

Chiara Colombi 

Great. Question for Tonic. Can Tonic identify PII from freeform text? Does the PII flag get set at the field level? Or does it identify PII at the field record intersection.

Adam Kamor 

So Tonic identifies PII at the column or the field level, depending on what database system you're using. Tonic can in some situations detect PII in free text. But we can also integrate with data catalogs. So for customers that already know where some of their PII resides, Tonic can integrate with those catalogs and, you know, treat those catalogs' PII as its own PII as well.

Chiara Colombi 

Great, thank you. Next one for Mockaroo: is there a way in Mockaroo to always generate the same data using a parameter similar to random state in Python?

Mark Brocato 

I'm not familiar with that parameter. You can—so there is with formulas, there's a way to look up from a data set based on a value. That's probably one of the ways that you could do something like this. So if you see that the value of some field is, you know, greater than 10, then always pull this particular record from another data set. I didn't show that. But that's definitely a powerful way to do that kind of data integrity. But I'm not familiar with that one function. So I apologize.

Chiara Colombi 

A question for Tonic about security. What security has been implemented as you're connecting to a client's database? Are there any other ways to read source data? Does it have to be production data? Or could it be fake data?

Adam Kamor 

That's a great question. Tonic can connect to any database, whether it has real or fake data in it to begin with. However, you should keep in mind that Tonic is typically deployed on your prem and it does not phone home. A lot of our customers even install Tonic on completely air-gapped environments that only have network access to other machines within the customer's own network. So if security is a concern for you, hopefully that should help mollify your concerns.

Chiara Colombi 

Perfect. A question about API endpoints for Mockaroo. Can I identify API endpoint locations? If I know that most of my customers are from Europe, for example?

Mark Brocato 

That's a great question. Um, you actually can't. Mockaroo runs on a fairly lean infrastructure. So it's all US west on AWS, at least for now. Maybe as we scale, we'll go multi tenant or multi high availability, but not yet. Sorry.

Chiara Colombi 

Another question, while we're on this subject of the REST API interface, for Mockaroo. I know you can generate REST API interfaces. Are there any plans to implement Streaming API interfaces from Mockaroo?

Mark Brocato 

Yeah, I saw that one of the list. There are no plans, but that's a really interesting question. And I'd love to hear more about that. So if you want to reach out to me, like through the support on the website, or just on Twitter or something, I'd be really interested to hear about your use case, because I think a lot of people would benefit from that.

Chiara Colombi 

Switching back to Tonic, a question about JSON. If I want to create NoSQL data of JSON format in Tonic, what is the step by step process? Can we see that live.

Adam Kamor 

So I'm not going to demo it right now just for the sake of time, but we actually do support Mongo as a NoSQL data source. And we have support for other databases coming out soon. Additionally, a lot of our customers actually store JSON within the columns of relational databases, JSON or XML, for that matter, and Tonic has transformations available on JSON columns. You can really think of it as essentially the user points at various keys in the JSON or attributes in the XML. And then you can, for each of those keys, you can tell Tonic what generator to apply. So it's actually very similar to the table experience I showed you, but you're applying generators to individual values within a JSON blob instead.

Chiara Colombi 

Perfect. There's a question also about flat files: does Tonic support working with flat files such as mass exports of application or log events?

Adam Kamor 

Certainly, we do, typically the way—you have a few approaches we can take there. One and this is more work on the customer side, so maybe you don't want to do it, you can load those logs into a relational database. Alternatively, if those logs already exists in a file system somewhere, Tonic typically will operate on flat files via a Spark cluster. So for example, let's say your flat files are sitting in Amazon S3, Tonic can connect to an Amazon EMR cluster that you'd spin up on your own Prem and then operate on those files that way.

Chiara Colombi 

And another question for Mocakroo: how well does Mockaroo generate and handle hierarchies? And can trends and distributions be applied to them also?

Mark Brocato 

Yeah. So you basically use the naming convention of the fields to generate hierarchies in a hierarchical output type, like JSON or XML. I think Cassandra supports that as well. So you can put dots between fields, and it will generate the hierarchical objects. And then if you use the formula type, you can navigate up and down that hierarchy to compute other values as well using array notation or dot notation to access the object. So the answer should be pretty well. Okay.

Chiara Colombi 

And here's a question for both of you again, about streaming data: is it possible to simulate streaming data in Tonic or Mockaroo?

Adam Kamor 

Mark would you like to go first?

Mark Brocato 

Well, two people have asked that tonight. So by tomorrow, I'll have it done. Apparently, it's a really—I need to learn more about this use case. But I actually just google MQTT, I've never used it before. But I'll definitely look into it. You can't do streaming right now. It's all just rest.

Adam Kamor 

So with tonic, we support streaming in a way. So when we talk about streaming, it's typically meant as a way of saying that you can, you know, take a database and generate an output. But let's say I have new records going into that database every day. And we want to just transform the new rows of data that have been added. So can we stream just the Delta across to the output database every day. And we do support a notion of streaming in several of our database systems, but I would need to hear more about the use case specifics to know if we support it out of the box right now.

Chiara Colombi 

Okay. A question for Tonic: can Tonic be used for database type testing? Let's say I have a production database that is Postgres, and I want to see if going to a NoSQL database would perform faster. Can I port the data from one type of database to another?

Adam Kamor 

Not today, we don't support that. We do have some future work where we're going to support moving from, for example, Postgres into like a data warehousing solution. But we haven't started that work. And I don't envision it moving towards being able to go from like a SQL to NoSQL type database.

Chiara Colombi 

Okay. Back to Mockaroo, do you have examples of geo-socio-cultural data generation? By that I mean, if generating data for a specific geography, can we ensure meaningful and proportionate, consistent person data given name, family name, religion, consistent, etc? I'm conscious of importance when in early interactions in a new account?

Mark Brocato 

Yeah, no, that's a great question. If I could do that automatically, that'd be so awesome. But so difficult. What you can do is use the the trend shaping capabilities that I showed early on in the presentation, to say, Alright, for a certain country, make this religion more common, in another country make that religion more common. So you can do that with that kind of discrete data. But you have to build in those rules yourself. It doesn't understand those kind of sociological differences. That'd be awesome if it could though, so maybe in the future.

Chiara Colombi 

Okay. And another question for Mockaroo: can Mockaroo generate data with meaningful relations and existing types, like healthcare codes, procedure codes, and diagnosis codes, for example?

Mark Brocato 

The diagnosis and procedure codes are related. There's some subfields within those two parts, I think the name and the code that are related, but it doesn't bridge that gap between procedure and diagnosis. So no, you'd have to model that yourself.

Chiara Colombi 

Okay. I've seen a lot of questions about streaming that I think we already covered. Another question for you, Mark. When  you say it will be possible to generate data from a schema, does that mean a JSON schema?

Mark Brocato 

No, right now, the only way is to upload a CSV file. You can't generate from a JSON data set yet, although I think that's probably coming fairly soon, because that is a common ask. And JSON is a pretty popular format. So thanks for asking. And let me put that on the backlog.

Chiara Colombi 

Can you generate phone and address data that actually looks real? It can't be real. I guess that kind of speaks to the question, you know, also controlling for not having the same data, making sure that your data is unique across the data set.

Mark Brocato 

Yeah, yeah. So the location fields, and I think the area code is one of them, they'll all line up. Like obviously, I don't know down to like the suffix and the phone. And the street names are not real. So the street names are just procedurally generated or just pulled randomly but city, state, zip code, area code, all those link up. One of the real common questions is, are those going to be real addresses? And it's kind of a monkeys on typewriters question which is, given a large enough volume, some of them may actually be real, the same is true for emails, which are, they're generated entirely procedurally, they're fake in that they don't have an origin of a real person. But if you pick a real company name or real domain, and you generate msmith@microsoft.com, there's probably a Mike Smith, someone that could get your email if you sent it to that address. So just be cognizant of that. The only way to like truly fake emails is to pick a domain either that you own, or to pick an invalid domain with a top level domain that just doesn't exist.

Chiara Colombi 

So this question is for both of you: do you see any support coming for GraphQL? For both platforms?

Mark Brocato 

Yeah, sorry, I'll take the first real quick. Um, so GraphQL is probably the next API that I'll implement. That one's super popular, so it's high on my list. Sorry, Adam, go ahead.

Adam Kamor 

We don't have plans for GraphQL, specifically at the moment, but I'd be curious to hear what your use case is with it. So if you'd like to reach out to us, you can reach me personally at adam at tonic dot AI or you can reach out to support@tonic.ai. Either way, someone will get back to you really quick.

Chiara Colombi 

Another question for both of you. What if one of your tools doesn't have a generator that works for my datatype?

Adam Kamor 

Sure, I can—yeah, I'll take it first. You can really think of Tonic generators or transformations as building blocks. And we have a lot of very generic building blocks that can be assembled together in ways to really get surprising outputs. And they're actually really powerful. However, there are always situations where, you know, you can't use those building blocks that we give you to get exactly what you need. And oftentimes, we'll work with the customer to either, you know, make our building blocks better or more generic. Or sometimes we'll even add custom generators for specific users and customers. We actually do that on a fairly regular basis. I don't know really, of any customers that need a generator that they don't yet have, I think, which speaks to, you know, how quickly and readily we work with customers to get what they need.

Mark Brocato 

So for Mockaroo, there's two ways. So datasets, you can upload your own datasets. So like, if you find public reference data for a certain datatype that you need, you just upload that into Mockaroo. And then Mockaroo, will randomly pick from that as it's generating records. So that's one for discrete data. And then for anything else that you need to just like create out of nowhere, formulas is the best bet: the formula data type. And I'm pretty sure there's a YouTube video on that, that's linked from Mockaroo. So if you look at the videos, there's a set of maybe six or seven tutorials; I think that's one of the things covered there, basically, how to generate data when there is no data type built in.

Chiara Colombi 

Right, another question came in for you Mark, any chance to be able to export/import schema definitions?

Mark Brocato 

Like a whole database schema? You can't do it now. I can totally see how that'd be useful, especially at scale. That's the thing that Tonic really shines on right now, honestly. And so the folks that come to me, looking to install Mockaroo in the enterprise, when they have huge databases with like, hundreds of tables, I have to send them to Tonic. That's really what they do.

Adam Kamor 

And we're very appreciative of it. Thank you, Mark.

Mark Brocato 

You're welcome. I'm appreciative of =not having the headache of having to deal with that software. Thanks, guys.

Chiara Colombi 

Another question for you, Mark: can you send out details on the computerized responses? And this may have been in follow up to another question that was asked earlier, one big issue is getting the correct responses in an API. Can I state what should be returned in response based on what is intent?

Mark Brocato 

Yeah. So if you read the little help on the right hand side, hopefully that gets you on the right track when you're creating a mock API in Mockaroo. But basically, you can access the request body and the query string. And so there's a way to set parameters that you reference in your schemas based on those things. So if you want to say like, when the ID query parameter is a certain value echo that back out as the ID of the record. You can totally do that. If you want to say like, if country is a certain value coming in on the post body, then use 10 as the mean for this numeric variable, you can do that as well. So go and create a mock API and look at the help text on the right, that should get you started on the right path. If not, you can just click contact support in Mockaroo and that comes directly to me, so I'll answer.

Chiara Colombi 

And this is for both of you: question about generating realistic time series data that follows a chronological pattern. How do your platforms handle that?

Adam Kamor 

Sure, we have several different transformations that can be used to generate time series like data or event data depending on exactly what you mean by time series. The way it works, and I'm going to try to keep it at a high level, but it is a little bit technical, we essentially look at the distribution of timestamps that exists in the source database already. And we simulate a new distribution of timestamps, and then basically try to re-simulate that distribution rather, we create a simulation that will achieve the same distribution using what's called a non homogenous Poisson process. And then you end up with very realistic looking timestamp data that is going to look very similar to your source data set.

Mark Brocato 

On Mockaroo, this is one of the trickier things, and the reason is that Mockaroo is meant to scale to infinity, basically. And in order to do that, you have to generate data in parallel across many different processes that are talking to one another. So the way to do it is Mockaroo will give you a row number datatype, it's actually like the first default data type in the default schema. And you can kind of compute a timestamp based on that if you like. Take the row number and add or subtract a random number of let's say, seconds or minutes or days, from a base date, that you just hard code. That's the way to do it at scale, it is one of the trickier things. So I'm looking for ways to improve that. But that's how people do that kind of time series data and make sure that the data's keep incrementing, as it goes down the list.

Chiara Colombi 

Great. And Adam, did you see the question come through for Tonic? Is there API support for Tonic?

Adam Kamor 

Yes, I forgot to show it during my demo. But everything I did in the UI during my demo can also be done via our API, and Tonic ships with API documentation.

Chiara Colombi 

Another question that just came through, I might have missed it. But is there a capability to have a procedural language, existing or custom, to do really complex generations and or transformations. I need to generate truven like data?

Mark Brocato 

I don't know what truven is. It depends. Everybody has their own definition of really complex. As I mentioned a number of times, the one thing to learn as a kind of a power skill with Mockaroo is formulas. Formulas is a Ruby syntax. And basically, anything you can write in Ruby, you could generate there in line as a formula. So people use that as a stopgap for everything in Mockaroo or anything that you know, is not built in, you generate the formulas. If you want to chat more about your specific use case, contact support, it'll come to me and we can chat.

Adam Kamor 

My answer is very similar to Mark's. Maybe by truven, you're referring to a type of healthcare data? I've heard of truven before in reference, I think to like IBM Watson and healthcare analytics data. But that could be me not remembering correctly. So Oh, yeah, it looks like that is the case. So we actually work with a lot of customers in the healthcare space. And we're very familiar with the types of complexity that can be found in healthcare data, for example, in insurance or in payer data. So if you want to, if you want to reach out to us, you're welcome to. You can reach out at Adam at tonic dot AI. And we can tell you a bit about what we can do in that space.

Chiara Colombi 

Perfect, thank you. Oh, another question for tonic: Is there a limit to the amount of data that can be generated using Tonic, similar to Mockaroo's limit?

Adam Kamor 

No, there's there's no practical limit. Tonic is deployed on your Prem, on your hardware, it can, it can scale both horizontally and vertically. And we have customers using Tonic from the gigabyte to the petabyte range.

Chiara Colombi 

That one we already covered. What is required for on premises deployment?

Adam Kamor  

Is that for Tonic or for Mockaroo.

Chiara Colombi 

It's not specified

Adam Kamor 

I'll go first. Mark, you can go second? I think the answers are actually similar, believe it or not. So we deploy all of our releases to a private Docker repository, and then we provide our customers with either a Docker compose or a Helm chart configuration, so that they can deploy Tonic on their own prem using basically whatever containerization orchestration service they prefer. So we have customers using Docker compose Helm charts, Azure Kubernetes, Elastic Container Service, Nomad and several other technologies as well. Install is very simple with Tonic; it takes maybe 20 or 30 minutes to get set up and going.

Mark Brocato 

Mockaroo is pretty similar. A Docker container; we can use any of those above techniques for orchestrating it. The only thing I would add in addition to that is it also relies on S3 or a compatible system. So like digitalocean has their own implementation of S3. But very commonly, people just use AWS for that. But basically a Docker container and an S3 compatible storage.

Chiara Colombi 

Perfect. We do have a lot of questions around how to get in touch, this might be a good time to just share one last slide with everyone, so that you can see how to get in touch. But if you want to share your contact information as well, feel free. Here, we've got some contact information, the website, email address, and also if you want to connect with us on Twitter, we do have time for a few more questions. So please feel free to keep those coming.

We do have a question here that just came in, what support do you provide? And I think that's for both of you guys.

Adam Kamor 

Go ahead, Mark.

Mark Brocato 

Yeah, yeah, me first, right. So Mockaroo is just me. That's the downside of Mockaroo. So I support it as best I can. But I do sleep. So it's not 24/7 support. But there is a community of Mockaroo users on the community forums that can answer a lot of questions. And you can at least be satisfied that when you click contact support it is going to meet. You can contact me on Twitter as well. But it is a one man show at least for now. I don't have a better answer for that.

Adam Kamor 

Yeah, well, better, I don't know. It's it's slightly different, though. With Tonic, we have various types of support available. With many of our customers, we actually open up a shared Slack channel, and then in that Slack channel, they have direct engineer support to ask any questions they have.

Chiara Colombi 

Great. So we have got a couple more questions that came in, I'll try to get through them all before the hour. Do any customers integrate Tonic with their CI/CD pipelines?

Adam Kamor 

Yes, a fair number of our customers? I would say they do that. And they do that just via our API. Yeah, it's actually what I have found is that, you know, when customers first start with Tonic, they're using the UI a lot. And over time, they use the UI less and the API more as their usage gets more sophisticated and more deeply integrated with the rest of their data system.

Chiara Colombi 

Question for Mockaroo: can Mockaroo create data sets for the healthcare field like medical supplies? The idea would be to start with the data structure and maybe some sample data and then create larger datasets.

Mark Brocato 

There are some medical types in Mockaroo, particularly ICD 9 and ICD 10, I think. But I think that's the extent of it. And then the thing you want to lean on is the ability to upload your own data set. So one of the nice things in the medical world is there is a lot of public data, particularly around ontologies. And then also things like adverse events as well. So if you can grab those things to CSV files, you know, from the FDA, or from NIH, then you can upload those into Mockaroo, and then you reference them in the other data that you're just creating from scratch.

Chiara Colombi 

Okay, another question for you Mark: can you do a skewed normal distribution in Mockaroo.

Mark Brocato 

There's no specific field for that. There are a few different distribution types, but you could probably tweak it with—this would be a great example of an inline formula, the little sigma button on the right hand side of every field, you could use that to nudge a distribution in one way or another, or just kind of tweak the value of the field. So you should experiment with that. I think that's probably the best way.

Chiara Colombi 

More questions coming through. Do you have any performance benchmark for any of these tools? Like how fast it can generate data?

Adam Kamor 

Sure, so I'll answer for Tonic first. The speed at which we generate data is typically constrained not by Tonic, but instead by the hardware that Tonic is running on, especially the hardware that the source and destination databases are running on. Tonic is rarely the bottleneck itself. And when it is, it can be scaled to run on larger boxes or on more boxes. I will say that we have customers generating data across a very wide range, like I said earlier from gigabyte to petabyte. We can in general make any data size work and make it performant and fast for you. So, you know, I wouldn't let the size of the database be a concern.

Chiara Colombi 

Sorry, I'm sorry.

Mark Brocato 

For Mockaroo. It's kind of the same answer. To give you a general idea if you were to use one like eight processor EC2 instance, then you're probably looking at being able to get like 10s to maybe low hundreds of millions of records out in a reasonable time, like minutes to maybe like 10 or 20 minutes. But obviously, you can keep adding more and more and more processors and get ridiculously large, then you just have to pay Amazon. But that's kind of the idea is 10s of millions can be done on one machine. And then if you get beyond that, you're probably multiple machines.

Chiara Colombi 

I think we've just got two questions left. So I think we can run a little over to get them covered. Are there templates or a capability to easily generate data in different formats, e.g. LDIF.

Adam Kamor 

So we don't support an LDIF format as an output, currently. With that being said, whenever we add support for new data sources or formats, it's because it's been requested by a customer. So if this is something that you think would be really useful for you, don't hesitate to reach out

Mark Brocato 

LDAP Data Interchange Format, I had to Google that. So that's not specifically supported. There is a custom output format, which is basically like custom delimiter for flat files. So you can do pipe delimited and other things. And then there's 10 or so built in data types. I think MongoDB is one of them. Cassandra is another one, but they're very common ones.

Chiara Colombi 

Okay, I think we've got a great question to end on, from your fans, Mark. Very curious about the kangaroo logo and name origin in Mockaroo's name.

Mark Brocato 

I wish I had a great answer for that. But it was literally like 2013. And I think it was very popular to have like, a thing and then an animal as the name. And I just thought of it in like five seconds. And it was meant to be a very light hearted thing. Like the logo is actually stolen from the public kangaroo crossing street sign in Australia. So that's an interesting story. But no thought put into it whatsoever. I just like the color scheme and the font, and I don't know, it's whimsical.

Chiara Colombi 

Yeah, I think tonic kind of has a similar origin story for our name. Adam, if you wanted to share that.

Adam Kamor 

Well, I mean, I always joke with people that it's really hard to find good domains that are left. So you're kind of limited in that way. But no, I unfortunately don't have a great story. Yeah.

Chiara Colombi 

Well, this has been great. We've had so many questions. Great to see everybody chime in in the chat. So thanks to everyone for joining us. And thanks so much to Mark and to Adam for speaking at this event. It sounds like there's a lot of people asking for us to have another so we'll see. And we'll keep everyone posted on that.

Mark Brocato 

Thank you, everybody.

Chiara Colombi 

Great. Thank you.

Adam Kamor

Thank you.