Not only do you have PB of data, you have it spread across multiple databases of different types. You have it in NoSQL databases. You have it in JSON blobs. You have foreign key constraints that aren’t always expressed. And your production data, heck maybe even your schema, changes every day. Chances are you’ve even got some time series data in there. And you’ve been tasked with recreating it realistically. Here’s a hankie.
And to think we used to sing the praises of big data. Building a data de-identification infrastructure that works with the scale of today’s data means grappling with performance issues, data sprawl, and PB of data sprinkled with sensitive data throughout. We’re talking fields full of haystacks full of needles, and it’s your job to find them.
