tekBlueprint

A short article about data anonymization methods. During development, it’s often necessary to test new system versions on data that closely resembles real data. In most cases, surrogate data doesn’t cover all possible use cases, leading to errors “leaking” to production.

This article will primarily focus on data cleansing at the DBMS level for use on test servers. However, other use cases will also be briefly touched upon. There are tools for processing, for example, file artifacts, but this is less common and can be found separately. The list of possible anonymization tools is likely incomplete, as I primarily used Greenmask. These are primarily commercial solutions, but there are also free and open-source options, which is important if you need to audit the code for the tools you use.

What is data anonymization?

Anonymization is the process of transforming data so that it is no longer associated with specific individuals or legal entities while still retaining its usefulness for specific purposes (testing, development, analytics).

The table below lists the options for data anonymization.

Why is data anonymization necessary? Besides setting up test environments, there are several other reasons:

1. Setting up a development and testing environment: Using realistic but anonymized copies of production data allows developers and testers to work effectively without risking real information leakage.
2. Protecting privacy: Preventing unauthorized access to personal information, personal data, personal information, financial, and other sensitive data. This data must be hidden and should not be accessible to the general public.
3. Compliance with legal requirements: Ensuring compliance with GDPR, CCPA, Federal Law 152, HIPAA, PCI DSS, and other regulations. This applies more to data transfer between systems, including geo-distributed systems, and to a lesser extent to test environments.
4. Data for analysis: Providing analysts with data for building models and generating reports without compromising privacy.
5. Reducing the risk of data leaks – even if the environment is compromised (for example, if a testing server is hacked), anonymized data does not pose a direct threat.

Not all data is equally sensitive to leaks. The following are critically important for protection:

• PII – full name, passport data (series, number), SNILS, TIN, email, phone numbers, residential addresses, photos, biometric data.
• Financial data – bank card numbers (PAN), IBAN, account details, credit ratings, transaction histories, employee payment amounts, etc.
• Medical data – diagnoses, medical histories, treatment data, test results, insurance information.
• Corporate confidential data – internal employee identifiers, database schemas, system configurations, information constituting a trade secret.
• Credentials and keys – passwords, API keys, access tokens. Storing such data in open form in a DBMS is already a clear violation of information security, but it occurs occasionally.

Main approaches to anonymization

The choice of approach depends on the purpose of using the anonymized data, the level of protection required, and the organization’s willingness to support such an approach within the allocated budget. For example, dynamic anonymization requires intervention at the DBMS level, which is not always possible. A simpler alternative may be static anonymization.

Static masking

A static, anonymized copy of a data set (usually from a production environment) is created for use in test environments. The anonymization process is performed once or periodically as the data is updated.

Examples:

• Complete replacement of a value with a dummy value. For example, “Ivan Ivanov” -> “Test User.”
• Partial anonymization, with masking of part of the value. For example, card number 9222-2222-2222-2222 -> 9222-XXXX-XXXX-2222, email test@example.com -> t***@e***.com.
• Encryption or decryption with reversible transformation using a key (often used in pseudonymization).
• Obfuscation – hashing, character rearrangement, format preservation with loss of original information.

Dynamic data masking

A more complex option with on-the-fly data anonymization in real time when querying a database or other storage. Users see different data depending on their privileges.

Examples:

• Anonymization based on access rights and roles: when different users (different roles) see different data representations (for example, an analyst sees a partially masked card number, while an administrator sees the full one).
• Anonymization based on context: depending on the context of the request. For example, a request from a testing application or a request from the administrative console.

Pseudonymization

Replacing direct identifiers (e.g., full name, ID) with pseudonyms using a cryptographic key. The original data can only be recovered with the key. Not relevant for setting up test loops, but I think it’s worth mentioning.

Examples:

• Scientific research.
• Collaborative data processing between organizations.
• Long-term data storage for potential recovery.

Anonymization

Irreversible deletion or modification of attributes, making it impossible to identify the data subject without additional information that is not present or cannot be used. This allows for maximum anonymization of data, but at the cost of losing useful information. However, even this option is often better than completely surrogate data.

Examples:

• Replacing identifiers with randomly generated sequences, which can lead to data inconsistency.
• Using metrics to evaluate the degree of obfuscation - k-anonymity, l-diversity, t-closeness, generalization, suppression.

Anonymization tools

As I mentioned above, I primarily used Greenmask. It’s not without its problems, but it’s a good option for a simple and quick data anonymization. For large systems with complex rules, it’s still worth considering commercial solutions or PostgreSQL Anonymizer.

Free and open source:

Commercial solutions:

What to consider when depersonalizing

A preliminary inventory and classification of data by sensitivity level is necessary. For example, according to the CIA matrix (no, not the CIA) – Confidentiality, Integrity, Availability. This is necessary to avoid needlessly depersonalizing non-critical data or missing critical data.

Be sure to document all depersonalization policies and rules for each data type and each environment (Dev, Test, Prod, Analytics).

• Document everything: which fields to mask using which method, who has access to the source data and keys.
• Record all changes to depersonalization policies. Maintain access logs to depersonalization tools and operations.
• Also, restrict access to the depersonalization tools themselves and to the pseudonymization keys.

Set up and automate the depersonalization process via CI/CD. For example, when creating a database copy for testing and ETL/ELT. This will significantly reduce the risk of human error and speed up the process.

Regular testing and validation are essential:

• For static depersonalization, a review of the received data is required to ensure that the copy is depersonalized correctly.
• For dynamic depersonalization, access rules must be verified to ensure that everyone sees the data they should.
• When pseudonymizing, ensure that recovery only works with the key.
• It is important to store keys in secure systems (e.g., HSM) and strictly control access to them with regular rotation.
• For anonymization, it is necessary to assess the risk of data recovery (if possible).

Conclusion

Data depersonalization for the organization of test environments is quite useful for organizing adequate release testing on data that is close to real data. As practice shows, surrogate data does not provide the completeness necessary to test all system use cases. However, the task itself can be relatively time-consuming, painstaking, and require careful work in describing rules and policies. Perhaps this could now be automated using AI. For example, for the initial description of rules, followed by human adjustments.