tekBlueprint

When implementing complex distributed systems, it is often necessary to synchronize data between them. These information systems may be geographically dispersed and have specific rules for transferring data between nodes (filtering, cleaning). Examples include multi-regional ERP systems (large industrial production, resource extraction companies, logistics).

Ведение

Integrating such systems isn’t just a matter of synchronizing operational and master data, but a set of engineering solutions at the level of architecture, reliability, consistency, and compliance.

Message brokers play a key role in this infrastructure–not just transport, but critical components that ensure asynchronous, fault-tolerant, scalable, and observable transmission.бизнес-событий и данных.

Specifics of geo-distributed data synchronization

Below is a comparison of standard on-premises integration with geo-distributed synchronization.

Typical scenarios that can be used when implementing a geo-distributed synchronization model:

• Master-master synchronization of reference books (products, customers, currencies, etc.).
• Transactional replication (orders, balance updates, etc.).
• Hybrid replication models – batch loading + event-driven correction (CDC + events);
• Failover to a backup region with state recovery in the event of regional node unavailability.

All of these scenarios, in one way or another, must be considered when designing the system architecture. It’s important to understand that these approaches are not mutually exclusive and can be combined within a single solution.

Typical architectural patterns

When considering approaches to system interaction, event-driven synchronization is the primary approach, as each node may be unavailable or experience latency. Building interactions based on P2P connections with consistent state is not always feasible. However, for simple scenarios, this can be achieved by building a redelivery mechanism and tracking the state of synchronization nodes.

Below, options using event queues and message brokers are discussed.

Central broker

Features:

• Regional brokers are created in each region.
• Regional brokers synchronize with the central broker (e.g., topics are created in the central broker, which are then replicated to the regional broker).

Pros:

• Simplified topology, as there is only one cluster manager.
• Simplified state reconciliation, as all nodes synchronize with the central broker and its database.
• Ability to operate autonomously for a period of time before reconnecting to the central broker.
• Relatively simple deployment and configuration.

Cons:

• Requires a constant and reliable connection between regions.
• There is a single point of failure in the form of the central broker, which may be unacceptable in some use cases.
• Difficulty scaling the cluster as the number of regions increases.

Decentralized network of brokers (mesh topology)

Features:

• Each region is a fully autonomous broker cluster.
• Synchronization via messaging mechanisms between clusters based on local data consistency rules.
• Each region can contain only the data necessary for its own region. For example, regions can store data only for local users, which is important for compliance with GDPR/FZ-152 and other regional requirements.

Pros:

• No single point of failure. Each node can operate autonomously until the connection is restored and data synchronization between clusters is initiated.
• No need for a centralized control node.

Cons:

• Highly complex deployment and maintenance.
• High network traffic volume between regions, as each node can be an event source.
• Difficulty scaling the cluster as the number of regions increases.
• Complexity of management and monitoring. A specialized solution for monitoring and managing all regions is required.
• Difficulty ensuring data consistency between nodes.

Hybrid approach: event mesh + API Gateway

Features:

• Use of Event Mesh where strict data consistency is not required.
• Integration with the API layer for synchronous requests where strict data consistency is required.

Pros:

• Ability to implement different data synchronization strategies depending on consistency requirements.
• Separation of responsibilities between different system components.
• Flexibility in choosing technologies for different synchronization types.

Cons:

• Complexity of the architecture due to the need to integrate various components.
• Need to manage complex interactions between components.
• Difficulty in debugging and monitoring in case of problems.
• Requires a high level of knowledge of various technologies.
• Difficulty in scaling the system due to the need for coordination between components.

Common problems and their causes

Like any complex system, data synchronization in geo-distributed systems has its own unique characteristics and challenges. Experience shows that all of these challenges are encountered in real-world projects, and this should be kept in mind at the early stages of designing a technical solution. Below is a table listing the main issues and their causes.

For each of these issues, there are several solutions, which are selected depending on the system requirements.

Reliable delivery and fault tolerance

When using message queues, we achieve reliable message delivery, but we have no way to track the message processing status and cannot be sure that the message was processed correctly. This means that we can only track whether a message has been delivered at the transport layer of the message broker itself, not at the business level (entity statuses).

To achieve this, we need to introduce a mechanism for tracking the message processing status. For example, we can use:

• Idempotent event handlers and a deduplication layer. This is often achieved by creating a unique key at the business logic level.
• Transactional Outbox Pattern – a pattern in which we write events to a table (outbox) and then send them to a message queue. This is used for data that needs to be processed within a single transaction and is guaranteed to be sent to the message queue.
• Dead Letter Queue (DLQ) – a queue for messages that could not be processed. Allows you to track which messages failed to be processed. It can also be used for debugging or reprocessing, both automatically and manually.
• Saga – a pattern for managing distributed transactions spread over time. By using compensating actions, Saga allows you to roll back operations in the event of failure.

All these mechanisms allow us to ensure that messages will be processed correctly, even if an error occurs.

Conflict management

In distributed systems, conflicts often arise when two or more participants (nodes) attempt to modify the same data. This creates the need to manage these conflicts. In this case, we can use various synchronization mechanisms, such as locks, transactions, conflict resolution rules, versioning, timestamps, etc.

There are two main types of constraints that help us manage these conflicts:

1. Concurrency constraints – for example, if we want only one user to modify data within a certain time interval, we can use locks or transactions.
2. Consistency constraints – for example, if we want data to be consistent in a certain order, we can use timestamps or transaction logs.

To implement them, we can use the following mechanisms:

• Using CRDTs (Conflict-free Replicated Data Types) - these are special data structures and algorithms that allow us to detect and resolve conflicts automatically:
  • State-based (CvRDTs), which send the full local state to other nodes, which merge it using the change merge function.
  • Operation-based (CmRDTs), which send only the operations (changes) themselves.
  • Delta-CRDTs, which send only the parts of the state that were recently changed, rather than the entire state, for increased efficiency.
• Vector Clocks / Lamport Timestamps - to order events and determine which occurred first or last.
• Conflict resolution at the application level:
  • Last-Write-Wins (LWW) with a timestamp from the ERP.
  • Based on predefined rules. For example, a change from HQ (head office) always takes precedence over a branch.
  • Manual conflict resolution by the user through the user interface.

Minimizing latency and traffic

Minimizing latency and traffic in distributed systems is a separate, important topic. This is especially relevant for unreliable and slow data transmission channels. Unfortunately, this aspect is often ignored by developers, who prefer simple solutions and consider them premature optimization. However, this leads to scaling and performance issues down the road. These issues will be especially noticeable when transferring large volumes of data.

What can and should be used:

• Message compression and binary data representation:
  • For data transfer between nodes, it is recommended to use binary data formats (e.g., Protocol Buffers). This reduces the amount of data transferred and speeds up encoding/decoding (unlike JSON).
  • Data compression at the transport layer (e.g., TCP or HTTP/2) and at the application layer (e.g., Snappy, Zstandard). Here, it will be necessary to select an appropriate compression algorithm and find the optimal balance between speed and compression ratio.
• Grouping messages into a single packet and using delta:
  • When transmitting a large number of small messages, it is recommended to combine them into packets and transmit them as a single message. This helps reduce the number of network calls.
  • Not all data can be grouped, meaning the loss of such a packet will require resending all the data in the packet. Or, the loss of such packets should not be critical, for example, in the case of sending location data.
  • Using delta allows you to transmit only changes, rather than the full data. This is especially effective when transmitting data that changes only slightly.

Observability and debugging

Given the complexity of distributed systems, observability and debugging are essential. This typically includes:

• End-to-end tracing of transmitted events and data – for example, OpenTelemetry, message traceability through ERP → broker → consumer.
• Metrics: message processing delays, replication delays (the full path from source to destination), errors, data accumulation in the broker, etc. Everything that should reflect the system’s state in real time.
• Visualization of the system topology, for example, using Grafana, Prometheus, Jaeger, etc.
• This also includes monitoring business logic regarding message delivery status.

Security and compliance

Distributed systems require special attention to security issues. It’s important to ensure:

• Use secure communication channels (TLS, mTLS, etc.) when transferring critical data between services. Almost all brokers offer the option of encrypting data at the transport level.
• Encrypt the data itself, if necessary, in case of data interception in unsecured systems.
• Audit important events through a separate topic for security events.

Message brokers

Message brokers enable asynchronous data transfer between services. They ensure reliable message delivery even during service failures. This makes them suitable for synchronization between geographically distributed nodes. However, not all brokers are equally useful, and it’s important to choose the right one for your specific needs.

Here, I strongly recommend pre-testing each broker, taking into account data synchronization requirements. At a minimum, the following points should be verified:

• Reliability of message delivery (at-least-once, at-most-once, exactly-once).
• Support for transactions (including distributed ones), if required.
• Recovery in case of disasters (for example, if one of the nodes fails).
• Support for various data formats (JSON, Avro, Protobuf, etc.).
• Support for various replication mechanisms and their reliable operation in unstable network environments.
• Load testing and scalability.

This is important because choosing the wrong broker can lead to serious problems down the road when the system is running in production. It’s always better to test and choose an approach early than to struggle with the consequences of a wrong choice later.

CDC and transactional patterns

Some of these have already been mentioned above. The core idea behind CDC and transactional patterns is that they enable the creation of reliable and scalable systems that can process data in real time and ensure data consistency across system nodes. Here are some of them:

Debezium

Reading from WAL/redo logs of a database (PostgreSQL, Oracle, SQL Server, DB2, SAP HANA via SLT)

Advantages:

• Zero impact on application performance.
• Accurate reflection of the database state at the time of reading.
• ‘Before/after’ support for processing changes.

Risks:

• Requires access to logs (not always possible if deployed in the cloud).
• Potential issues with some column formats (BLOB/CLOB).

Outbox Pattern

The application writes to the outbox table → CDC (Debezium) → broker. A separate CDC block can be eliminated and implemented on the application side as additional logic. However, this complicates the logic when using application clusters, as it requires additional coordination between nodes.

Advantages:

• 100% atomicity (database change + event task in a single transaction).
• Suitable for any database.

Risks:

• Requires application code changes (“invasiveness”). Application customization is not always possible.
• Risk of database locks when writing to the outbox.
• Performance issues with large data volumes. Optimizations to the data model and database storage will be required.

Transactional Outbox + SAGA

Combination: outbox + compensating transactions in case of failure. When using SAGA, in the event of a failure, compensating actions roll back changes made by previous steps. Moreover, if the database has an outbox record, it is always possible to re-execute the event task.

Advantages:

• Everything that the Outbox Pattern provides.
• Support for long business processes (e.g., order → payment → shipment → invoice).
• Ability to roll back changes in case of failure.

Risks:

• Everything that the Outbox Pattern provides.
• Complexity of orchestration, including during failures.
• Complexity of implementation and support.

SynchDB / Materialized View CDC

Periodic synchronization via SQL queries (e.g., SELECT * FROM orders WHERE updated_at > last_sync ). This approach delays data transfer from one system to another, but eliminates the need to build a complex infrastructure.

Advantages:

• Easy to implement in legacy systems.
• Does not require access to transaction logs.

Risks:

• Constant polling for changes → high latency and database load.
• Risk of missed updates if updated_at is not atomic. Can be mitigated by introducing additional audit tables and triggers.
• No tombstone events (deletions) unless specifically implemented. This can lead to missed updates to deleted records. For example, a record with ID=1 was created → deleted with ID=1, then rewritten with ID=1. Deleting ID=1 may be lost if synchronization is not triggered in a timely manner.

Custom log parsing implementation

Direct WAL/Binlog/Redo/Archive Log parsing for PostgreSQL WAL, MySQL binlog, Oracle Redo, and SAP DB6 archive logs. This provides row-level synchronization with minimal latency and high reliability. A compelling reason for implementing your own implementation is required. Not recommended for most use cases.

Advantages:

• Maximum row-level accuracy. Row-level synchronization.
• Suitable for legacy systems without synchronization APIs and customization options.
• Minimal synchronization latency.
• High reliability.

Risks:

• High development and support complexity.
• Requires custom log format parsing and parser implementation.
• Custom support for new DBMS versions and log formats.

Warnings and anti-Patterns

I can’t remember all the possible issues, but here are a few:

Additional recommendations:

1. Don’t skimp on system and data flow observability. Implement end-to-end tracing, end-to-end latency metrics, and data drift (e.g., order_total discrepancies between systems).
2. Test with realistic failure scenarios. Use Chaos Engineering (chaos mesh, fuzzing testing) to simulate gaps between regions, broker failures, high latencies, service failures, etc. Believe me, this will save you a huge amount of time debugging problems in the future.
3. Define and agree on contracts between services. Each event is a contract with an SLA:
   YAML

   event: sales_order.confirmed
   schema_version: 2.1
   sla:
     max_latency: 5s (within region), 60s (cross-region)
     retention: 7d
     delivery_guarantee: exactly_once
     pii: false

4. Plan 5 years ahead:
   • Increase in workload, number of services, and number of regions.
   • Planned upgrades to the tech stack and infrastructure.

Conclusion

Geo-distributed data synchronization isn’t just a matter of technology, but also a matter of balancing consistency, availability, and reasonable complexity. Message brokers can help solve this problem, but they aren’t a 100% guarantee of success. It’s important to consider all aspects: from architecture to SLAs, from scalability to fault tolerance.

And finally, choose a broker that guarantees reliable operation within your specific infrastructure and business requirements, rather than the trendiest and most popular one.