tekBlueprint

A little bit about integrations in the “bloody” and not so enterprise. Basically, this is an overview article without a detailed dive into technical aspects and examples of implementing solutions. I will try to give a general idea of what integration approaches exist, what problems arise when using them.

A general description of EIP (Enterprise Integration Patterns) can be found on the website enterpriseintegrationpatterns.com.

Introduction

Integration is the process of combining different systems and services to ensure efficient data exchange and interaction between them.

If we leave aside the technical details of implementation, we can distinguish several approaches to organizing exchange between systems:

1. Remote Procedure Invocation (RPI/RPC) – interaction where one system calls a method of another system through some API with a known contract.
2. Messaging – interaction through the exchange of small messages between two systems using, for example, message brokers or their analogues.
3. File Transfer – data exchange through files located on a shared network resource or through distributed file systems.
4. Shared Database – exchange data stored in a common database accessible to multiple systems.

Remote Procedure Invocation (RPI)

In most cases, this is a synchronous interaction by nature, a request and a response. Although, after the request is made (the connection is established), one or both sides may continue to work in an asynchronous mode. As an example – gRPC Streaming, WebFlux.

Used in cases:

• When you need to get a response to a request immediately or in a short time.
• When queues are too expensive and complicated or it is necessary to review the architecture of existing applications.
• Almost any client-server interaction for desktop/web applications. Although it is possible to access queues from a browser in the case of NATS, via WebSockets.
• “We have an old system – we won’t rewrite it.”

Examples of protocols:

• REST
• gRPC
• SOAP
• JSON-RPC

What are the advantages of this approach:

• Simplicity, since this is probably the most familiar and understandable approach to calling a function in an external system (P2P), similar to calling procedures in code.
• We receive a response immediately after the request, without the need to use statuses/states for requests and wait for a response.
• Using predefined contracts for APIs that are described in the format of the protocol used, for example:
• OpenAPIv3 for REST API.
• Protocol Buffers for gRPC.

What are the disadvantages of this approach:

• Possible problems with delays (latency) on large volumes of data or high request frequency. We run into hardware limitations of the network, processor, memory, etc.
• Relatively low availability of systems. System failure leads to a refusal to process calls, which in turn must be correctly processed on the client side. Naturally, clusters are built and load balancers are configured to improve availability, but not always and not everywhere, unfortunately.
• Synchronous calls block resources on the client side and on the server side until the call is completed.

Messaging

Used for asynchronous interaction. Although there were schemes with an attempt to make it synchronous, through blocking waits for responses on the client.

Mainly used where an immediate response is not required:

• A quick response is not required and inter-service interaction can be organized asynchronously.
• Operations are performed for quite a long time and you do not want to block resources.
• For stream processing of large amounts of data (ETL/Data Streaming).

Examples of message brokers:

• ActiveMQ
• Kafka
• RabbitMQ
• NATS
• MQTT

What are the advantages of this approach:

• No locks on the client and server side, low latency.
• Support for guaranteed message delivery with confirmation of receipt.
• In general, the ability to easily scale the performance of message processing by adding new consumers or producers of messages.
• High availability of the system due to queue replication, deployment in a cluster.
• Just like with RPI, it is possible to describe the message format, as well as the interaction contract. Examples:
• AVRO
• JSON Schema
• AsyncAPI – an analogue of OpenAPIv3, but for asynchronous APIs.

What are the disadvantages of this approach:

• Increases the complexity of the system due to the use of an approach with asynchronous interaction. In the case of EDA, it is necessary to think more carefully about how its components will exchange data and change their state.
• It is not always so easy to take and set up a message broker, each with its own problems and features:
• Dependence on the hardware component - falling and failing servers, low network performance, etc.
• Dependence on the software component - a separate headache, these are various bugs and problems in the brokers themselves, client libraries, the lack of necessary functionality (for example, reliable replication between broker nodes).
• Relatively high complexity in setting up and maintaining all system components.

File Transfer

A kind of asynchronous interaction. We put one file in a directory, and another system will pick it up someday and possibly put a file with a response.

Can be used for:

• Exchanging files between old systems, when there is no opportunity to improve and add a new API.
• Transferring large files, when their transfer through other channels can incur large overhead costs. For example, in the case of using message brokers, files are usually uploaded to disk, and then a link to them is transmitted in a message.

Examples:

• MinIO
• Ceph
• HDFS
• GlusterFS
• CIFS/SMB
• NFS

What are the advantages of this approach:

• An easy-to-implement and understand method of exchange. We put a file (with an indication of blocking for the duration of the operation), another system takes it and processes it.
• Relatively good reliability of data storage, if the storage is built taking into account fault tolerance and access rights to it are limited.

What are the disadvantages of this approach:

• Heterogeneity of file formats, since anything can be uploaded. It is necessary to check the integrity and completeness of data by processing.
• It is quite easy to lose data, variations of commands like a rm -rf /.
• Perhaps now with SSD/NVME disks and not a disadvantage at all, but with large volumes of transferred data, disks can become a bottleneck in IOPS.
• Low fault tolerance in most cases. In practice, they are not so often ready to invest human and financial resources in the deployment of distributed file systems, they use conventional network (NFS/SMB) with all their related problems.

Shared Database

Synchronous or asynchronous interaction - it depends on how you look at it. If the operation is DML/DDL, then it is probably synchronous. If in terms of data exchange through tables, then it looks like asynchronous. And some DBMS support subscriptions to DBMS events (PostgreSQL LISTEN)

Can be used for:

• As in the case of file transfer - data exchange between old systems, when there is no way to improve and add a new API.
• Processing events that cannot be added to an existing system (for example, an outdated system being replaced is no longer supported and is not being developed). A separate service monitors changes in tables and performs certain actions, such as sending notifications or updating data in another system.
• Shared data in two or more systems. Yes, this is bad practice and violates isolation, but it is also possible.

What are the advantages of this approach:

• No need to modify existing systems when it is necessary to gain direct access to their operational data.
• No need to modify APIs or complicate infrastructure (file systems, brokers, etc.).

What are the disadvantages of this approach:

• Violation of data isolation, which can lead to problems when changing the data model from one of the systems.
• Complexity of support and debugging, since data can be changed by several systems simultaneously.
• Potential problems with data consistency, especially if changes occur simultaneously in several systems.
• Security issues, all systems have access to data, additional control is needed.

Enterprise Service Bus (ESB)

Enterprise Service Buses (ESB) are worth mentioning separately.

ESB is a middleware that provides centralized and unified event-oriented messaging between different information systems based on the principles of service-oriented architecture.

ESB can often combine several approaches to integration. They are often used to provide centralized management of interaction processes between different systems. That is, in heterogeneous environments (SOA, MSA), where one service is written in C# (SOAP), another in Java (REST) and the third in Node.js (GraphQL). ESB can take on the role of an intermediary and provide a uniform interface for interaction between them, hiding the implementation details of the API of each of the services.

I won’t go into detail about their pros and cons here, it’s better to do that in a separate article. But it’s worth noting that ESB, although it has lost popularity, giving way to integration microservices (maybe they have a correct name, I call them that :), is still used.

ETL

It is impossible not to mention tools for organizing ETL pipelines. But in short, since the topic is quite extensive.

Probably the most famous and popular of them (at least for me :) is Apache Camel. Quite a powerful tool that allows you to create routes for processing and transforming data between different sources and supports all the main EIP.

The second of them, we can say Apache Camel with GUI – Apache NiFi. It allows you to create graphical pipelines for data processing and has many options for monitoring and managing these pipelines.

Here you can also add:

• Apache Flink – a distributed system for processing streaming data with high performance.
• Apache Airflow – a platform for managing workflows that allows you to automate data processing tasks.

Observability of systems

I would like to highlight the problems of observability of integration flows separately. No matter what approaches and solutions they are built with, ensuring monitoring, logging and tracing is one of the mandatory requirements.

The main problem with debugging and supporting integration flows is the need to understand the ongoing processes and monitor the data transferred between systems. That is, it is necessary not only to see logs with some messages, but also to understand the end-to-end process of transferring a data packet from one system to another along the entire chain of participants in the integration exchange.

It is also important to be able to assess the consistency and consistency of data in different systems, but this is solved not by monitoring, but by specialized tools for data analysis.

In conclusion, I will be Captain Obvious. There is one point that is worth mentioning - the observer should not be observed, no matter how strange it may sound. This means that you do not need to monitor the same system that needs to be monitored. If we’ve down such a system and lose its monitoring.

The list is far from complete, but probably the most common and well-known tools are:

• Мониторинг + логирование
  • PLG Stack Prometheus, Grafana, Loki
  • ELK Stack Elasticsearch, Logstash, Kibana.
  • VictoriaMetrics, VictoriaLogs.
• Jaeger, Zipkin – distributed tracing system.
• OpenTelemetry – a certain standard, an attempt to unite various tools for ensuring observability of systems into a single ecosystem.

Conclusion

Briefly this time. There is no universal approach or tool, everything depends on the project, requirements, deadlines and restrictions of the Customer. The choice of tools should take into account all the input data of your project, and not someone’s abstract recommendations :)