tekBlueprint

A summary article on choosing a load balancing approach depending on the task. I decided to collect the main algorithms and approaches. I think some of the information overlaps with what’s described in the book “Designing Data-Intensive Applications” (the one with the boar). Although I might be confused, as I read it a long time ago.

The algorithms listed below can be applied not only to routing between servers but also to internal application logic. For example, in task schedulers and data processing pipelines.

Introduction

When developing modern services/microservices, it is often necessary to scale available resources as the load increases. However, it is impossible to scale vertically indefinitely, so a horizontal scaling approach is used. This necessitates the correct selection of a load balancing method. Key system characteristics, such as latency, reliability, and state consistency, directly depend on the efficiency and correctness of the balancing algorithm.

It would seem that the task is simple – accept an incoming request and route it to one of the available servers in the pool. However, reality is much more complex. Not all servers are created equal: they may have different computing power, varying amounts of free resources (CPU, RAM, I/O), and different current workloads (number of active connections, request processing time). Requests are also not the same: some require a millisecond response (API calls), others require a persistent connection (WebSockets, gRPC streams, database connections), and still others require maintaining user session state or managing caches. An incorrect algorithm choice can lead to:

• Overloading servers while others remain idle.
• Increased latency when requests reach servers whose processing queues are already overloaded.
• Loss of previous state when rebalancing cached data or sessions may send requests to the “wrong” server, requiring reauthentication or a cache reset.
• Violation of fault tolerance if traffic continues to flow to a downed server until a health check detects the problem.

Load balancers can be placed at different layers of the OSI model:

• L4 (Transport Layer):
  • Operate at the TCP/UDP layer.
  • Only see IP addresses and ports.
  • Do not analyze request contents (HTTP headers, cookies).
  • Faster, but dumber.
  • Examples: Linux IPVS, HAProxy in TCP mode, AWS NLB, DSR.
• L7 (Application Layer):
  • Operate at the application protocol layer (HTTP, HTTPS, gRPC, etc.).
  • Understand request contents (URL, headers, request body), allowing decisions to be made based on request data (e.g., cookie-based sticky sessions, URL-based routing).
  • Examples: HAProxy in HTTP mode, Nginx, Envoy, Yandex Application Load Balancer, AWS ALB/ELB.

Examples of deployments for L7 load balancers:

• Sidecar Proxy:
  • A lightweight proxy deployed next to each application in a container (Kubernetes) or microservice.
  • Load balancing occurs locally for each service, often with additional observability and traffic management capabilities.
  • Examples: Can be built with a service mesh (e.g., Istio, Linkerd).
• Client-Side Load Balancer:
  • A client application or library.
  • The application automatically selects a server from a list received from the Discovery Service or from a predefined list of servers.
  • Load balancing algorithms are built into the application or library.
  • Examples: Ribbon in Spring Cloud, gRPC Client Load Balancing.

Choosing a load balancing algorithm is always a tradeoff between simplicity, speed, resource efficiency, adaptability to current load, and the need to preserve state. This article will examine the logic behind these choices, explaining not only how the algorithms work, but also why and when to use them to ensure the system remains fast, reliable, and predictable. We’ll cover basic approaches, adaptive strategies, and, most importantly, methods for solving the problem of maintaining state in distributed systems.

Basic algorithms

These algorithms are suitable for simple scenarios where the servers in the pool are homogeneous (same capacity, same configuration) and requests are relatively short and have similar load patterns. Their main advantages are ease of implementation and predictability of distribution (except for random selection).

Round Robin (RR)

The load balancer maintains an internal counter or pointer to the next server in the pool. Each new request is sequentially routed to the next server in the list, after which the pointer advances to the next. If the list of servers is [S1, S2, S3], the order is: S1 -> S2 -> S3 -> S1 -> S2 -> S3 ….

Weighted Round Robin (WRR)

Each server is assigned a weight proportional to its capacity or desired traffic share. The load balancer “virtually” duplicates the server in the list proportional to its weight. For example:

• Server S1 (weight=2), S2 (weight=1), S3 (weight=1).
• Virtual list: [S1, S1, S2, S3].
• Request order: S1 -> S1 -> S2 -> S3 -> S1 -> S1 -> S2 -> S3 ...
• As a result, S1 will receive 50% of the traffic (2 of 4 “virtual” slots), S2 and S3 – 25% each.

Ideal when servers in a pool have different performance capabilities (e.g. different number of CPUs/cores, different disk types) or when you want to gradually introduce new, less powerful servers into operation with a smaller percentage of traffic.

Random

The load balancer randomly selects one server from the available pool for each new request. Unlike Round Robin, where the order is strictly defined, Random uses a random number generator to select the target server. For example:

1. For each incoming request, a random number is generated.
2. This number is used to select one server from the pool (e.g., random_index = random(0, N-1), where N is the number of servers).
3. The request is routed to the selected server.

Adaptive algorithms that take current load into account

When requests vary greatly in processing time (for example, a mix of simple GET and complex POST requests to a DBMS) or servers are heterogeneous, simple RR/WRR algorithms become ineffective. Adaptive algorithms help solve such problems by monitoring the current state of servers.

Least Connections (LC)

The load balancer continuously monitors the number of active (open, processing) connections on each server. A new request is routed to the server with the minimum number of active connections. For example:

• Server S1 (connections=0), S2 (connections=1), S3 (connections=5).
• Request order: S1 -> S1 -> S2 ...
• As a result, S1 will receive 2 connections, S2 will receive 1 connection, and S3 will be left with no new connections.
• S1 (connections=2), S2 (connections=2), S3 (connections=5)

This algorithm is well suited for long-lived connections. For long-lived connections (WebSockets, gRPC streams, database connections, FTP), the number of connections is a much more meaningful load metric than simply the number of requests (which can be fixed for such connections). LC effectively distributes the “weight” of such long connections.

Weighted Least Connections (WLC)

Combines the idea of ​​LC and WRR. Routes a new request to the server that has the fewest active connections relative to the server’s weight. The weight is usually proportional to the server’s capacity (e.g., number of CPU cores, available memory). For example:

• S1: weight=2, connections=10 -> priority=5.0
• S2: weight=1, connections=8 -> priority=8.0
• S3: weight=1, connections=6 -> priority=6.0
• The next request will go to S1 (5.0 < 6.0 < 8.0), despite the high number of connections, since its relatively high capacity allows it to handle them.

Server priority calculation formula.

Next, the server with the lowest priority is selected.

Least Response Time (LRT)

The load balancer measures the average response time (Round Trip Time, RTT) or the most recent response time from each server and routes the new request to the server with the shortest measured response time. For example:

• S1: rtt=10ms
• S2: rtt=20ms
• S3: rtt=30ms
• The next request will go to S1, since the last request completed in 10 milliseconds.

Resource-based

More advanced adaptive algorithms can perform more complex calculations than simply counting connections or measuring RTT. To obtain an accurate picture of the actual load, system resource usage is monitored on each server: CPU, memory, disk I/O, and network. The idea is to route requests not just to less loaded servers, but to servers with sufficient free resources to effectively process a new request. The load balancer periodically receives metrics from monitoring agents (or built-in modules) on each server. Based on these metrics, a “load index” or “resource priority” is calculated. A new request is routed to the server with the lowest value of this index.

For example, consider the following distribution.

The next request will go to server S1, as it has the lowest priority.

Example of the priority calculation formula.

Where w1, w2, w3, w4 are weights reflecting the importance of each resource for your specific server. For example, for a CPU-intensive application, the CPU weight will be higher than for one that uses disk I/O intensively.

Use examples:

• When a pool contains servers with different configurations (different numbers of CPUs, different types of SSD/HDD drives), this balancing type can more accurately distribute the load based on actual capabilities.
• The pool simultaneously runs both CPU-intensive tasks (video processing, calculations) and I/O-intensive tasks (file handling, databases). The algorithm can attempt to balance these types of workloads.

Stateful load balancing

Basic and adaptive algorithms are ideal for stateless services, where each request is independent and can be processed by any server. However, for stateful applications (caches, user sessions, queues, databases), it is critical that related requests (e.g., from the same user or for the same cache key) reach the same server where the state is stored. Hashing-based algorithms can be used here.

IP Hash

This is one of the simplest and most common algorithms for ensuring “sticky sessions.” Instead of distributing requests evenly or based on current load, the load balancer calculates a hash of the client’s IP address and routes all requests with the same hash to the same server. For example:

• Server pool: [S1, S2, S3]
• Client IP: 192.168.1.100
• hash("192.168.1.100") % 3 = 1 → request is routed to S2
• All subsequent requests from 192.168.1.100 will also be routed to S2

Info

Problems with session loss when changing the number of servers can be solved by allocating external storage for storing user session data outside the service (Redis, Memcached, Hazelcast, etc.).

Consistent Hashing

If IP Hash balancing is used and there is no external session storage, the following situation may occur. When adding a new server to the pool or removing an existing one, the active session for the user will be lost. Why? Because the hash function (e.g., hash(key) % N, where N is the number of servers) will produce completely different results for the same key when N changes. For example:

• hash("User 1") % 3 = 0 (server S1)
• Add S4. Now hash("User 1") % 4 = 1 (server S2). The entire cache for User 1 is lost.
• Consistent hashing solves this problem by minimizing the number of keys that “move” when the topology changes (adding/removing servers).

Simplified concept of operation (details and research can be found separately):

1. Ring – an abstract ring with values from 0 to 2^32-1 (or 2^64). This is the hash space.
2. Virtual Nodes – each physical server in the pool is represented not as a single point on the ring, but as a set (e.g., 100 or 200) of virtual nodes. The hash value for the server’s name (or its IP:port) is calculated, and this value is “inserted” into the ring. The virtual nodes of this server are then distributed around this hash on the ring (not adjacent). This ensures a more even distribution. A physical server can always be identified by a virtual node.
3. The hash of a key (e.g., a session ID, a cache key) is calculated and also “inserted” into the ring. The key belongs to the server whose closest virtual node is located clockwise.
4. When a server is added, hashes are calculated for it and its virtual nodes. Keys that previously belonged to servers whose virtual nodes are located between the new point and its nearest clockwise “neighbor” are redistributed to the new server. All other keys remain in their original locations.
5. When a server is deleted, all virtual nodes of the server are deleted. Keys that pointed to it are redistributed to the server whose nearest virtual node is now located counterclockwise (i.e., the next server in the ring after the deleted virtual nodes).

Resilient Hashing

The term “robust hashing” is not always strictly defined and is often used synonymously with Consistent Hashing in the context of its ability to resiliently tolerate topology changes. However, it can also refer to specific improvements or alternative algorithms, such as:

• Rendezvous Hashing (Highest Random Weight - HRW) – an algorithm that does not require a ring. A “weight” is calculated for each key and server list pair (key, server). The key is always routed to the server with the highest weight. When a server is added or removed, only keys that previously belonged to the removed server (or now belong to the new one) are redistributed. It does not require virtual nodes.
• Maglev Hashing – an algorithm that strives for uniform distribution and a predictable route (unlike Consistent Hashing, where the route can change when a server is added). It uses the idea of ​​an ideal hash table to construct the lookup table.

In Envoy/Nginx proxies, “Resilient Hashing” often refers to Consistent Hashing with carefully tuned parameters (number of virtual nodes) to achieve an optimal balance between uniformity and resilience to changes.

The main value of these algorithms for stateful systems is their resilience to failures and topology changes. As discussed in Consistent Hashing, if one out of 100 servers goes down, traffic is redistributed only for keys that were “bound” to that server (1% for 100 virtual nodes per server).

Balancing using ML models

Balancing using machine learning models is a separate area of ​​focus. This enables a transition from static and adaptive algorithms to proactive and predictive approaches. Instead of reacting to current load, ML models can predict system behavior and optimize traffic distribution in advance.

Key concepts of ML-balancing:

1. Load Forecasting. ML models analyze historical data on traffic, seasonality, user behavior, and external factors (holidays, marketing campaigns) to predict future load. This forecast allows for resource provisioning or load balancing before peak periods.
   • Features:
     • Timestamps (hour of day, day of week, month).
     • Historical metrics (RPS, CPU, Memory, Network I/O).
     • External events (holidays, releases, marketing activities).
     • Geographic distribution of users.
     • Behavioral patterns (request frequency, operation types).
   • Forecasting models:
     • ARIMA/SARIMA – traditional time series for seasonal patterns.
     • Prophet – Facebook library for forecasting with seasonality and holidays.
     • LSTM/GRU – neural networks for complex nonlinear dependencies.
     • Gradient Boosting (XGBoost, LightGBM) – for combining multiple features.
2. Server Clustering. Instead of using all servers as a single pool, ML can automatically group servers based on similar characteristics.
   • Clustering Algorithms:
     • k-means – grouping by current resource load.
     • DBSCAN – detecting anomalies and natural groups.
     • Hierarchical Clustering – building a hierarchy of servers based on performance.
   • For example, servers can be grouped into clusters:
     • “High-CPU, Low-I/O” for computationally intensive tasks.
     • “Low-CPU, High-I/O” for disk I/O-intensive tasks.
     • “Balanced” for mixed workloads.
3. Routing Optimization. ML models can find optimal routes for requests based on a variety of factors:
   • Multi-armed Bandit (MAB) algorithms – balance exploration (sending to new servers) and exploitation (sending to trusted servers).
   • Reinforcement Learning (RL) – the agent is trained to make routing decisions to maximize reward (minimize latency, maximize throughput).
   • Neural Networks – predict the optimal server based on hundreds of features.

Machine learning (ML) balancing is not a replacement for traditional algorithms. It is most effective when combined with classical approaches, where ML handles complex predictive tasks while traditional algorithms provide basic fault tolerance and simplicity.

Comparison table

Practical tips

Do you have three identical web servers serving static files and simple stateless APIs? Round Robin is your best friend. Simplicity, predictability, and no monitoring overhead. Don’t implement Envoy or Consistent Hashing “just in case.”

Using Weighted Round Robin? You have server S1 with a weight of 10 (powerful) and S2 with a weight of 1 (test). If S1 suddenly starts processing very long requests (for example, due to a database failure), WRR will continue sending 90% of the traffic to it while S2 is almost idle. Combine WRR with adaptive algorithms (LC, WLC) or actively monitor latency and implement Circuit Breaker.

Health checks are essential for ensuring resilience:

• Regardless of the chosen balancing algorithm, it must be resilient to server failures. No matter how smart the algorithm is, it sends traffic to dead servers.
• The load balancer periodically “pings” servers (HTTP GET to /health, TCP connect) at a specified interval. If a server doesn’t respond or returns an error, it is removed from the pool.
• The load balancer monitors user request responses. Too many errors (5xx, 4xx) or a high response time from a specific server? Remove it. This requires careful threshold adjustment.

Ensure server observability:

• Metrics must be collected:
• Number of requests per second. Is traffic evenly distributed? Are there any “hot” servers?
• Latency (P50, P90, P99). Is one server slowing everyone down? Does an adaptive algorithm (LC/WLC/LRT) help? * Number of active connections. For LC/WLC, these are key metrics. Does the connection distribution meet expectations?
• Error rate (4xx, 5xx). Helps identify issues with a specific server.
• Health Check. How many servers in the pool are healthy?
• What tools can be used? Prometheus + Grafana, Datadog, New Relic, built-in statistics in Nginx/HAProxy/Envoy.

Don’t rely on theoretical calculations. Use tools like vegeta, wrk, locust, jmeter, and k6 to simulate real load and test distribution, latency, and system behavior during failures (server kill -9, network throttling).

In architecture documents (ADRs) or in configuration comments, always indicate the rationale for choosing a specific balancing algorithm. This will help future users and your colleagues understand the context and tradeoffs.

Conclusion

There is no universally “best” algorithm. RR is simple but not adaptive. LC/WLC are adaptive but do not account for power differences. Consistent Hashing is ideal for caches and sessions, but is complex to implement and configure. Your choice will always be a compromise. It is important to understand the nature of your requests (short/long, stateful/stateless) and servers (homogeneous/heterogeneous). This is the main driver of your choice.

It is important to remember that system stability is ensured not only by the balancing algorithm, but also by mandatory Active/Passive health checks, Circuit Breaker for quick response to slow servers, and observability for real-time health monitoring.

It is easiest to start with the simplest approach (RR), which solves the current problem. You can then increase complexity (WRR, LC, Consistent Hashing) when the current approach becomes a bottleneck.