Spreading the Load: How Backends Decide Which Server Gets Your Request

When one server is no longer enough, you add more and put a load balancer in front. That sounds simple until you realize the balancer is now making a routing decision on every request, and a naive choice can leave one server melting while another sits idle. Coming from a single Node process where every request hits the same memory, the shift is mental: state can no longer live in one process, and any two requests from the same user might land on different machines. This article covers the major balancing algorithms and when each fits, the difference between layer 4 and layer 7 balancing, why health checks matter more than the algorithm, how sticky sessions both help and hurt, and the failure modes that turn a load balancer from a scaling tool into a single point of failure or a source of cascading outages.

Round Robin And Its Limits

Round robin is the default everyone starts with: cycle through the server list, sending each new request to the next one in line. It is simple, predictable, and fine when every request costs roughly the same and every server is identical. The trouble appears when those assumptions break. If one request triggers a heavy report while the next is a trivial lookup, round robin happily sends the next heavy request to a server already drowning, because it counts requests, not work. Servers of different sizes suffer the same way since each gets an equal share regardless of capacity. Weighted round robin patches the capacity problem by giving bigger servers a larger slice, but it still ignores what each server is actually doing right now. Treat round robin as a reasonable baseline, not a finished answer, and reach for load-aware algorithms once your request costs vary widely.

Least Connections And Load Aware

Least connections routes each request to whichever server currently has the fewest active connections, which adapts to reality in a way round robin cannot. A server stuck on slow requests accumulates open connections and naturally stops receiving new ones until it catches up, so load self-corrects without you tuning weights. This shines when request durations vary a lot, which describes most real applications. A refinement, least response time, factors in how quickly each server has been answering, steering traffic toward the genuinely faster ones. The cost is that the balancer must track per-server state, which is more work than a simple counter, and in a distributed balancer that state can be imperfect. Still, for workloads with mixed request costs, least connections usually beats round robin handily, and it is the sensible default when you cannot assume every request and every server are interchangeable.

Layer 4 Versus Layer 7

Load balancers operate at one of two layers, and the difference shapes what they can do. A layer 4 balancer works at the transport level, routing by IP and port without looking inside the traffic, which makes it extremely fast and protocol-agnostic but blind to anything application-specific. A layer 7 balancer reads the actual HTTP request, so it can route by path, host header, or cookie, terminate TLS, rewrite headers, and make smart content-based decisions. That intelligence costs CPU because it decrypts and parses every request, but it unlocks routing like sending API traffic to one pool and static assets to another. Most modern setups use layer 7 at the edge because the flexibility is worth it, reserving layer 4 for raw throughput needs like database or non-HTTP traffic. Knowing which layer you are at tells you immediately what routing decisions are even possible.

Health Checks Decide Everything

The balancing algorithm only matters if every server in the pool actually works, and that is what health checks guarantee. The balancer periodically probes each backend, and a server that fails the check is pulled from rotation so no user traffic reaches it until it recovers. The depth of the check matters enormously. A shallow check that only confirms the port is open will keep sending traffic to a process that accepts connections but cannot reach its database, turning every routed request into an error. A deep check hits a dedicated health endpoint that verifies real dependencies, catching the broken instance before users do. Tune the thresholds carefully: too aggressive and a brief hiccup ejects a healthy server, too lax and a dead one keeps taking traffic. Good health checks turn a pool of servers into a system that quietly routes around its own failures.

Sticky Sessions And State

Sticky sessions pin a given client to the same backend for the duration of a session, usually by setting a cookie or hashing the client address. They exist because some applications keep per-user state in process memory, so bouncing a user between servers would lose their session or break an in-progress connection like a WebSocket upgrade. Stickiness solves that, but it quietly undermines balancing: traffic clumps onto whichever servers caught the heavy users, and when a sticky server dies, all its pinned clients lose state at once. The better long-term fix is to make servers stateless by pushing session data into a shared store like Redis, so any server can handle any request and you can scale and deploy freely. Use stickiness as a pragmatic bridge when you genuinely cannot externalize state, but treat statelessness as the architecture you are working toward, not a luxury.

Hashing For Cache Affinity

Sometimes you want the same key to always reach the same server, not for sessions but for efficiency, and that is where hash-based routing helps. By hashing a request attribute like a user id or a cache key and mapping it to a server, you ensure requests for the same data land on the server most likely to have it warm in its local cache, dramatically improving hit rates. The naive version breaks badly when you add or remove a server, because a plain modulo remaps almost every key at once and stampedes your backends with cache misses. Consistent hashing fixes this by arranging servers on a ring so that adding or removing one only reshuffles a small fraction of keys. This technique underpins distributed caches and sharded systems, and it explains why some scaling events cause a cache-miss storm while others pass unnoticed.

The Balancer As Bottleneck

A load balancer concentrates all your traffic through one point, which is convenient and also dangerous, because if it fails everything behind it becomes unreachable no matter how healthy your servers are. The standard answer is to run the balancer itself redundantly, with a second instance ready to take over via a floating address or DNS, so a single failure does not take down the whole tier. Capacity matters too: the balancer must handle the aggregate connection count and, at layer 7 with TLS termination, the CPU cost of decrypting every request, which can quietly become your real ceiling. Watch its own metrics, not just the backends, because a saturated balancer adds latency to every request uniformly and is easy to overlook when you are staring at application dashboards. Design the entry point with the same redundancy you demand of the services it protects.

Balancing In Containers

Once you move to containers and orchestration, load balancing splits into layers that are easy to conflate. Inside a cluster, a service abstraction balances traffic across the current set of healthy container instances, and because containers come and go constantly, this balancing is tied to service discovery rather than a static server list. An external balancer or ingress sits at the edge handling traffic from the public internet and routing it into the cluster, often doing layer 7 work like TLS and path routing. The instances behind any service are ephemeral, so health checks and discovery do the heavy lifting of keeping the pool correct as deployments roll. The mental upgrade from a fixed pool of virtual machines is that membership is never stable; the balancer is continuously fed an updated list, and your job is to make draining and startup graceful so rolling deploys do not drop requests.