Skip to content

Routing messages between stateful servers

When clients hold long-lived connections, a message for one user has to reach the one server holding that user's socket. The sender's server finds it through a registry, a pub/sub backplane or a hash of the ID.

On this page

What it is

A stateless web server can answer any request, so a load balancer may send each one anywhere. A server that holds a WebSocket or other persistent connection is different: the connection lives in one process, and a message meant for that client has to be handed to that process. The ASP.NET Core SignalR documentation describes the result in a server farm: each server knows its own connections and not the others', so a broadcast from one server reaches its own clients and no one else's.

Routing between stateful servers is the mechanism that bridges the gap. When a message for Bob arrives at the server holding Alice's socket, that server needs to answer one question, which server holds Bob, and then needs a way to hand the message over. Three designs answer the question differently.

A connection registry. When a client connects, its server records user -> server (and often the connection ID) in a shared store. A sender looks the user up and forwards the message to that server over an internal channel. The managed version is what Amazon API Gateway offers for WebSocket APIs: the gateway holds the sockets, and your backend sends to a client with a POST to @connections/{connection_id}, so the design still needs somewhere to look up which connection ID belongs to Bob. A GoneException comes back when the client has already disconnected, which is the registry's main weakness in one error: the entry can outlive the connection.

A pub/sub backplane. Each server subscribes to channels for the users or rooms it serves; a sender publishes and lets the broker deliver to whichever servers subscribed. SignalR's Redis backplane works this way: connection information is passed to the backplane, a server sends to it, and the message reaches clients through their own servers. The Redis documentation states that Pub/Sub delivery is at-most-once: a message is sent once if at all, and one the subscriber misses, through an error or a network disconnect, is lost. Since Redis 7.0, sharded Pub/Sub assigns shard channels to cluster slots the same way keys are assigned, so the backplane itself can be spread across cluster nodes.

Deterministic placement. Hash the conversation ID to a server with consistent hashing, so any server can compute the owner without a lookup. Slack's Channel Servers are built this way: each is stateful and in memory, each is mapped to a subset of channel IDs by consistent hashing, and the Gateway Servers that hold users' WebSockets subscribe to the Channel Servers that own the user's channels. Slack reports about 16 million channels served per host at peak and a replacement Channel Server ready in under 20 seconds, during which users of the affected channels see higher delivery latency.

When to use it

Use one of these designs as soon as persistent connections are spread across more than one process and a message can originate somewhere other than the recipient's server: chat, presence, collaborative editing, live dashboards, notifications pushed to an open tab. SignalR's documentation lists a single server in a single process as the case where none of this is needed, which is the line to draw in the interview.

Pick a pub/sub backplane when most traffic is to rooms rather than to individuals and you want the fewest moving parts: servers subscribe to the rooms their clients are in, and a message to a room reaches just the servers that subscribed. Pick a registry plus direct forwarding when most messages are one-to-one and you want delivery you can confirm and retry per recipient, or when a managed gateway already holds the sockets, as with API Gateway's connection IDs. Pick consistent hashing when a conversation needs one owner anyway, for ordering or for in-memory history, which is what Slack's Channel Servers hold.

Whichever you pick, plan for one user with several devices: the registry maps a user to a set of connections, or each device subscribes separately, and a message fans out to all of them. Give registry entries a time to live refreshed by heartbeats, so a server that dies without cleaning up stops attracting traffic after a bounded delay rather than indefinitely.

When not to

One process. With a single server there is nothing to route between; both SignalR and Socket.IO treat the multi-node machinery as something you add when you scale out. Don't add a backplane to a design that fits on one box, and say so.

Request/response APIs. If clients poll or make ordinary HTTP calls, the servers can stay stateless, state lives in a database, and a plain load balancer does the routing. The cost of persistent connections is real: SignalR's documentation notes that they stay open while the client is idle and consume memory per connection, and that a backplane-based app has to scale out by number of clients even when few messages are sent. Compare the options in WebSockets, server-sent events and polling before committing to sockets.

A managed connection service fits. Azure SignalR Service and API Gateway WebSocket APIs hold the sockets for you, so your servers keep a small, constant number of connections and the routing problem moves to the provider. SignalR's documentation recommends its service for apps hosted on Azure and names the exception: on-premises apps with low-latency or high-throughput needs and significant latency to an Azure data center, where the Redis backplane is the better fit.

Small fan-out that a broadcast absorbs. If you run a handful of servers and message volume is modest, publishing each message on one shared channel and letting each server drop what isn't for it is simpler than a registry and good enough. The registry earns its place when server count times message rate makes that waste visible.

Compared

ApproachHow the sender finds the recipient's serverWhat grows with scaleThe weaker option is right when

Pub/sub backplane

It doesn't: it publishes to a channel and subscribed servers receive

Subscriptions per server; at-most-once loss on a dropped subscriber

Room traffic dominates, a lost message is tolerable or recovered from the store, and you want the least code

Connection registry + direct forward

Looks up user -> server in a shared store, then calls that server

Registry writes on each connect and disconnect; stale entries after crashes

Traffic is mostly one-to-one and you need per-recipient confirmation and retries

Consistent hashing of the conversation ID

Computes the owner from the ID; no lookup

Rebalancing when the ring changes; a hot conversation lands on one host

A conversation needs a single owner for ordering or in-memory history

Managed gateway (API Gateway, Azure SignalR Service)

Stores connection IDs and posts to the provider's callback endpoint

Whatever the provider charges and the latency of the round trip to it

You are on that cloud and latency to it is acceptable

Sticky sessions alone

It can't: affinity keeps one client on one server and says nothing about others

Connections per server still grow, but no cross-server path is added, which is why it looks complete

A single server, or when all you need is keeping one client's handshake and polling on one node

In a picture

connectbob = B, TTLBob's clientServer BConnectionregistry

Step 1 of 4

Bob connects and is registered

Bob's WebSocket lands on server B. B records that Bob is on B in a shared registry, with a time to live it refreshes while the socket is open.

How to say it in an interview

The common mistake

Quiz

A chat server crashes without cleaning up. Registry entries for its 20,000 users still point at it. What bounds how long senders keep forwarding to the dead server?
You route through Redis Pub/Sub. Server B's subscriber connection drops for two seconds while a message for one of its users is published. What happens to that message?
Why did Slack place channels on Channel Servers by consistent hashing rather than looking each one up in a registry?
Which deployment needs none of these routing mechanisms?

Questions that use this

Hear it in

Related

Sources

  1. Microsoft Learn: ASP.NET Core SignalR production hosting and scaling (ASP.NET Core 10.0)
  2. Redis documentation: Redis Pub/sub
  3. Slack Engineering: Real-time Messaging
  4. Amazon API Gateway Developer Guide: Use @connections commands in your backend service
  5. Socket.IO v4 documentation: Using multiple nodes
Sign in to report a mistake