Routing messages between stateful servers
When clients hold long-lived connections, a message for one user has to reach the one server holding that user's socket. The sender's server finds it through a registry, a pub/sub backplane or a hash of the ID.
What it is
A stateless web server can answer any request, so a load balancer may send each one anywhere. A server that holds a WebSocket or other persistent connection is different: the connection lives in one process, and a message meant for that client has to be handed to that process. The ASP.NET Core SignalR documentation describes the result in a server farm: each server knows its own connections and not the others', so a broadcast from one server reaches its own clients and no one else's.
Routing between stateful servers is the mechanism that bridges the gap. When a message for Bob arrives at the server holding Alice's socket, that server needs to answer one question, which server holds Bob, and then needs a way to hand the message over. Three designs answer the question differently.
A connection registry. When a client connects, its server records user -> server (and often the connection ID) in a shared store. A sender looks the user up and forwards the message to that server over an internal channel. The managed version is what Amazon API Gateway offers for WebSocket APIs: the gateway holds the sockets, and your backend sends to a client with a POST to @connections/{connection_id}, so the design still needs somewhere to look up which connection ID belongs to Bob. A GoneException comes back when the client has already disconnected, which is the registry's main weakness in one error: the entry can outlive the connection.
A pub/sub backplane. Each server subscribes to channels for the users or rooms it serves; a sender publishes and lets the broker deliver to whichever servers subscribed. SignalR's Redis backplane works this way: connection information is passed to the backplane, a server sends to it, and the message reaches clients through their own servers. The Redis documentation states that Pub/Sub delivery is at-most-once: a message is sent once if at all, and one the subscriber misses, through an error or a network disconnect, is lost. Since Redis 7.0, sharded Pub/Sub assigns shard channels to cluster slots the same way keys are assigned, so the backplane itself can be spread across cluster nodes.
Deterministic placement. Hash the conversation ID to a server with consistent hashing, so any server can compute the owner without a lookup. Slack's Channel Servers are built this way: each is stateful and in memory, each is mapped to a subset of channel IDs by consistent hashing, and the Gateway Servers that hold users' WebSockets subscribe to the Channel Servers that own the user's channels. Slack reports about 16 million channels served per host at peak and a replacement Channel Server ready in under 20 seconds, during which users of the affected channels see higher delivery latency.
When to use it
Use one of these designs as soon as persistent connections are spread across more than one process and a message can originate somewhere other than the recipient's server: chat, presence, collaborative editing, live dashboards, notifications pushed to an open tab. SignalR's documentation lists a single server in a single process as the case where none of this is needed, which is the line to draw in the interview.
Pick a pub/sub backplane when most traffic is to rooms rather than to individuals and you want the fewest moving parts: servers subscribe to the rooms their clients are in, and a message to a room reaches just the servers that subscribed. Pick a registry plus direct forwarding when most messages are one-to-one and you want delivery you can confirm and retry per recipient, or when a managed gateway already holds the sockets, as with API Gateway's connection IDs. Pick consistent hashing when a conversation needs one owner anyway, for ordering or for in-memory history, which is what Slack's Channel Servers hold.
Whichever you pick, plan for one user with several devices: the registry maps a user to a set of connections, or each device subscribes separately, and a message fans out to all of them. Give registry entries a time to live refreshed by heartbeats, so a server that dies without cleaning up stops attracting traffic after a bounded delay rather than indefinitely.
When not to
One process. With a single server there is nothing to route between; both SignalR and Socket.IO treat the multi-node machinery as something you add when you scale out. Don't add a backplane to a design that fits on one box, and say so.
Request/response APIs. If clients poll or make ordinary HTTP calls, the servers can stay stateless, state lives in a database, and a plain load balancer does the routing. The cost of persistent connections is real: SignalR's documentation notes that they stay open while the client is idle and consume memory per connection, and that a backplane-based app has to scale out by number of clients even when few messages are sent. Compare the options in WebSockets, server-sent events and polling before committing to sockets.
A managed connection service fits. Azure SignalR Service and API Gateway WebSocket APIs hold the sockets for you, so your servers keep a small, constant number of connections and the routing problem moves to the provider. SignalR's documentation recommends its service for apps hosted on Azure and names the exception: on-premises apps with low-latency or high-throughput needs and significant latency to an Azure data center, where the Redis backplane is the better fit.
Small fan-out that a broadcast absorbs. If you run a handful of servers and message volume is modest, publishing each message on one shared channel and letting each server drop what isn't for it is simpler than a registry and good enough. The registry earns its place when server count times message rate makes that waste visible.
Compared
| Approach | How the sender finds the recipient's server | What grows with scale | The weaker option is right when |
|---|---|---|---|
Pub/sub backplane | It doesn't: it publishes to a channel and subscribed servers receive | Subscriptions per server; at-most-once loss on a dropped subscriber | Room traffic dominates, a lost message is tolerable or recovered from the store, and you want the least code |
Connection registry + direct forward | Looks up | Registry writes on each connect and disconnect; stale entries after crashes | Traffic is mostly one-to-one and you need per-recipient confirmation and retries |
Consistent hashing of the conversation ID | Computes the owner from the ID; no lookup | Rebalancing when the ring changes; a hot conversation lands on one host | A conversation needs a single owner for ordering or in-memory history |
Managed gateway (API Gateway, Azure SignalR Service) | Stores connection IDs and posts to the provider's callback endpoint | Whatever the provider charges and the latency of the round trip to it | You are on that cloud and latency to it is acceptable |
Sticky sessions alone | It can't: affinity keeps one client on one server and says nothing about others | Connections per server still grow, but no cross-server path is added, which is why it looks complete | A single server, or when all you need is keeping one client's handshake and polling on one node |
In a picture
Step 1 of 4
Bob connects and is registered
Bob's WebSocket lands on server B. B records that Bob is on B in a shared registry, with a time to live it refreshes while the socket is open.