Real-Time State Sync Over WebSockets

By Robin Dowling · 2 months ago

Streaming simulation state to browser clients at 10 fps means every part of the pipeline has to handle the reality that clients fall behind, messages get dropped, and the world keeps moving regardless. Here's how the state sync works in Wilden, from the server's outbound queue to the client's state buffer.

Bounded drop-oldest outbound queue

Each connected client has a bounded outbound channel on the server, with a capacity of 8 messages. When the broadcast loop produces a new state message and the client's queue is full, the oldest undelivered message is evicted to make room for the newest one.

This means a slow client always converges toward the most current world state rather than falling further behind. The server never accumulates a growing backlog of stale state for a client that can't keep up, and a stalled client can never grow server memory.

The alternative — an unbounded queue or drop-newest — would either let a lagging client consume unbounded memory on the server, or force the client to process a long queue of outdated state before catching up to the present. Drop-oldest avoids both.

Delta state with forced full recovery

Most broadcasts are deltas: only the entity properties that changed since the last broadcast are sent, tracked per session. This reduces typical payload sizes by 70-90% compared to sending the full entity set every time.

But deltas are fragile. If a client misses a delta (because the outbound queue evicted it), the client's view drifts from reality. Two independent mechanisms recover from this:

Forced full state. Every 200 broadcasts, the server sends the complete entity set regardless of what the client already knows. Any accumulated delta drift is wiped clean. This is a timer-based safety net that requires no client cooperation.

Client feedback. When the client detects missing entities after applying a delta, it sends a feedback message to the server. The server clears its record of what that client knows and sends a full state on the next broadcast. This is a demand-driven recovery that kicks in immediately rather than waiting for the forced-full cadence.

Either mechanism alone would be sufficient. Having both means delta gaps are a transient visual glitch lasting at most a few hundred milliseconds, not a corruption bug that compounds over time.

Client-side state buffering

The client maintains a buffer of up to 5 pending state updates, keyed by iteration number. When a new state arrives, the buffer tries to find the next sequential iteration. If it's there, it's applied immediately. If it's not, the buffer waits — as long as it hasn't filled up.

When the buffer reaches capacity without the expected next iteration arriving, it falls back to the oldest available state. This handles the gaps created by the server's drop-oldest queue: the client sees iteration 5, then 7, with 6 never arriving. Rather than stalling, it applies what it has.

A world transition (server restart, new world) flushes all buffered states from the previous world. The first message from the new world always carries a full entity set, so the client rebuilds cleanly without waiting for the buffer to fill.

This buffering creates smoother visual transitions than immediately applying every state as it arrives. States that arrive slightly out of sequence get a chance to settle into order before being rendered. The behaviour ends up being similar to what you'd get with an unreliable transport like UDP — latest data wins, stale data is gone — but implemented at the application layer where the queue depth and fallback behaviour are tuneable.

The serialization pipeline

The wire format is a three-stage sandwich designed to minimize payload size:

1. Custom minimization. Rich entity state is encoded into compact key-value strings with single-character keys. A field like worldId becomes a, coordinates become b and c. This strips the structural overhead that makes JSON verbose.

2. JSON serialization + Brotli compression. The minimized structure is serialized to JSON (the interchange format between Rust's serde and the browser's JSON.parse), then Brotli-compressed. Brotli handles the remaining redundancy in the JSON encoding itself.

3. MessagePack envelope. The compressed bytes are wrapped in a MessagePack binary map with a type discriminator and sent as a WebSocket binary frame. This eliminates UTF-8 framing overhead on the wire.

On the client side, the pipeline reverses: MessagePack decode, Brotli decompress, JSON parse, custom de-minimization. The result is a typed StateData object ready for the state buffer.

How the pieces fit together

The system is layered so each piece handles one concern:

The outbound queue handles backpressure. The delta protocol handles bandwidth. The forced-full cadence and client feedback handle recovery. The client buffer handles sequencing and smooth application. No single layer tries to solve everything, and each one is independent enough that tuning one (queue depth, forced-full interval, buffer size) doesn't break another.

The result is a state sync pipeline that handles thousands of entities across multiple concurrent clients, with typical per-update payloads around 1.5KB and peak bandwidth under 15KB/s — down from the 1.5MB per update and 10MB/s of the original always-full-state implementation.