#148 — server eviction-storm robustness (field-safe net)

Status: investigation + fix spec. No code changed, nothing deployed.

This is the FIELD-SAFE NET for #148: it makes the SERVER absorb a client that opens duplicate/rapid sockets, so a thrashing device converges to ONE stable connection and stays online — protecting devices that will never get the APK fix (the duplicate-socket root cause is a separate task). Complementary to the APK fix; likely shippable as server-only 1.9.2-patch2.

Confirmed eviction behaviour (ws/deviceSocket.js)

evictPriorSocket(deviceId, exceptSocketId) looks up the device's current connection (heartbeat.getConnection) and, if it's a different socket, oldSocket.disconnect(true) — the OLD socket is dropped ("io server disconnect" on that client). Single-session enforcement: the newest register wins, the prior is evicted. Called on the reconnect path (line 486) and the fingerprint/pairing path (line 397), after both rate gates.
Eviction does NOT bypass the limiters. The register handler runs, in order: flapLimiter.check (line ~325) → validateDeviceToken → reconnectThrottle.check (line ~469) → then evictPriorSocket (line 486). A reconnect must pass both gates before it evicts anyone.
Why 8-in-9s tripped neither gate:
- Flap-limiter = connectRateMax 20 / 5min. 8-in-9s is far under it — it correctly does nothing (it targets a device flapping over a LONG window, not a 9-second burst).
- Reconnect-throttle = reconnectBaseMax 5 / reconnectWindowMs 10s — which 8-in-9s WOULD trip… except during the reconnectWarmupMs (30s) cold-start window, where only the hard ceiling (20/10s) applies (if (warmup) return allow(...)). 8 < 20 → allowed. The patch1 deploy restarted the container, so the observed storm was almost certainly inside that 30s warm-up — every device reconnecting after a restart is warm-up-lenient, and a thrashing device is undamped for those 30s.
- This matches the evidence exactly: flat loop, no flap/throttle fires, pure eviction churn — eviction is cheap (a disconnect + a register), so the server stays healthy.
Post-warm-up, the server ALREADY absorbs it. Once the 30s warm-up passes, the throttle rejects the 6th+ reconnect and returns before evictPriorSocket — so the incumbent socket is preserved and the device stays online while the duplicate is refused. So the robustness gap is specifically the warm-up window after every restart/deploy, plus the churn itself (the authoritative socket flipping N times) even when it ends online.

One thrashing client, or two instances?

Socket.IO's io server disconnect does NOT auto-reconnect the client (unlike transport close/ping timeout). So an evicted socket's client does not come back on its own — the 8 fresh sockets are the client explicitly opening new connections, i.e. the APK duplicate-socket bug (one buggy instance opening repeatedly, and/or ≥2 service instances each with its own socket). It is not an eviction→reconnect loop. Definitive attribution (one instance vs two) needs the MAXHUB logcat, but the server pattern rules out "eviction causes the reconnect." Consequence for the fix: the net must tolerate a client that keeps opening duplicates, and simply stop the churn and keep the device online on one socket.

The risk to close

Warm-up churn (every restart/deploy): for 30s a thrasher's sockets all pass, each evicting the prior → the device's authoritative socket flips N times → dashboard online/offline thrash, and a race between "evict old / register new" can briefly leave the device with no live socket. It usually ends online on the last socket if the client settles — but nothing forces it to settle.
The device following the wrong socket: heartbeats key on device_id and update the one tracked connection, which points at the newest-registered socket. If the client's bug keeps advancing which socket it heartbeats, the tracked socket can go stale between flips.

Fix spec — a warm-up-independent per-device SESSION-SETTLE debounce (do NOT implement yet)

Add a per-device settle window around eviction, checked in the register handler just before evictPriorSocket, independent of the reconnect-throttle warm-up so it works right after a restart:

Track sessionSettle: device_id -> { untilMs, socketId }.
On a genuine new register for X (post-auth):
- If X has a LIVE incumbent socket and now < untilMs and the new socket ≠ the held one: this is a rapid duplicate → SOFT-REFUSE the NEW socket (device:throttled {reason:'session_settle', retry_after_ms} + disconnect the NEW socket) and keep the incumbent. Do NOT evict. The device stays online on the held socket; the duplicate is dropped (and, being io server disconnect, won't auto-reconnect).
- Liveness safeguard: only refuse if the incumbent is genuinely alive (its socket is in the /device namespace). If the incumbent has died/gone, ACCEPT the new socket (a genuine takeover after the incumbent dropped) — never strand a device on a dead held socket.
- Else (no live incumbent, or window elapsed): ACCEPT — evictPriorSocket + register as today, and set sessionSettle[X] = { untilMs: now + SESSION_SETTLE_MS, socketId }.
SESSION_SETTLE_MS env-tunable, ~2–3s: long enough to swallow a duplicate burst, short enough that a genuine device MOVE (real new socket after the incumbent is gone or the window passed) is accepted within seconds.

Why prefer the incumbent (hold), not the newest: the incumbent is the connection already receiving heartbeats and proven live; holding it avoids the flip-churn entirely, and the liveness safeguard means we never keep a dead one. (Prefer-newest was considered — it tracks the app's latest socket but keeps flipping, and can't tell which duplicate the app will heartbeat; hold-incumbent + a short window + the liveness safeguard is simpler and stays online. A fuller convergence when the app advances its own socket is the APK fix's job — the server net's contract is "stop the churn, stay online.")

Alignment with the limiters / patch1:

This is a soft refusal (like the flap soft cooldown), never a quarantine — a paired, thrashing device is calmed, not locked out (reuses patch1's paired-safe philosophy).
Single-session enforcement is intact: exactly one authoritative socket; a legitimate move still replaces cleanly once the incumbent is gone or the short window elapses.
Genuinely abusive/unpaired flapping is unchanged — it still meets the flap-limiter over the long window and the reconnect-throttle's hard ceiling.
Cheap and O(1) per register; no loop impact (the storm was already flat-loop).

(Also consider raising the confidence of the existing net: the reconnect-throttle warm-up leniency is what let the storm through — but warm-up exists so a full-fleet reconnect after a deploy isn't throttled. The settle debounce is the right lever because it dampens a SINGLE thrashing device_id without touching the fleet-reconnect leniency.)

Tests

Storm converges: one device_id opens N sockets in a few seconds (in AND out of the warm-up window) → it ends with exactly ONE stable authoritative socket, stays online throughout, and is NOT hard-quarantined (paired) — no evict↔reconnect churn.
Legitimate single move: a device with a dead/absent incumbent opening one new socket is accepted immediately (old replaced cleanly); single-session still enforced.
Liveness safeguard: if the held incumbent dies during the window, the next new socket is accepted (not stranded).
Abuse still caught: an unpaired/unprovisioned flap is still limited (flap-limiter / reconnect-throttle hard ceiling).
Health: under an N-socket storm the loop stays flat (no maintenance/flap impact).

Framing

Field-safe relief that works on un-updated devices tonight — it caps the server's own contribution to the thrash (churn + the warm-up gap) and keeps a thrashing paired device ONLINE on one connection. It does not fix the client opening duplicates (the APK duplicate-socket bug) — that remains the root fix. Shippable as server-only 1.9.2-patch2.

8.3 KiB Raw Blame History Unescape Escape