From 9922a0c30d6cbde7305ca2e49a4b58fa26055afc Mon Sep 17 00:00:00 2001 From: ScreenTinker Date: Thu, 2 Jul 2026 19:12:46 -0500 Subject: [PATCH] docs(#148): server eviction-storm analysis (field-safe net spec) --- docs/148-server-eviction-storm-analysis.md | 124 +++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/148-server-eviction-storm-analysis.md diff --git a/docs/148-server-eviction-storm-analysis.md b/docs/148-server-eviction-storm-analysis.md new file mode 100644 index 0000000..2c32752 --- /dev/null +++ b/docs/148-server-eviction-storm-analysis.md @@ -0,0 +1,124 @@ +# #148 — server eviction-storm robustness (field-safe net) + +**Status: investigation + fix spec. No code changed, nothing deployed.** + +This is the FIELD-SAFE NET for #148: it makes the SERVER absorb a client that opens +duplicate/rapid sockets, so a thrashing device converges to ONE stable connection and stays +online — protecting devices that will never get the APK fix (the duplicate-socket root cause +is a separate task). Complementary to the APK fix; likely shippable as server-only 1.9.2-patch2. + +## Confirmed eviction behaviour (ws/deviceSocket.js) + +- **`evictPriorSocket(deviceId, exceptSocketId)`** looks up the device's current connection + (`heartbeat.getConnection`) and, if it's a different socket, `oldSocket.disconnect(true)` — + the **OLD** socket is dropped ("io server disconnect" on that client). Single-session + enforcement: the newest register wins, the prior is evicted. Called on the reconnect path + (line 486) and the fingerprint/pairing path (line 397), **after** both rate gates. + +- **Eviction does NOT bypass the limiters.** The register handler runs, in order: + `flapLimiter.check` (line ~325) → `validateDeviceToken` → `reconnectThrottle.check` + (line ~469) → **then** `evictPriorSocket` (line 486). A reconnect must pass both gates + before it evicts anyone. + +- **Why 8-in-9s tripped neither gate:** + - *Flap-limiter* = `connectRateMax 20 / 5min`. 8-in-9s is far under it — it correctly does + nothing (it targets a device flapping over a LONG window, not a 9-second burst). + - *Reconnect-throttle* = `reconnectBaseMax 5 / reconnectWindowMs 10s` — which 8-in-9s WOULD + trip… **except during the `reconnectWarmupMs` (30s) cold-start window, where only the hard + ceiling (20/10s) applies** (`if (warmup) return allow(...)`). 8 < 20 → allowed. + **The patch1 deploy restarted the container, so the observed storm was almost certainly + inside that 30s warm-up** — every device reconnecting after a restart is warm-up-lenient, + and a *thrashing* device is undamped for those 30s. + - This matches the evidence exactly: **flat loop, no flap/throttle fires, pure eviction + churn** — eviction is cheap (a disconnect + a register), so the server stays healthy. + +- **Post-warm-up, the server ALREADY absorbs it.** Once the 30s warm-up passes, the throttle + rejects the 6th+ reconnect and `return`s **before** `evictPriorSocket` — so the **incumbent + socket is preserved** and the device stays online while the duplicate is refused. So the + robustness gap is specifically the **warm-up window after every restart/deploy**, plus the + churn itself (the authoritative socket flipping N times) even when it ends online. + +## One thrashing client, or two instances? + +**Socket.IO's `io server disconnect` does NOT auto-reconnect the client** (unlike +`transport close`/`ping timeout`). So an *evicted* socket's client does **not** come back on +its own — the 8 fresh sockets are the **client explicitly opening new connections**, i.e. the +APK duplicate-socket bug (one buggy instance opening repeatedly, and/or ≥2 service instances +each with its own socket). It is **not** an eviction→reconnect loop. Definitive attribution +(one instance vs two) needs the MAXHUB `logcat`, but the server pattern rules out +"eviction causes the reconnect." Consequence for the fix: the net must tolerate a client that +keeps opening duplicates, and simply **stop the churn and keep the device online on one socket**. + +## The risk to close + +1. **Warm-up churn (every restart/deploy):** for 30s a thrasher's sockets all pass, each + evicting the prior → the device's authoritative socket flips N times → dashboard + online/offline thrash, and a race between "evict old / register new" can briefly leave the + device with no live socket. It usually ends online on the last socket *if the client + settles* — but nothing forces it to settle. +2. **The device following the wrong socket:** heartbeats key on `device_id` and update the one + tracked connection, which points at the newest-registered socket. If the client's bug keeps + advancing which socket it heartbeats, the tracked socket can go stale between flips. + +## Fix spec — a warm-up-independent per-device SESSION-SETTLE debounce (do NOT implement yet) + +Add a per-device settle window around eviction, checked in the register handler **just before +`evictPriorSocket`**, independent of the reconnect-throttle warm-up so it works right after a +restart: + +- Track `sessionSettle: device_id -> { untilMs, socketId }`. +- On a genuine new register for X (post-auth): + - **If X has a LIVE incumbent socket and `now < untilMs` and the new socket ≠ the held one:** + this is a rapid duplicate → **SOFT-REFUSE the NEW socket** (`device:throttled + {reason:'session_settle', retry_after_ms}` + disconnect the NEW socket) and **keep the + incumbent**. Do NOT evict. The device stays online on the held socket; the duplicate is + dropped (and, being `io server disconnect`, won't auto-reconnect). + - **Liveness safeguard:** only refuse if the incumbent is genuinely alive (its socket is in + the `/device` namespace). If the incumbent has died/gone, ACCEPT the new socket (a genuine + takeover after the incumbent dropped) — never strand a device on a dead held socket. + - **Else** (no live incumbent, or window elapsed): ACCEPT — `evictPriorSocket` + register as + today, and set `sessionSettle[X] = { untilMs: now + SESSION_SETTLE_MS, socketId }`. +- `SESSION_SETTLE_MS` env-tunable, ~2–3s: long enough to swallow a duplicate burst, short + enough that a genuine device MOVE (real new socket after the incumbent is gone or the window + passed) is accepted within seconds. + +**Why prefer the incumbent (hold), not the newest:** the incumbent is the connection already +receiving heartbeats and proven live; holding it avoids the flip-churn entirely, and the +liveness safeguard means we never keep a dead one. (Prefer-newest was considered — it tracks +the app's latest socket but keeps flipping, and can't tell which duplicate the app will +heartbeat; hold-incumbent + a short window + the liveness safeguard is simpler and stays +online. A fuller convergence when the app advances its own socket is the APK fix's job — the +server net's contract is "stop the churn, stay online.") + +**Alignment with the limiters / patch1:** +- This is a **soft refusal** (like the flap soft cooldown), **never a quarantine** — a paired, + thrashing device is calmed, not locked out (reuses patch1's paired-safe philosophy). +- Single-session enforcement is intact: exactly one authoritative socket; a legitimate move + still replaces cleanly once the incumbent is gone or the short window elapses. +- Genuinely abusive/unpaired flapping is unchanged — it still meets the flap-limiter over the + long window and the reconnect-throttle's hard ceiling. +- Cheap and O(1) per register; no loop impact (the storm was already flat-loop). + +*(Also consider raising the confidence of the existing net: the reconnect-throttle warm-up +leniency is what let the storm through — but warm-up exists so a full-fleet reconnect after a +deploy isn't throttled. The settle debounce is the right lever because it dampens a SINGLE +thrashing device_id without touching the fleet-reconnect leniency.)* + +### Tests +- **Storm converges:** one device_id opens N sockets in a few seconds (in AND out of the + warm-up window) → it ends with exactly ONE stable authoritative socket, stays `online` + throughout, and is NOT hard-quarantined (paired) — no evict↔reconnect churn. +- **Legitimate single move:** a device with a dead/absent incumbent opening one new socket is + accepted immediately (old replaced cleanly); single-session still enforced. +- **Liveness safeguard:** if the held incumbent dies during the window, the next new socket is + accepted (not stranded). +- **Abuse still caught:** an unpaired/unprovisioned flap is still limited (flap-limiter / + reconnect-throttle hard ceiling). +- **Health:** under an N-socket storm the loop stays flat (no maintenance/flap impact). + +## Framing + +Field-safe relief that works on **un-updated devices tonight** — it caps the server's own +contribution to the thrash (churn + the warm-up gap) and keeps a thrashing paired device +ONLINE on one connection. It does **not** fix the client opening duplicates (the APK +duplicate-socket bug) — that remains the root fix. Shippable as server-only **1.9.2-patch2**.