docs(#148): server eviction-storm analysis (field-safe net spec)

docs/148-server-eviction-storm-analysis.md

@@ -0,0 +1,124 @@
+# #148 — server eviction-storm robustness (field-safe net)
+
+**Status: investigation + fix spec. No code changed, nothing deployed.**
+
+This is the FIELD-SAFE NET for #148: it makes the SERVER absorb a client that opens
+duplicate/rapid sockets, so a thrashing device converges to ONE stable connection and stays
+online — protecting devices that will never get the APK fix (the duplicate-socket root cause
+is a separate task). Complementary to the APK fix; likely shippable as server-only 1.9.2-patch2.
+
+## Confirmed eviction behaviour (ws/deviceSocket.js)
+
+- **`evictPriorSocket(deviceId, exceptSocketId)`** looks up the device's current connection
+  (`heartbeat.getConnection`) and, if it's a different socket, `oldSocket.disconnect(true)` —
+  the **OLD** socket is dropped ("io server disconnect" on that client). Single-session
+  enforcement: the newest register wins, the prior is evicted. Called on the reconnect path
+  (line 486) and the fingerprint/pairing path (line 397), **after** both rate gates.
+
+- **Eviction does NOT bypass the limiters.** The register handler runs, in order:
+  `flapLimiter.check` (line ~325) → `validateDeviceToken` → `reconnectThrottle.check`
+  (line ~469) → **then** `evictPriorSocket` (line 486). A reconnect must pass both gates
+  before it evicts anyone.
+
+- **Why 8-in-9s tripped neither gate:**
+  - *Flap-limiter* = `connectRateMax 20 / 5min`. 8-in-9s is far under it — it correctly does
+    nothing (it targets a device flapping over a LONG window, not a 9-second burst).
+  - *Reconnect-throttle* = `reconnectBaseMax 5 / reconnectWindowMs 10s` — which 8-in-9s WOULD
+    trip… **except during the `reconnectWarmupMs` (30s) cold-start window, where only the hard
+    ceiling (20/10s) applies** (`if (warmup) return allow(...)`). 8 < 20 → allowed.
+    **The patch1 deploy restarted the container, so the observed storm was almost certainly
+    inside that 30s warm-up** — every device reconnecting after a restart is warm-up-lenient,
+    and a *thrashing* device is undamped for those 30s.
+  - This matches the evidence exactly: **flat loop, no flap/throttle fires, pure eviction
+    churn** — eviction is cheap (a disconnect + a register), so the server stays healthy.
+
+- **Post-warm-up, the server ALREADY absorbs it.** Once the 30s warm-up passes, the throttle
+  rejects the 6th+ reconnect and `return`s **before** `evictPriorSocket` — so the **incumbent
+  socket is preserved** and the device stays online while the duplicate is refused. So the
+  robustness gap is specifically the **warm-up window after every restart/deploy**, plus the
+  churn itself (the authoritative socket flipping N times) even when it ends online.
+
+## One thrashing client, or two instances?
+
+**Socket.IO's `io server disconnect` does NOT auto-reconnect the client** (unlike
+`transport close`/`ping timeout`). So an *evicted* socket's client does **not** come back on
+its own — the 8 fresh sockets are the **client explicitly opening new connections**, i.e. the
+APK duplicate-socket bug (one buggy instance opening repeatedly, and/or ≥2 service instances
+each with its own socket). It is **not** an eviction→reconnect loop. Definitive attribution
+(one instance vs two) needs the MAXHUB `logcat`, but the server pattern rules out
+"eviction causes the reconnect." Consequence for the fix: the net must tolerate a client that
+keeps opening duplicates, and simply **stop the churn and keep the device online on one socket**.
+
+## The risk to close
+
+1. **Warm-up churn (every restart/deploy):** for 30s a thrasher's sockets all pass, each
+   evicting the prior → the device's authoritative socket flips N times → dashboard
+   online/offline thrash, and a race between "evict old / register new" can briefly leave the
+   device with no live socket. It usually ends online on the last socket *if the client
+   settles* — but nothing forces it to settle.
+2. **The device following the wrong socket:** heartbeats key on `device_id` and update the one
+   tracked connection, which points at the newest-registered socket. If the client's bug keeps
+   advancing which socket it heartbeats, the tracked socket can go stale between flips.
+
+## Fix spec — a warm-up-independent per-device SESSION-SETTLE debounce (do NOT implement yet)
+
+Add a per-device settle window around eviction, checked in the register handler **just before
+`evictPriorSocket`**, independent of the reconnect-throttle warm-up so it works right after a
+restart:
+
+- Track `sessionSettle: device_id -> { untilMs, socketId }`.
+- On a genuine new register for X (post-auth):
+  - **If X has a LIVE incumbent socket and `now < untilMs` and the new socket ≠ the held one:**
+    this is a rapid duplicate → **SOFT-REFUSE the NEW socket** (`device:throttled
+    {reason:'session_settle', retry_after_ms}` + disconnect the NEW socket) and **keep the
+    incumbent**. Do NOT evict. The device stays online on the held socket; the duplicate is
+    dropped (and, being `io server disconnect`, won't auto-reconnect).
+  - **Liveness safeguard:** only refuse if the incumbent is genuinely alive (its socket is in
+    the `/device` namespace). If the incumbent has died/gone, ACCEPT the new socket (a genuine
+    takeover after the incumbent dropped) — never strand a device on a dead held socket.
+  - **Else** (no live incumbent, or window elapsed): ACCEPT — `evictPriorSocket` + register as
+    today, and set `sessionSettle[X] = { untilMs: now + SESSION_SETTLE_MS, socketId }`.
+- `SESSION_SETTLE_MS` env-tunable, ~2–3s: long enough to swallow a duplicate burst, short
+  enough that a genuine device MOVE (real new socket after the incumbent is gone or the window
+  passed) is accepted within seconds.
+
+**Why prefer the incumbent (hold), not the newest:** the incumbent is the connection already
+receiving heartbeats and proven live; holding it avoids the flip-churn entirely, and the
+liveness safeguard means we never keep a dead one. (Prefer-newest was considered — it tracks
+the app's latest socket but keeps flipping, and can't tell which duplicate the app will
+heartbeat; hold-incumbent + a short window + the liveness safeguard is simpler and stays
+online. A fuller convergence when the app advances its own socket is the APK fix's job — the
+server net's contract is "stop the churn, stay online.")
+
+**Alignment with the limiters / patch1:**
+- This is a **soft refusal** (like the flap soft cooldown), **never a quarantine** — a paired,
+  thrashing device is calmed, not locked out (reuses patch1's paired-safe philosophy).
+- Single-session enforcement is intact: exactly one authoritative socket; a legitimate move
+  still replaces cleanly once the incumbent is gone or the short window elapses.
+- Genuinely abusive/unpaired flapping is unchanged — it still meets the flap-limiter over the
+  long window and the reconnect-throttle's hard ceiling.
+- Cheap and O(1) per register; no loop impact (the storm was already flat-loop).
+
+*(Also consider raising the confidence of the existing net: the reconnect-throttle warm-up
+leniency is what let the storm through — but warm-up exists so a full-fleet reconnect after a
+deploy isn't throttled. The settle debounce is the right lever because it dampens a SINGLE
+thrashing device_id without touching the fleet-reconnect leniency.)*
+
+### Tests
+- **Storm converges:** one device_id opens N sockets in a few seconds (in AND out of the
+  warm-up window) → it ends with exactly ONE stable authoritative socket, stays `online`
+  throughout, and is NOT hard-quarantined (paired) — no evict↔reconnect churn.
+- **Legitimate single move:** a device with a dead/absent incumbent opening one new socket is
+  accepted immediately (old replaced cleanly); single-session still enforced.
+- **Liveness safeguard:** if the held incumbent dies during the window, the next new socket is
+  accepted (not stranded).
+- **Abuse still caught:** an unpaired/unprovisioned flap is still limited (flap-limiter /
+  reconnect-throttle hard ceiling).
+- **Health:** under an N-socket storm the loop stays flat (no maintenance/flap impact).
+
+## Framing
+
+Field-safe relief that works on **un-updated devices tonight** — it caps the server's own
+contribution to the thrash (churn + the warm-up gap) and keeps a thrashing paired device
+ONLINE on one connection. It does **not** fix the client opening duplicates (the APK
+duplicate-socket bug) — that remains the root fix. Shippable as server-only **1.9.2-patch2**.