screentinker/docs/148-server-eviction-storm-analysis.md

# #148 — server eviction-storm robustness (field-safe net)

**Status: investigation + fix spec. No code changed, nothing deployed.**

This is the FIELD-SAFE NET for #148: it makes the SERVER absorb a client that opens
duplicate/rapid sockets, so a thrashing device converges to ONE stable connection and stays
online — protecting devices that will never get the APK fix (the duplicate-socket root cause
is a separate task). Complementary to the APK fix; likely shippable as server-only 1.9.2-patch2.

## Confirmed eviction behaviour (ws/deviceSocket.js)

- **`evictPriorSocket(deviceId, exceptSocketId)`** looks up the device's current connection
  (`heartbeat.getConnection`) and, if it's a different socket, `oldSocket.disconnect(true)` —
  the **OLD** socket is dropped ("io server disconnect" on that client). Single-session
  enforcement: the newest register wins, the prior is evicted. Called on the reconnect path
  (line 486) and the fingerprint/pairing path (line 397), **after** both rate gates.

- **Eviction does NOT bypass the limiters.** The register handler runs, in order:
  `flapLimiter.check` (line ~325) → `validateDeviceToken` → `reconnectThrottle.check`
  (line ~469) → **then** `evictPriorSocket` (line 486). A reconnect must pass both gates
  before it evicts anyone.

- **Why 8-in-9s tripped neither gate:**
  - *Flap-limiter* = `connectRateMax 20 / 5min`. 8-in-9s is far under it — it correctly does
    nothing (it targets a device flapping over a LONG window, not a 9-second burst).
  - *Reconnect-throttle* = `reconnectBaseMax 5 / reconnectWindowMs 10s` — which 8-in-9s WOULD
    trip… **except during the `reconnectWarmupMs` (30s) cold-start window, where only the hard
    ceiling (20/10s) applies** (`if (warmup) return allow(...)`). 8 < 20 → allowed.
    **The patch1 deploy restarted the container, so the observed storm was almost certainly
    inside that 30s warm-up** — every device reconnecting after a restart is warm-up-lenient,
    and a *thrashing* device is undamped for those 30s.
  - This matches the evidence exactly: **flat loop, no flap/throttle fires, pure eviction
    churn** — eviction is cheap (a disconnect + a register), so the server stays healthy.

- **Post-warm-up, the server ALREADY absorbs it.** Once the 30s warm-up passes, the throttle
  rejects the 6th+ reconnect and `return`s **before** `evictPriorSocket` — so the **incumbent
  socket is preserved** and the device stays online while the duplicate is refused. So the
  robustness gap is specifically the **warm-up window after every restart/deploy**, plus the
  churn itself (the authoritative socket flipping N times) even when it ends online.

## One thrashing client, or two instances?

**Socket.IO's `io server disconnect` does NOT auto-reconnect the client** (unlike
`transport close`/`ping timeout`). So an *evicted* socket's client does **not** come back on
its own — the 8 fresh sockets are the **client explicitly opening new connections**, i.e. the
APK duplicate-socket bug (one buggy instance opening repeatedly, and/or ≥2 service instances
each with its own socket). It is **not** an eviction→reconnect loop. Definitive attribution
(one instance vs two) needs the MAXHUB `logcat`, but the server pattern rules out
"eviction causes the reconnect." Consequence for the fix: the net must tolerate a client that
keeps opening duplicates, and simply **stop the churn and keep the device online on one socket**.

## The risk to close

1. **Warm-up churn (every restart/deploy):** for 30s a thrasher's sockets all pass, each
   evicting the prior → the device's authoritative socket flips N times → dashboard
   online/offline thrash, and a race between "evict old / register new" can briefly leave the
   device with no live socket. It usually ends online on the last socket *if the client
   settles* — but nothing forces it to settle.
2. **The device following the wrong socket:** heartbeats key on `device_id` and update the one
   tracked connection, which points at the newest-registered socket. If the client's bug keeps
   advancing which socket it heartbeats, the tracked socket can go stale between flips.

## Fix spec — a warm-up-independent per-device SESSION-SETTLE debounce (do NOT implement yet)

Add a per-device settle window around eviction, checked in the register handler **just before
`evictPriorSocket`**, independent of the reconnect-throttle warm-up so it works right after a
restart:

- Track `sessionSettle: device_id -> { untilMs, socketId }`.
- On a genuine new register for X (post-auth):
  - **If X has a LIVE incumbent socket and `now < untilMs` and the new socket ≠ the held one:**
    this is a rapid duplicate → **SOFT-REFUSE the NEW socket** (`device:throttled
    {reason:'session_settle', retry_after_ms}` + disconnect the NEW socket) and **keep the
    incumbent**. Do NOT evict. The device stays online on the held socket; the duplicate is
    dropped (and, being `io server disconnect`, won't auto-reconnect).
  - **Liveness safeguard:** only refuse if the incumbent is genuinely alive (its socket is in
    the `/device` namespace). If the incumbent has died/gone, ACCEPT the new socket (a genuine
    takeover after the incumbent dropped) — never strand a device on a dead held socket.
  - **Else** (no live incumbent, or window elapsed): ACCEPT — `evictPriorSocket` + register as
    today, and set `sessionSettle[X] = { untilMs: now + SESSION_SETTLE_MS, socketId }`.
- `SESSION_SETTLE_MS` env-tunable, ~2–3s: long enough to swallow a duplicate burst, short
  enough that a genuine device MOVE (real new socket after the incumbent is gone or the window
  passed) is accepted within seconds.

**Why prefer the incumbent (hold), not the newest:** the incumbent is the connection already
receiving heartbeats and proven live; holding it avoids the flip-churn entirely, and the
liveness safeguard means we never keep a dead one. (Prefer-newest was considered — it tracks
the app's latest socket but keeps flipping, and can't tell which duplicate the app will
heartbeat; hold-incumbent + a short window + the liveness safeguard is simpler and stays
online. A fuller convergence when the app advances its own socket is the APK fix's job — the
server net's contract is "stop the churn, stay online.")

**Alignment with the limiters / patch1:**
- This is a **soft refusal** (like the flap soft cooldown), **never a quarantine** — a paired,
  thrashing device is calmed, not locked out (reuses patch1's paired-safe philosophy).
- Single-session enforcement is intact: exactly one authoritative socket; a legitimate move
  still replaces cleanly once the incumbent is gone or the short window elapses.
- Genuinely abusive/unpaired flapping is unchanged — it still meets the flap-limiter over the
  long window and the reconnect-throttle's hard ceiling.
- Cheap and O(1) per register; no loop impact (the storm was already flat-loop).

*(Also consider raising the confidence of the existing net: the reconnect-throttle warm-up
leniency is what let the storm through — but warm-up exists so a full-fleet reconnect after a
deploy isn't throttled. The settle debounce is the right lever because it dampens a SINGLE
thrashing device_id without touching the fleet-reconnect leniency.)*

### Tests
- **Storm converges:** one device_id opens N sockets in a few seconds (in AND out of the
  warm-up window) → it ends with exactly ONE stable authoritative socket, stays `online`
  throughout, and is NOT hard-quarantined (paired) — no evict↔reconnect churn.
- **Legitimate single move:** a device with a dead/absent incumbent opening one new socket is
  accepted immediately (old replaced cleanly); single-session still enforced.
- **Liveness safeguard:** if the held incumbent dies during the window, the next new socket is
  accepted (not stranded).
- **Abuse still caught:** an unpaired/unprovisioned flap is still limited (flap-limiter /
  reconnect-throttle hard ceiling).
- **Health:** under an N-socket storm the loop stays flat (no maintenance/flap impact).

## Framing

Field-safe relief that works on **un-updated devices tonight** — it caps the server's own
contribution to the thrash (churn + the warm-up gap) and keeps a thrashing paired device
ONLINE on one connection. It does **not** fix the client opening duplicates (the APK
duplicate-socket bug) — that remains the root fix. Shippable as server-only **1.9.2-patch2**.