screentinker/docs/146-hardening-plan.md

# #146 — Final event-loop hardening pass (beta7, ALPHA-ONLY)

**Goal / acceptance bar:** a server that *cannot* be frozen or driven into a restart
loop by any single misbehaving device, a bloated table, or a SNAT'd request flood.
**Core invariant:** no synchronous operation may block the event loop for more than
~50ms, ever, regardless of table size or request rate. The system self-recovers to
healthy *without a process restart*.

Branch `fix/146-hardening` off the beta6 tip (keeps the crash-isolation from beta6;
beta7 = beta6 + this). **Do not bump/tag/release/deploy** — Dan bumps + pulls to alpha.

---

## (b) Confirmed failure model (verified against current code)

Two interlocked mechanisms, not one:

1. **TRIGGER** — a device flapping on a multi-second cadence (connect → ~5 content
   acks → disconnect → reconnect every ~3–5s). Each cycle is expensive (register +
   `buildPlaylistPayload` + acks) *and* writes one `device_status_log` row/cycle,
   bloating the table over the retention window. The existing burst throttle
   (`reconnectBaseMax=5` / `reconnectWindowMs=10s`, deviceSocket.js:425) does **not**
   catch it: ~2–3 connects/10s passes clean. → **Item B**.
2. **AMPLIFIER** — `pruneStatusLog()` (db/database.js:763) is a **whole-table
   `ROW_NUMBER() OVER (PARTITION BY device_id …)` sort**. On the bloated table
   (1,116,544 rows in the incident) it blocks the loop **40–48s synchronously**
   (confirmed by `mean=NaNms p99=0ms` lag samples beside `[status-log] pruned` lines).
   It runs on the heartbeat interval **and at startup** → a bloated table freezes boot
   → healthcheck fails → restart. → **Item A**.
3. **INTERLOCK** — the reconnect throttle is in-memory and resets every restart, then
   re-enters a 30s warmup where throttling is relaxed. The prune-induced restart loop
   wipes the throttle's history every cycle, so the flapper is never caught. **Ending
   the restart loop (Item A) is what lets the flap limiter (Item B) gain traction.**

---

## (a) Blast-radius audit — every synchronous, potentially-unbounded op

Legend: **EXPOSED** = can block >50ms under load/growth; **MITIGATED** = already bounded.

### Maintenance sweeps
| Path | Shape | Worst-case block | Reachable in outage? | Class | Fix |
|---|---|---|---|---|---|
| `pruneStatusLog` (db/database.js:763) | whole-table `ROW_NUMBER` sort | **40–48s @ 1.1M rows** (O(N log N)) | YES (interval + startup) | **EXPOSED — the amplifier** | A |
| `pruneLag` (loop-lag.js:92) | indexed range `DELETE … sampled_at < ?` | O(deleted); one statement can delete all-old if stalled | YES (interval + startup) | **EXPOSED** | E |
| event_loop_lag INSERT (loop-lag.js:71) | sync INSERT per sample | small, but per-tick on the loop; worsens under DB contention | YES | **EXPOSED (freq)** | E |
| `pruneTelemetry(id)` (database.js:773) | per-device `NOT IN (SELECT … LIMIT 6000)`, idx_telemetry_device | one device's excess in one DELETE; **runs per-heartbeat** (deviceSocket.js:626) | partial | **EXPOSED (freq)** | A (helper + throttle) |
| play_logs 90d prune (heartbeat.js:64) | indexed range DELETE | O(>90d rows) one statement | YES (interval) | **EXPOSED** | A |
| `pruneProvisioningDevices` (heartbeat.js:94) | DELETE devices + FK cascade | provision-flood → big cascade delete | YES (interval) | **EXPOSED** | A |
| `pruneScreenshots(id)` (database.js:785) | per-device keep-newest-1 | ~rows-per-device | partial | MITIGATED | — |
| team/workspace invite prunes (heartbeat.js:73,78) | expiry DELETE, small tables | negligible | YES | MITIGATED (route for uniformity) | A(opt) |

### Hot paths
| Path | Worst-case | Class | Fix |
|---|---|---|---|
| `device:register` (deviceSocket.js:282) — token validate, blocked SELECT, evict, DB writes, `buildPlaylistPayload` | full cost per flap; burst throttle misses 3–5s flappers | **EXPOSED** | B |
| content-ack (deviceSocket.js:647) | limiter sheds the loop cost, **but `state` Map has no eviction** (unbounded device keys) | **EXPOSED (unbounded map)** | E |
| `/api/update/check` (server.js:585) | `resolveApkPath` `existsSync`×2 **unconditional** (even on rate-backoff no-offer); `console.log` per check | **EXPOSED** | C |
| `/download/apk` (server.js:734) | `resolveApkPath` per req; **IP-keyed** log throttle (SNAT-collapsed → hid the flood); no concurrency/rate/band cap on 8.8MB `sendFile` | **EXPOSED** | C |

### Unbounded in-memory structures (invariant: every Map bounded)
`content-ack state`, `lastPlayLogAt`, `lastReclaimRejectLogAt`, writer `lastWritten`
have **no eviction** — grow with distinct device_ids (a SNAT flood minting provisioning
ids inflates them). `otaDownloadLoggedAt` is **IP-keyed** (wrong under SNAT). Bounded
already: reconnect-throttle (#146 sweep), ota-breaker (#144 sweep), `pendingOfflines`,
`evictedSockets`. → **Item E adds a shared bounded-map sweep; Item C replaces the IP map.**

### Logging
Band lines, per-request OTA lines, repetitive "Device reconnected" — synchronous
`console.log` at storm rate blocks the loop. → **Item E** coalescing buffer.

---

## Identity fallback chain (applies to Items B & D — HARD CONSTRAINT)

SNAT: the whole fleet egresses as one IP (10.10.10.1). **Never key on IP.** Resolve
identity as: `device_id` → `fingerprint` (map via `device_fingerprints`→device_id when
it resolves, else the raw fingerprint string) → `device_token` → a **single bounded
GLOBAL anon bucket**. `fingerprint` is present in the `device:register` payload
(deviceSocket.js:289). An unidentifiable client is still bucketed (global anon) so an
anonymous flood is capped, never unthrottled. Mirrors `lib/ota-breaker`'s
device_id-or-version fallback. Extracted as `lib/device-identity.js`.

---

## (c) Sequenced implementation plan — isolated commits

- **A. Non-blocking maintenance.** New `lib/chunked-prune.js` (`chunkedPrune`: bounded
  `LIMIT` batch DELETE + `setImmediate` yield between batches, optional band-gate,
  re-entrancy guard). Rewrite `pruneStatusLog` → per-device indexed prune (keep newest
  `statusLogMaxRowsPerDevice` per device via idx_device_status_log_device_ts; delete
  older + past retention), chunked, async, band-gated on the interval / un-gated at
  startup. Route play_logs, provisioning-cascade, telemetry, event_loop_lag prune
  through the helper. `STATUS_LOG_PRUNE_BATCH=2000`. Callers fire-and-forget, no
  stacking. **Tests:** correctness, ≥300k non-blocking in batches, band-gate, re-entrancy.
- **B. Flap-rate limiter.** New `lib/flap-limiter.js` — per-identity long-window connect
  frequency (`CONNECT_RATE_WINDOW_MS=300000`, `CONNECT_RATE_MAX=20`), keyed via the
  fallback chain, bounded (sweep + global anon bucket). Checked at the register gate
  before heavy work; over-limit → backoff notice + disconnect. Optional auto-quarantine
  via Item D block after N refusals. **Tests:** 4s-flapper refused, normal never,
  two device_ids independent, fingerprint-bucketed, anon-bucketed, never IP.
- **C. OTA under SNAT.** Early-return before any FS on no-offer verdict; cache APK
  resolution (path/size/mtime at boot + interval refresh); `/download/apk` global
  concurrency + global rate + band shed (503 Retry-After); replace IP log-throttle with
  a bounded aggregate per-window counter. **Tests:** rate-backoff ⇒ 0 fs calls; download
  sheds 429/503 past global cap + under critical; nothing keys on IP.
- **D. Operator block.** Move blocked check behind the identity fallback chain (catch a
  device_id-less reconnect of a blocked device via fingerprint→device_id); dashboard
  block/unblock toggle (DB write + UI + next-register effect, no restart); in-code outage
  procedure. **Tests:** blocked rejected cheaply at handshake; device_id-less+mapped-
  fingerprint still rejected; unblock next-register.
- **E. Log/write self-protection.** `lib/log-coalescer.js` (dedup+count, interval flush,
  bounded); buffer+batch event_loop_lag inserts; route its prune through `chunkedPrune`;
  add a shared bounded-map sweep for the un-evicted per-device Maps. **Tests:** N identical
  lines ⇒ 1 summarized, buffer bounded.
- **STORM HARNESS:** flapper + pre-bloated status_log + OTA flood from one SNAT IP +
  a sweep, all at once; assert loop-lag never enters a multi-second freeze and the
  server stays responsive (no NaN-sample block).

## Before/after worst-case blocking (targets)
| Hot path | Before | After (target) |
|---|---|---|
| pruneStatusLog @1.1M | 40–48s | <50ms/batch, many batches |
| play_logs / provisioning / lag prune | O(all-old) one stmt | <50ms/batch |
| /api/update/check (no-offer) | existsSync×2 + log | 0 FS, coalesced log |
| /download/apk flood | unbounded sendFile + hidden | global-capped, 503 shed, visible |
| event_loop_lag telemetry | sync INSERT/sample | batched flush |
| device:register (flapper) | full build every ~4s | refused at gate, cheap |