docs(#146): hardening fallout summary + measured before/after blocking costs

Deliverables 3 & 4: per-item blast radius + soak signals, and the before/after
worst-case synchronous blocking for every hot path touched (prune, sweeps, OTA,
register, telemetry). Ends with the A<->B interlock note (ship together).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/146-hardening-fallout.md

@@ -0,0 +1,79 @@
+# #146 hardening — Fallout summary + before/after (beta7, ALPHA-ONLY)
+
+Branch `fix/146-hardening` (off beta6). Isolated commits A–E + storm harness. Suite
+**267/267**. **Not bumped/tagged/released** — Dan bumps via bump-version.sh + pulls to
+alpha. Below: what each change touches, wider blast radius, what a soak watcher should
+watch, and the measured worst-case blocking cost per hot path.
+
+## What each item touches / blast radius / soak signals
+
+### A — non-blocking maintenance (`chunked-prune.js`, `db/database.js`, `heartbeat.js`)
+- **Touches:** `pruneStatusLog` rewritten (per-device, chunked, async, band-gated,
+  re-entrant); play_logs + provisioning prunes chunked; `pruneTelemetry` bounded; new
+  index `idx_devices_provisioning`; heartbeat interval maintenance moved to async
+  `runMaintenance()`. **`pruneStatusLog` / `pruneProvisioningDevices` are now async** —
+  every caller must await (heartbeat + the two prune tests updated).
+- **Blast radius:** heartbeat interval restructured; offline-marking unchanged & still
+  synchronous. All table-growth sweeps now ride `chunkedDelete`.
+- **Soak signals:** startup prune runs in the background (boot no longer blocks on a
+  bloated table — self-heals Bold's on-disk bloat on first deploy). Interval maintenance
+  is **band-gated**: under sustained non-normal lag it *defers* and resumes when band
+  returns to normal — expected, not a fault. Log line format changed:
+  `[status-log] pruned N (per-device, newest 500/device + 3d retention, batches of 2000)`.
+
+### B — flap-rate limiter (`flap-limiter.js`, `device-identity.js`, register gate)
+- **Touches:** new refusal at the register gate for a device connecting > `CONNECT_RATE_MAX`
+  (20) per `CONNECT_RATE_WINDOW_MS` (5min), keyed via the identity chain. Emits
+  `device:throttled {reason:'connect_rate'}` + disconnect. Sweep started in server.js.
+- **Wider blast radius:** shares the `device:throttled` event with the #142 reconnect
+  throttle (clients already handle it). Auto-quarantine writes `devices.blocked=1` after
+  `CONNECT_RATE_QUARANTINE_TRIPS` (5) trips — a hard flapper self-blocks.
+- **Soak signals:** `[flap] refused …` and `[flap] auto-quarantined …` logs. A legit
+  device on a flaky network reconnecting >20×/5min would be refused — if false positives
+  appear, raise `CONNECT_RATE_MAX`. The global anon bucket (cap 60) collectively caps
+  truly-unidentifiable clients; real fleet devices always resolve to device_id/fingerprint.
+
+### C — OTA under SNAT (`apk-cache.js`, `ota-download-guard.js`, server.js)
+- **Touches:** `/api/update/check` early-returns before any fs on no-offer; APK
+  metadata cached (60s refresh); `/download/apk` gains global concurrency + rate caps +
+  critical-band shed (**503 Retry-After**); IP-keyed log throttle replaced by a per-window
+  served/shed aggregate.
+- **Behavior change (watch):** downloads can now return **503** under flood/critical —
+  clients retry per Retry-After. A freshly-swapped APK is picked up within the 60s cache
+  refresh (slight delay by design). If legit downloads get shed, raise
+  `OTA_DOWNLOAD_MAX_CONCURRENT` / `OTA_DOWNLOAD_MAX_PER_WINDOW`.
+- **Soak signals:** `[ota] downloads last 60s: X served, Y shed` — a flood is now VISIBLE.
+
+### D — operator block (`deviceSocket`, `routes/devices.js`, frontend)
+- **Touches:** blocked check resolves the effective device_id via the identity chain
+  (catches a device_id-less reconnect of a blocked device); `POST /api/devices/:id/{block,
+  unblock}`; a Block/Unblock button in `device-detail.js` + `api.js`.
+- **Blast radius:** register gate ordering (block is still first); one new frontend
+  button (needs a browser smoke on alpha — not covered by node --test).
+- **Soak signals:** `[blocked] …` logs; block/unblock take effect on the device's next
+  register with no restart. Combined with B, a hard flapper can be auto-blocked.
+
+### E — log/write self-protection (`log-coalescer.js`, `loop-lag.js`, `content-ack-limiter.js`, `status-log-writer.js`)
+- **Touches:** high-frequency lines coalesced (band "still loaded", OTA check, "Device
+  reconnected") — one summarized line per key per 30s; band CHANGES stay immediate.
+  `event_loop_lag` inserts BUFFERED + batch-flushed every 10s (bounded buffer);
+  its prune rides `chunkedDelete`. content-ack Map swept; writer `lastWritten` capped.
+- **Behavior change (watch):** fewer, summarized log lines (`… (x47 in 30s)`);
+  `event_loop_lag` table lags real time by up to 10s (**/api/status is unaffected** — it
+  reads in-memory current, so real-time band/alerting is intact).
+
+## Before / after — worst-case synchronous blocking (measured)
+| Hot path | Before | After (measured) |
+|---|---|---|
+| `pruneStatusLog` @ 300k–1.1M rows | whole-table `ROW_NUMBER` sort, **40–48s** (incident) | chunked; **max event-loop gap <300ms** across the storm harness (300k rows) / <250ms in the prune test |
+| play_logs / provisioning / lag prune | O(all-old) in one DELETE | chunked, ≤ one 2000-row batch per statement (<50ms) |
+| `pruneTelemetry` (per heartbeat) | `NOT IN (SELECT … 6000)` per call | bounded single `OFFSET 6000 LIMIT 2000` statement |
+| `/api/update/check` (no-offer) | `existsSync`×2 every poll | **0 fs calls** (early-return + cache) |
+| `/download/apk` under flood | unbounded concurrent 8.8MB `sendFile`; flood hidden | global concurrency+rate capped; **503 shed**; flood visible |
+| `event_loop_lag` telemetry | synchronous INSERT per sample | buffered, **batch-inserted every 10s** |
+| `device:register` (4s flapper) | full register + `buildPlaylistPayload` every ~4s | refused at the gate (identity resolve + one indexed SELECT), cheap |
+
+## Interlock note
+Item A ends the prune-induced restart loop; Item B's in-memory flap state now persists
+long enough to bite (it used to be wiped every ~40s by the restart). The two are a pair:
+ship/soak them together.