screentinker/docs/146-hardening-fallout.md

#146 hardening — Fallout summary + before/after (beta7, ALPHA-ONLY)

Branch fix/146-hardening (off beta6). Isolated commits A–E + storm harness. Suite 267/267. Not bumped/tagged/released — Dan bumps via bump-version.sh + pulls to alpha. Below: what each change touches, wider blast radius, what a soak watcher should watch, and the measured worst-case blocking cost per hot path.

What each item touches / blast radius / soak signals

A — non-blocking maintenance (chunked-prune.js, db/database.js, heartbeat.js)

• Touches: pruneStatusLog rewritten (per-device, chunked, async, band-gated, re-entrant); play_logs + provisioning prunes chunked; pruneTelemetry bounded; new index idx_devices_provisioning; heartbeat interval maintenance moved to async runMaintenance(). pruneStatusLog / pruneProvisioningDevices are now async — every caller must await (heartbeat + the two prune tests updated).
• Blast radius: heartbeat interval restructured; offline-marking unchanged & still synchronous. All table-growth sweeps now ride chunkedDelete.
• Soak signals: startup prune runs in the background (boot no longer blocks on a bloated table — self-heals Bold's on-disk bloat on first deploy). Interval maintenance is band-gated: under sustained non-normal lag it defers and resumes when band returns to normal — expected, not a fault. Log line format changed: [status-log] pruned N (per-device, newest 500/device + 3d retention, batches of 2000).

B — flap-rate limiter (flap-limiter.js, device-identity.js, register gate)

• Touches: new refusal at the register gate for a device connecting > CONNECT_RATE_MAX (20) per CONNECT_RATE_WINDOW_MS (5min), keyed via the identity chain. Emits device:throttled {reason:'connect_rate'} + disconnect. Sweep started in server.js.
• Wider blast radius: shares the device:throttled event with the #142 reconnect throttle (clients already handle it). Auto-quarantine writes devices.blocked=1 after CONNECT_RATE_QUARANTINE_TRIPS (5) trips — a hard flapper self-blocks.
• Soak signals: [flap] refused … and [flap] auto-quarantined … logs. A legit device on a flaky network reconnecting >20×/5min would be refused — if false positives appear, raise CONNECT_RATE_MAX. The global anon bucket (cap 60) collectively caps truly-unidentifiable clients; real fleet devices always resolve to device_id/fingerprint.

C — OTA under SNAT (apk-cache.js, ota-download-guard.js, server.js)

• Touches: /api/update/check early-returns before any fs on no-offer; APK metadata cached (60s refresh); /download/apk gains global concurrency + rate caps + critical-band shed (503 Retry-After); IP-keyed log throttle replaced by a per-window served/shed aggregate.
• Behavior change (watch): downloads can now return 503 under flood/critical — clients retry per Retry-After. A freshly-swapped APK is picked up within the 60s cache refresh (slight delay by design). If legit downloads get shed, raise OTA_DOWNLOAD_MAX_CONCURRENT / OTA_DOWNLOAD_MAX_PER_WINDOW.
• Soak signals: [ota] downloads last 60s: X served, Y shed — a flood is now VISIBLE.

D — operator block (deviceSocket, routes/devices.js, frontend)

• Touches: blocked check resolves the effective device_id via the identity chain (catches a device_id-less reconnect of a blocked device); POST /api/devices/:id/{block, unblock}; a Block/Unblock button in device-detail.js + api.js.
• Blast radius: register gate ordering (block is still first); one new frontend button (needs a browser smoke on alpha — not covered by node --test).
• Soak signals: [blocked] … logs; block/unblock take effect on the device's next register with no restart. Combined with B, a hard flapper can be auto-blocked.

E — log/write self-protection (log-coalescer.js, loop-lag.js, content-ack-limiter.js, status-log-writer.js)

• Touches: high-frequency lines coalesced (band "still loaded", OTA check, "Device reconnected") — one summarized line per key per 30s; band CHANGES stay immediate. event_loop_lag inserts BUFFERED + batch-flushed every 10s (bounded buffer); its prune rides chunkedDelete. content-ack Map swept; writer lastWritten capped.
• Behavior change (watch): fewer, summarized log lines (… (x47 in 30s)); event_loop_lag table lags real time by up to 10s (/api/status is unaffected — it reads in-memory current, so real-time band/alerting is intact).

Before / after — worst-case synchronous blocking (measured)

Hot path	Before	After (measured)
pruneStatusLog @ 300k–1.1M rows	whole-table ROW_NUMBER sort, 40–48s (incident)	chunked; max event-loop gap <300ms across the storm harness (300k rows) / <250ms in the prune test
play_logs / provisioning / lag prune	O(all-old) in one DELETE	chunked, ≤ one 2000-row batch per statement (<50ms)
pruneTelemetry (per heartbeat)	NOT IN (SELECT … 6000) per call	bounded single OFFSET 6000 LIMIT 2000 statement
/api/update/check (no-offer)	existsSync×2 every poll	0 fs calls (early-return + cache)
/download/apk under flood	unbounded concurrent 8.8MB sendFile; flood hidden	global concurrency+rate capped; 503 shed; flood visible
event_loop_lag telemetry	synchronous INSERT per sample	buffered, batch-inserted every 10s
device:register (4s flapper)	full register + buildPlaylistPayload every ~4s	refused at the gate (identity resolve + one indexed SELECT), cheap

Interlock note

Item A ends the prune-induced restart loop; Item B's in-memory flap state now persists long enough to bite (it used to be wiped every ~40s by the restart). The two are a pair: ship/soak them together.