screentinker/docs/146-hardening-fallout.md

#146 hardening — Fallout summary + before/after (beta7, ALPHA-ONLY)

Branch fix/146-hardening (off beta6). Isolated commits A–E + storm harness. Suite 267/267. Not bumped/tagged/released — Dan bumps via bump-version.sh + pulls to alpha. Below: what each change touches, wider blast radius, what a soak watcher should watch, and the measured worst-case blocking cost per hot path.

What each item touches / blast radius / soak signals

A — non-blocking maintenance (chunked-prune.js, db/database.js, heartbeat.js)

• Touches: pruneStatusLog rewritten (per-device, chunked, async, band-gated, re-entrant); play_logs + provisioning prunes chunked; pruneTelemetry bounded; new index idx_devices_provisioning; heartbeat interval maintenance moved to async runMaintenance(). pruneStatusLog / pruneProvisioningDevices are now async — every caller must await (heartbeat + the two prune tests updated).
• Blast radius: heartbeat interval restructured; offline-marking unchanged & still synchronous. All table-growth sweeps now ride chunkedDelete.
• Soak signals: startup prune runs in the background (boot no longer blocks on a bloated table — self-heals Bold's on-disk bloat on first deploy). Interval maintenance is band-gated: under sustained non-normal lag it defers and resumes when band returns to normal — expected, not a fault. Log line format changed: [status-log] pruned N (per-device, newest 500/device + 3d retention, batches of 2000).

B — flap-rate limiter (flap-limiter.js, device-identity.js, register gate)

• Touches: new refusal at the register gate for a device connecting > CONNECT_RATE_MAX (20) per CONNECT_RATE_WINDOW_MS (5min), keyed via the identity chain. Emits device:throttled {reason:'connect_rate'} + disconnect. Sweep started in server.js.
• Wider blast radius: shares the device:throttled event with the #142 reconnect throttle (clients already handle it).
• Auto-quarantine (P0 — behavior change): after CONNECT_RATE_QUARANTINE_TRIPS (5) trips a hard flapper is quarantined IN-MEMORY for CONNECT_RATE_QUARANTINE_MS (30min) and AUTO-CLEARS — it is NOT a DB block. A stuck-then-recovered device comes back on its own with no human action. devices.blocked is written only by an operator (dashboard / direct SQLite). (Was: a permanent blocked=1 auto-write — a self-healing auto-action must not survive as a durable DB row.)
• Soak signals: [flap] quarantined <id> for 30m after N trips (logged once at the START); repeat refusals are coalesced. Also visible on /api/status → debug.flap.{buckets,quarantined}. A legit device on a flaky network reconnecting

  20×/5min would be refused — if false positives appear, raise CONNECT_RATE_MAX. The global anon bucket (cap 60) collectively caps truly-unidentifiable clients; real fleet devices always resolve to device_id/fingerprint.

C — OTA under SNAT (apk-cache.js, ota-download-guard.js, server.js)

• Touches: /api/update/check early-returns before any fs on no-offer; APK metadata cached (60s refresh); /download/apk gains BAND-AWARE global concurrency + rate caps + critical-band shed (503 Retry-After); IP-keyed log throttle replaced by a per-window served/shed aggregate.
• Band-aware downloads (P1.2 — behavior): the concurrency/rate caps are a load-time backstop, not a healthy-state limiter. Under band=normal downloads serve FREELY (no cap) — a coordinated whole-fleet rollout is NOT staggered when the server is healthy. The caps engage only under elevated; critical sheds (503). This fixes rollout staggering (a shed 503 costs a client a full ~30-min re-check cycle).
• Behavior change (watch): downloads can return 503 only under elevated/critical — clients retry per Retry-After. A freshly-swapped APK is picked up within the 60s cache refresh (slight delay by design). Observable on /api/status → debug.ota_download.{inFlight,servedThisWindow,shedThisWindow}. If legit downloads get shed under load, raise OTA_DOWNLOAD_MAX_CONCURRENT / OTA_DOWNLOAD_MAX_PER_WINDOW.
• Soak signals: [ota] downloads last 60s: X served, Y shed — a flood is now VISIBLE.

D — operator block (deviceSocket, routes/devices.js, frontend)

• Touches: blocked check resolves the effective device_id via the identity chain (catches a device_id-less reconnect of a blocked device); POST /api/devices/:id/{block, unblock}; a Block/Unblock button in device-detail.js + api.js.
• Blast radius: register gate ordering (block is still first); one new frontend button (needs a browser smoke on alpha — not covered by node --test).
• Soak signals: [blocked] … logs; block/unblock take effect on the device's next register with no restart. Combined with B, a hard flapper can be auto-blocked.

E — log/write self-protection (log-coalescer.js, loop-lag.js, content-ack-limiter.js, status-log-writer.js)

• Touches: high-frequency lines coalesced (band "still loaded", OTA check, "Device reconnected") — one summarized line per key per 30s; band CHANGES stay immediate. event_loop_lag inserts BUFFERED + batch-flushed every 10s (bounded buffer); its prune rides chunkedDelete. content-ack Map swept; writer lastWritten capped.
• Behavior change (watch): fewer, summarized log lines (… (x47 in 30s)); event_loop_lag table lags real time by up to 10s (/api/status is unaffected — it reads in-memory current, so real-time band/alerting is intact).

/api/status.debug THROUGHPUT counters (soak observability)

Each subsystem now exposes gauges and throughput (running Total + last-completed- LastWindow, DEBUG_STATS_WINDOW_MS default 60s) so the server tells the flapper/flood story on its own — no client trust:

• flap.refusedLastWindow — refusals/window. Climbing while band=normal = the limiter absorbing a flapper cheaply (the healthy signature — this is what "36 buckets, 0 quarantined" looked like but couldn't show). Climbing WITH band elevated/critical = investigate.
• flap.quarantineStartsTotal/LastWindow — a quarantine EVENT is visible even though the quarantined gauge decays.
• ota_breaker.rateBackoffLastWindow — a device=none 1.8.x update-check flood shows here.
• ota_download.servedTotal/shedTotal — a download flood: shedTotal climbing = the global cap engaging (only under elevated/critical).
• maintenance.sweepsTotal — confirms the prune is FIRING on its interval, not stalled (with deleted/ms for cost). All aggregate-only, cheap in-memory reads.
• devices_connected — the ALWAYS-ON live-fleet gauge (top-level, next to loop_lag, never gated): devices with a live WS socket THIS INSTANT (from the heartbeat connection map), NOT devices.status='online' (which lags by the offline-timeout). The debug block is now admin-toggleable (Admin tab → "Status endpoint" → "Expose /api/status debug metrics"; persisted in app_settings, default follows STATUS_DEBUG_ENABLED); the toggle takes effect on the next status poll with no restart, and when off the debug key is omitted entirely while loop_lag + devices_connected remain.

Before / after — worst-case synchronous blocking (measured)

Hot path	Before	After (measured)
pruneStatusLog @ 300k–1.1M rows	whole-table ROW_NUMBER sort, 40–48s (incident)	chunked; max event-loop gap <300ms across the storm harness (300k rows) / <250ms in the prune test
play_logs / provisioning / lag prune	O(all-old) in one DELETE	chunked, ≤ one 2000-row batch per statement (<50ms)
pruneTelemetry (per heartbeat)	NOT IN (SELECT … 6000) per call	bounded single OFFSET 6000 LIMIT 2000 statement
/api/update/check (no-offer)	existsSync×2 every poll	0 fs calls (early-return + cache)
/download/apk under flood	unbounded concurrent 8.8MB sendFile; flood hidden	global concurrency+rate capped; 503 shed; flood visible
event_loop_lag telemetry	synchronous INSERT per sample	buffered, batch-inserted every 10s
device:register (4s flapper)	full register + buildPlaylistPayload every ~4s	refused at the gate (identity resolve + one indexed SELECT), cheap

Kill switches (env — disable a subsystem with a flip + restart, no redeploy)

Every new subsystem is disable-able so a misfire on alpha is neutralized without a code change or bisect. All read at process start.

Subsystem	Env	Disable value	Effect when off
Flap limiter	FLAP_LIMITER_ENABLED	false	check() always allows — no connect-frequency limiting
Auto-quarantine	CONNECT_RATE_QUARANTINE_TRIPS	0	flappers still cool down, but are never quarantined
Download guard	OTA_DOWNLOAD_GUARD_ENABLED	false	/download/apk never sheds (no concurrency/rate/band caps)
Maintenance band-gate	MAINTENANCE_BAND_GATE_ENABLED	false	interval maintenance runs regardless of loop-lag band
Flap window / cap (tune, not off)	CONNECT_RATE_MAX, CONNECT_RATE_WINDOW_MS	raise MAX	loosen if healthy devices are refused
Download caps (tune)	OTA_DOWNLOAD_MAX_CONCURRENT, OTA_DOWNLOAD_MAX_PER_WINDOW	raise	loosen the elevated-band caps
Prune batch (tune)	STATUS_LOG_PRUNE_BATCH	lower	smaller batches = smaller max block

Note the startup prune is intentionally NEVER band-gated (it must clear a boot-time backlog); MAINTENANCE_BAND_GATE_ENABLED only affects the interval run.

P2 findings (investigated; measured)

• Playlist build under mass reconnect — MITIGATED, no change. buildPlaylistPayload is synchronous (indexed SELECTs + one JSON.parse of the published snapshot). Measured with a 200-item snapshot: avg 0.078ms, max 0.70ms per call; a 230-device fleet-wide reconnect is ~18ms of CPU spread across 230 separate socket.io handler invocations (the loop yields between them), never one block. Per-call is the real unit and is far under the 50ms invariant. (test/playlist-build-cost.test.js.)
• Boot health during a large startup trim — CONFIRMED. Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s while the table is still large (prune trickling), and the backlog drains to the cap in the background with the server responsive throughout. The old whole-table sort froze boot ~40s. This is the point of the async/chunked, un-gated startup prune. (test/boot-health.test.js.)

Interlock note

Item A ends the prune-induced restart loop; Item B's in-memory flap state now persists long enough to bite (it used to be wiped every ~40s by the restart). The two are a pair: ship/soak them together.