1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updated the item-B section for the in-memory time-limited auto-quarantine (no DB block,
auto-clears) and the item-C section for band-aware downloads (serve freely when healthy,
caps only under load). Both reference the new /api/status debug observability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s
while the table is still large (chunked startup prune trickling in the background), and
the backlog drains to the cap with the server responsive throughout. The old whole-table
sort froze boot ~40s. Fallout doc gets the P2 findings section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deliverables 3 & 4: per-item blast radius + soak signals, and the before/after
worst-case synchronous blocking for every hot path touched (prune, sweeps, OTA,
register, telemetry). Ends with the A<->B interlock note (ship together).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
