Fallback for when the automatic fingerprint-match restore can't fire (factory reset / new
hardware / changed fingerprint). From a device's detail view (UX b): 'Restore from removed
device…' opens a picker of the workspace's removed-device snapshots (GET /devices/removed),
showing device_name + last_seen/removed_at + restore summary (orientation/timezone/playlist),
a Blocked badge, and an Apply action (POST /devices/:id/re-adopt) with a confirm — including an
explicit warning that applying a blocked snapshot re-blocks the target. Refreshes the device
view on success; handles 404/403/400; empty state. Fingerprint shown truncated on-hover only.
Frontend only. Local, no bump/tag.
Delete+re-pair mints a new device row whose INSERT omits every setting, silently resetting
orientation/name/playlist/etc to defaults (Bold MDM churn). Add a fingerprint-keyed
device_settings table (no FK to devices -> survives the cascade): snapshot on DELETE, auto-
restore on fingerprint-match re-pair (relinking the fp to the new id), operator re-adopt API
(GET /devices/removed + POST /devices/:id/re-adopt) for the changed-fingerprint case. Purge on
workspace/user/org deletion (no cross-tenant bleed). Orientation enum-validated on PUT + restore.
blocked preserved (re-enforced by the register kill-switch). Wall membership deferred (TODO).
Backend only — frontend re-adopt UI NOT built (awaiting API review). Local only, no bump/tag.
Close the disk-fill trap: with wal_autocheckpoint=0 a dead worker means nothing checkpoints.
Controller now respawns an unexpectedly-dead worker (bounded: RespawnMax/RespawnWindowMs +
backoff); on exhaustion it re-arms a conservative inline autocheckpoint (FallbackPages) on the
main connection + reclaims the backlog, logging loudly. Clean stopWalCheckpointer() teardown is
distinguished via a 'stopping' flag so SIGTERM never triggers respawn. Env-gated worker
fault-injection (WAL_CKPT_FAIL_START) for tests. Local only — no bump/tag.
Disable wal_autocheckpoint on the main connection; run PASSIVE checkpoints from a
worker_threads worker with its OWN better-sqlite3 handle, escalating to TRUNCATE on a
size high-water or PASSIVE-starvation. Removes the synchronous fsync-heavy checkpoint
from the event loop. Config: walCheckpointIntervalMs/HighWaterMB/StarvationRuns.
Local only — no bump/tag.
The player opened duplicate/rapid WebSocket connections for the same device_id: connect() was
unconditional (disconnect + forceNew socket) and reachable from every lifecycle entry point
(boot, service start, MainActivity/ProvisioningActivity bind, foreground re-bind, START_STICKY).
A ROM that re-binds on foreground (MAXHUB PROC_STATE_TOP, isBindService:true) therefore
re-invoked connect() repeatedly -> a burst of sockets, each evicted by the next (the 8-in-9s
storm). Fire TV never re-binds like that, so it never reproduced.
- ConnectionGuard (new, pure/testable — service is the shell, per the OtaThrottle pattern):
shouldOpenNewSocket(hasSocket, sameUrl, socketActive) — reuse a live/self-healing socket to
the same url; open a new one only when none is usable.
- WebSocketService: connect() is now idempotent (@Synchronized + ConnectionGuard) — every entry
point reuses the one socket, never opens a duplicate; body split into openSocket(). socketActive
/ currentUrl track the single socket.
- Single owner: onStartCommand now calls connect() so the SERVICE owns the one connection
(idempotent across START_STICKY restarts), not whichever activity binds.
- Reconnect discipline: on io server/client disconnect (which Socket.IO does NOT auto-reconnect)
mark the socket inert and schedule exactly ONE backed-off re-open — never a blind re-open loop;
a transport drop keeps socketActive=true so Socket.IO's own reconnect is reused.
Test: ConnectionGuardTest (5, incl. 8-rapid-binds-all-reuse). :app:testDebugUnitTest green
(ConnectionGuard 5, OtaThrottle 7, ScheduleEval 1). NOT bumped/signed/released — Dan builds+signs
with the BMG keystore; 1.9.2-patch2 (server net) covers un-updated devices.
Field-safe SERVER net. A device opening duplicate/rapid sockets (the APK duplicate-socket bug,
separate track) currently churns through evictions during the reconnect-throttle's 30s
post-restart WARM-UP (only the hard ceiling 20 applies then, so an 8-in-9s burst passes
undamped and each new socket evicts the prior). This makes the server absorb it: a thrashing
PAIRED device converges to ONE stable connection and stays online.
- lib/session-settle.js (decision only; bounded, swept): shouldHold(deviceId, incumbentAlive)
— true only when a socket was accepted for this device within SESSION_SETTLE_WINDOW_MS
(config, default 2500ms) AND the incumbent is alive. Warm-up-independent.
- deviceSocket register gate (just before evictPriorSocket): if a LIVE incumbent exists and
we're inside the window, SOFT-REFUSE the new socket (device:throttled reason=session_settle
+ disconnect) and keep the incumbent; else accept + evict + (re)arm the window.
- LIVENESS SAFEGUARD (load-bearing): only hold when the incumbent socket is actually in the
/device namespace — a dead/half-open incumbent is replaced, NEVER stranding the device (max
hold is the 2.5s window from the incumbent's accept, then any new socket is accepted).
- Soft refusal, NEVER a quarantine (reuses patch1's paired-safe philosophy); single-session
enforcement intact for a legitimate move; unpaired/abusive flapping still caught by the
existing limiters. O(1), no loop impact.
Tests (liveness first-class): live incumbent holds + DEAD incumbent replaced (not stranded);
storm of 6 sockets converges to ONE, stays online, not quarantined (during warm-up); single-
session move past the window replaces cleanly; unit decision + bounded sweep. The
evicted-socket-rearm test shrinks its settle window so it still exercises the eviction path.
Suite 336/336.
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
The flap-limiter could 30-min quarantine a PAIRED, legitimate device on reconnect churn.
Behind Bold's single SNAT IP a repeated edge flush -> every device reconnects -> trips flap
-> quarantined -> a recoverable blip becomes a SUSTAINED FLEET-WIDE LOCKOUT we caused.
check(key, now, {paired}) now skips (and clears) the quarantine escalation for a paired
device — it still gets the brief soft cooldown if it truly hammers, but never the long
lockout. The register gate computes paired = device_id && validateDeviceToken(...) (a
matching STORED token, false for missing/mismatch) so a spoofed device_id can't claim the
exemption; unpaired/anon flapping (attacker / unprovisioned hammering) still quarantines.
Tests: unpaired flapper still quarantined; paired never quarantined (soft cooldown only);
paired creds RELEASE an in-flight quarantine; N paired devices from one SNAT IP all admitted
on reconnect and never quarantined across repeated flush cycles.
Live error: "Uncaught (in promise) TypeError: PlayerMediaHealth.shouldShowIdle is not a
function" inside the device:paired socket handler.
WHAT WAS ACTUALLY WRONG: not a missing/misnamed definition — the module DOES export
shouldShowIdle (served + unit-tested). It's a VERSION SKEW: /player/* is served network-first
by the service worker, so a transient module-fetch failure falls back to the STALE cache (an
older player-media-health.js that predates shouldShowIdle) while index.html loads fresh with
the call. window.PlayerMediaHealth then exists but lacks the method, and the call site
guarded the OBJECT (`window.PlayerMediaHealth ? ...`) not the METHOD — so it threw, aborting
the rest of the device:paired handler (the showStatus after it was skipped).
FIX: guard the METHOD at both call sites (typeof X.method === 'function') so a stale/partial
module can never throw — it falls back to the safe inline default (!isPlaying for the idle
decision) and the handler runs to completion.
SIBLING (errors travel in pairs): the needsReattach call in the "Playlist unchanged" branch
had the SAME object-not-method guard. It was inside a try/catch so it couldn't throw uncaught,
but a stale module would silently skip the re-attach/hideStatus. Guarded it the same way.
No service-worker change needed: network-first already self-heals on the next good load; the
method-guard covers the transient/offline-fallback skew permanently.
Tests: player-media-health.test.js +1 module-surface test (both needsReattach and
shouldShowIdle are exported functions — catches the define-vs-call class). Inline player JS
syntax-checked; both guards verified present. Suite 319/319.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONFIRMED from a live console capture (restore cache -> video plays -> reconnect ->
"Playlist unchanged" -> screen falls to "Waiting for content..."). Audio survived because
only the "showing content" VIEW was covered, not the audio path.
ROOT CAUSE: the server re-emits device:paired on EVERY re-register of an already-paired
device (ws/deviceSocket.js:510) — i.e. on every reconnect, while content is already playing.
The player's device:paired handler called showStatus('Waiting for content...') UNCONDITIONALLY
(the "falls through to idle" sibling), putting the idle overlay OVER the live video. The
following device:playlist-update -> "Playlist unchanged" branch returned early and never
cleared it, so the idle screen stuck on top of playing content.
FIX (idle screen only when genuinely idle; unchanged is a strict no-op that keeps playback):
- lib/player-media-health.js: new shouldShowIdle(state) — idle ONLY when nothing is playing
AND there's genuinely no content. Already-playing (or content-present-about-to-render) is
never idle.
- device:paired handler: gate showStatus on shouldShowIdle({isPlaying, hasContent}) instead
of showing it unconditionally. On a reconnect while playing -> no-op.
- "Playlist unchanged" branch: when healthy playback is confirmed, hideStatus() to clear any
stale idle overlay a reconnect's device:paired may have put up — so the confirmation can
never leave "Waiting for content..." over live content. Still leaves the actual media
element exactly as-is (no teardown, no flicker).
- sw.js cache v10 -> v11.
SIBLING SCAN: device:paired was the only unconditional idle reset. The connect() idle
prompts (connecting / connecting_muted) were already guarded by !isPlaying; empty-playlist
and no-renderable idles are genuine.
Tests: player-media-health.test.js +2 (shouldShowIdle: playing never idle; idle only when
empty+not-playing). Inline player JS syntax-checked; module served + guard referenced on a
booted server. Suite 318/318.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT CAUSE (hypothesis A, pre-existing — NOT a beta7 regression; server/player/index.html
is untouched since v1.9.2-beta6): handlePlaylistUpdate's "Playlist unchanged" branch blindly
returned. The media re-attach (renderContent) lives ONLY in the content-changed branch, so
if the <video> surface was lost (element detached from the DOM while still decoding — video
gone, audio still playing) a no-new-content refresh never re-attached it. New-content
refreshes were fine because they re-render.
FIX (make the refresh idempotent for the media surface, no flicker on the healthy path):
- server/lib/player-media-health.js (new, UMD + unit-testable, mirrors schedule-eval.js):
needsReattach(state) — re-attach ONLY when playback should be happening but the current
item's surface is actually lost (video null / detached / ended / errored; non-video: no
mounted surface). A healthy attached+live video returns false, so a routine poll stays a
no-op (no re-render, no flicker). Served at /player/player-media-health.js from the single
source; loaded by the player.
- index.html no-change branch: extract the current item's DOM facts and, iff
PlayerMediaHealth.needsReattach, call playCurrentItem() to re-render the current item.
Wrapped so the health check can never break a refresh.
- teardownCurrentMedia: also release currentVideoEl even when it was DETACHED from the
container — a detached-but-playing <video> keeps emitting audio and the container-scoped
querySelectorAll can't find it. This kills the "ghost audio" on re-attach.
- sw.js cache bumped v9 -> v10 so players pick up the new index.html + module.
Tests: test/player-media-health.test.js (6) exercises the branch selection — healthy video
-> no re-attach; detached/null/ended/errored -> re-attach; idle -> never; non-video by
surface presence. Inline player JS syntax-checked; module served + referenced verified on a
booted server. Suite 316/316.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The billing:read scope + dual-path gate were built but there was no way to MINT a token
(and it must NOT go in the workspace-scoped, self-service API-Tokens UI). Adds a server-side,
owner-only CLI — no new UI, no network endpoint. Owner-only BY CONSTRUCTION: it's a
host-side script, so filesystem/shell access = the platform owner.
- server/lib/billing-token.js (testable): mintBillingToken/revokeBillingToken/
listBillingTokens. Reuses the EXACT existing token path — same secret (st_ + 32 bytes
base64url), same SHA-256 hashing (hashToken), same api_tokens columns — no second format.
Resolves the platform OWNER (oldest platform_admin/superadmin; #14 collapsed superadmin ->
platform_admin so that's the top tier) and binds to their workspace. api_tokens.user_id +
workspace_id are BOTH NOT NULL (no platform-level token exists); the workspace binding is
VESTIGIAL for billing (billing:read is off-ladder -> can't reach any workspace router;
billing is platform-global), documented in-file rather than loosening NOT NULL pre-release.
- scripts/mint-billing-token.js: thin CLI wrapper. --name mints and prints the secret ONCE
(+ id, + "run as owner on host" warning), --list, --revoke <id> (soft revoke, mirrors the
dashboard DELETE).
Tests (4, test/billing-token-mint.test.js): minted row is scope EXACTLY billing:read with a
matching SHA-256 hash and no read/write/full/agency scope; the token reads GET
/api/billing/usage (200) but is refused on /api/devices (403) and /api/admin (401) — scope
isolation; revocation -> 401; mint requires a name; revoke refuses a non-billing id. CLI
smoked live (mint/list/revoke). Suite 310/310.
SPEC-vs-REALITY (again): spec said bcrypt + JSON `scopes`; this codebase uses SHA-256 + a
single `scope` TEXT column. Built to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Least-privilege way to read GET /api/billing/usage without requiring platform admin.
Additive + isolated: reuses the existing api_tokens scope system (the off-ladder 'agency'
scope is the precedent) and does NOT touch the shared role/permission checks other
endpoints rely on.
- New off-ladder scope 'billing:read' (routes/tokens.js SCOPES). Like 'agency' it is NOT
on the read<write<full ladder, so tokenScopeGate rejects a billing token on every
PUBLIC_ROUTER and JWT-only routers reject any st_ token -> the scope grants billing-read
and NOTHING else.
- DUAL-PATH gate requireBillingRead (middleware/apiToken.js), written as an EXPLICIT OR:
authorize if (billing:read token) OR (platform-admin session). Admins keep read access
but are NOT required to; the token path doesn't lock out admins or vice versa. Billing
route now mounted with bearerAuth (token OR JWT front door) + requireBillingRead (was
requireAuth + requirePlatformAdmin).
- MINTING is platform-admin only (stricter than read/write/full/agency, which any
workspace member may mint) since a billing:read token grants GLOBAL billing-read. Note:
no finer "owner" tier exists here (#14 collapsed superadmin->platform_admin), so
PLATFORM_ROLES is the top level required.
Tests (5, test/billing-authz.test.js): dual-path positive (token AND admin session both
200) + negative (user 403 / anon 401); scope isolation (billing token 403 on /api/devices,
401 on /api/admin; read token 200 on devices but 403 on billing); minting owner-only
(user + ordinary-admin 403, platform-admin 201); revocation -> 401. Existing token
firewall/partition suite (api.test.js) + billing-endpoint tests unchanged & green. Reused
the exact SHA-256 token-verification path (no bcrypt/new mechanism). Suite 306/306.
NOTE: spec described bcrypt + JSON `scopes` + an analytics:read precedent; this codebase
actually uses SHA-256 + a single `scope` TEXT column + 'agency' as the off-ladder
precedent. Implemented faithfully to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updated the item-B section for the in-memory time-limited auto-quarantine (no DB block,
auto-clears) and the item-C section for band-aware downloads (serve freely when healthy,
caps only under load). Both reference the new /api/status debug observability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The storm harness asserted a tick COUNT (environment-timing-sensitive) that could flake
in CI. Kept the real invariant (max event-loop gap < 300ms) and loosened the tick
assertion to >=2 (enough to have measured a real gap), which cannot flake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the new internal states so we can SEE the limiters biting during the alpha soak
instead of grepping logs. /api/status now carries debug: {
flap: {buckets, quarantined},
ota_download: {inFlight, servedThisWindow, shedThisWindow, windowCount},
maintenance: {deleted, ms, at, running}, // last status-log prune
log_coalescer_buffer,
}. Aggregate counts only (no device ids/secrets), cheap in-memory reads. stats()
added to flap-limiter + ota-download-guard (singleton prod state), getMaintenanceStats
from database. Asserted in the booted /api/status test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coalesced loop-lag summary carried an arbitrary sample's p99. Now record() tracks
the MAX (peak) over the window and the summary emits it — the peak is the number that
matters during an incident: '[loop-lag] band=critical (x47 in 30s, peak 1502ms)'. Band
CHANGES still log immediately, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s
while the table is still large (chunked startup prune trickling in the background), and
the backlog drains to the cap with the server responsive throughout. The old whole-table
sort froze boot ~40s. Fallout doc gets the P2 findings section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigated the fleet-wide simultaneous reconnect concern. buildPlaylistPayload is
synchronous but cheap: measured avg 0.078ms / max 0.70ms per call with a 200-item
snapshot; 230 devices = ~18ms CPU spread across 230 separate handler invocations (the
loop yields between them), never one block. Per-call << 50ms invariant. No change
needed; documented with the measured number.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block is a real lever now, so cover the authz path. Booted server + JWT; device and a
viewer membership seeded via the DB file. Proves POST /api/devices/:id/{block,unblock}:
owner (workspace_admin) -> 200; unauthenticated -> 401; cross-workspace user -> 403;
workspace_viewer -> 403 (read-only), with the DB blocked column asserted at each step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fixed OTA_DOWNLOAD_MAX_CONCURRENT=10 throttled a legitimate coordinated rollout even
on a perfectly healthy server, and a shed 503 costs a client a full ~30-min re-check
cycle. Made the download guard's concurrency + rate caps band-aware:
- normal -> serve FREELY (no cap): a whole-fleet rollout isn't staggered when healthy
- elevated -> the configured caps engage (early backpressure)
- critical -> shed 503 (the real protection, unchanged)
Kill switch OTA_DOWNLOAD_GUARD_ENABLED=false disables it entirely.
Tests updated: normal serves 50/50 with 0 shed; caps + shed now asserted under elevated;
storm harness OTA flood runs under elevated (the loaded state). Suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolveIdentity runs on every register (block + flap gates). It already returned on
device_id before any DB access; memoized the device_fingerprints statement (prepared
once, lazily) and documented the invariant. Test asserts a device_id-present resolve
prepares/runs ZERO device_fingerprints queries; a device_id-absent resolve does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flap limiter's auto-quarantine used to run `UPDATE devices SET blocked = 1` — a
PERMANENT, human-cleared block on an automatic trigger. A stuck-then-recovered device
stayed dark until someone noticed.
- Removed the auto-write from ws/deviceSocket.js. devices.blocked is now written ONLY by
an operator (dashboard endpoint / direct SQLite).
- Quarantine moved into lib/flap-limiter.js as IN-MEMORY, TIME-LIMITED state: after
connectRateQuarantineTrips trips in a window the identity is quarantinedUntil = now +
connectRateQuarantineMs (new, default 30m); check() then refuses cheaply with
reason:'quarantined' and AUTO-CLEARS when the window passes. Safe in-memory now that
Item A ended the restart loop, and a self-healing auto-action must not survive as a DB row.
- Log quarantine START once; repeat refusals go through the coalescer. Stale
"-> blocked=1" comments updated.
- connectRateQuarantineTrips=0 still disables it.
Tests: quarantine engages after N trips, refuses cheaply during the window, auto-clears
after connectRateQuarantineMs; and an integration flapper is quarantined while
devices.blocked stays 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deliverables 3 & 4: per-item blast radius + soak signals, and the before/after
worst-case synchronous blocking for every hot path touched (prune, sweeps, OTA,
register, telemetry). Ends with the A<->B interlock note (ship together).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Combines all four spiral inputs at once: a pre-bloated device_status_log (300k rows), a
maintenance sweep running over it (the old whole-table sort froze 40-48s), a hard
flapper, and an OTA download flood from a single SNAT IP. Asserts:
- the event loop never enters a multi-second freeze (max 10ms-ticker gap < 300ms,
vs 40-48s pre-fix) and the ticker keeps firing throughout,
- the chunked sweep drains the 300k backlog to the per-device cap,
- every limiter still bites under load: the flapper is refused, the OTA flood is shed
past the global window cap (served capped, never per-IP).
Suite 267/267.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Don't let telemetry/logging cook the loop under a storm.
- lib/log-coalescer.js: dedup+count high-frequency lines, flush ONE summarized line per
key per window ("[loop-lag] band=critical (x47 in 30s)"). Bounded buffer (auto-flush
at MAX_KEYS). Applied to the loop-lag "still loaded" line (band CHANGES stay immediate),
the per-request OTA check line, and "Device reconnected".
- loop-lag: event_loop_lag rows are BUFFERED and batch-inserted on a flush interval
(was a synchronous INSERT per sample); the buffer is bounded (drop-oldest). Its
retention prune now rides the Item-A chunkedDelete so this table can never repeat the
status_log bloat-then-freeze. /api/status still reads in-memory current (real-time
band unaffected).
- Bounded the previously un-evicted per-device Maps: content-ack limiter gets an idle
sweep (started in server.js); status-log-writer.lastWritten is capped (drop-oldest;
it only suppresses a redundant consecutive row, so eviction is safe).
Tests: N identical lines -> one counted line; single line verbatim; coalescer buffer
bounded under a distinct-key flood; content-ack Map swept of idle buckets.
loop-lag-integration updated for the batched-insert cadence. Suite 266/266.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Enforcement (deviceSocket): resolve identity ONCE via the SNAT-safe chain and check
blocked against the RESOLVED device_id (device_id directly OR fingerprint->device_id),
so a blocked device that reconnects WITHOUT a device_id is still caught — the old
"if (device_id)" gate let a device_id-less reconnect slip past. Still the first gate,
before flap/throttle/DB/playlist. Nulling the token still does NOT block (it
re-provisions) — the blocked column is the lever.
- Dashboard toggle: POST /api/devices/:id/{block,unblock} (write-gated + workspace-scoped
via checkDeviceOwnership) writes devices.blocked; takes effect on the device's NEXT
register with no restart. api.js + a Block/Unblock button in device-detail.js.
- Outage procedure documented in-code: direct SQLite
"UPDATE devices SET blocked = 1 WHERE id = <id>" works with the dashboard down.
Tests: blocked refused at handshake with no playlist build; device_id-less reconnect
with a mapped fingerprint still refused; unblock effective on next register, no restart.
Suite 262/262.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fleet SNATs to one IP, so nothing on the OTA path may key on IP.
- /api/update/check: EARLY-RETURN before any filesystem call when the breaker won't
offer (rate-backoff / up-to-date / phantom / client-newer). A looping client that
gets rate-backoff now does ZERO fs — the flood can't become a statSync flood.
- lib/apk-cache.js: resolve APK path/size/mtime once at boot + refresh on an interval;
the check/download endpoints read cached metadata (get() does no fs, proven by test).
- lib/ota-download-guard.js + /download/apk: GLOBAL concurrency + rate caps + critical-
band shed (503 Retry-After), NEVER per-IP. Replaces the per-IP-per-10min log throttle
(which hid the flood under SNAT) with a per-window served/shed aggregate so a download
flood is VISIBLE. Bounded single rolling-state object; in-flight released on finish/close.
- Breaker unchanged; no IP limiting or device_id requirement added (legacy field clients
send no device_id on OTA checks — must keep working).
Tests: apk-cache get() = 0 statSync over 1000 reads; download guard sheds past global
concurrency + per-window rate + critical band; admit() has no IP parameter. Suite 259/259.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #142 burst throttle (5/10s) misses a device flapping every 3-5s (~2-3/10s) — yet
each cycle is an expensive register+build+acks and one status_log row (the spiral
trigger). Now that Item A ends the restart loop that used to wipe in-memory throttle
state every ~40s, an in-memory sustained limiter can finally bite.
- lib/device-identity.js: SNAT-safe identity resolution — device_id -> fingerprint
(map via device_fingerprints -> device_id, else raw fp) -> device_token -> ONE bounded
global anon bucket. NEVER IP (the fleet SNATs to 10.10.10.1). An unidentifiable client
is still bucketed (collectively) so an anon flood is capped, never unthrottled.
- lib/flap-limiter.js: per-identity connect-frequency over a long window
(CONNECT_RATE_WINDOW_MS=5min, CONNECT_RATE_MAX=20; anon bucket cap 60). Over the rate
-> refuse + disconnect (cheap). Bounded by an idle sweep (anon bucket never swept).
Optional auto-quarantine: a device_id-resolved hard flapper -> blocked=1.
- Wired at the device:register gate BEFORE fingerprint tracking/throttle/DB/build,
skipping same-socket playlist refreshes. Sweep started in server.js.
Tests: 4s-flapper refused after the window max; 60s-normal never; two device_ids
independent (never IP); device_id-less bucketed by fingerprint; neither id nor
fingerprint capped via global anon; idle sweep preserves the anon bucket. Suite 254/254.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 0 deliverable for the beta7 (alpha-only) hardening pass. Enumerates every
synchronous unbounded op (maintenance sweeps, register hot path, content-ack, OTA
endpoints, loop-lag telemetry, log volume, un-evicted per-device Maps), classifies
each exposed/mitigated with worst-case blocking, confirms the two-mechanism spiral
(whole-table prune freeze <-> restart-loop throttle wipe), and sequences items A-E.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
