* feat(dashboard): version indicator + GHCR update check with admin panel
- Add server/lib/ghcr-check.js: GHCR tag poller (36h cache, semver filter)
- Extend /api/version with latest_version and update_available
- Add POST /api/admin/check-update (force GHCR poll)
- Add POST /api/admin/trigger-update (Docker compose or manual instructions)
- Sidebar footer: version label + amber badge when update available
- Admin > System: version comparison card with Check/Update buttons
- 14 new tests (10 unit + 4 integration), 68/68 passing
Closes#163
* fix(dashboard): gate trigger-update to platform-admin + add GHCR fetch timeout
Review follow-up on #165 (the two blockers):
- trigger-update runs `docker compose up -d` on the HOST via docker.sock
(root-equivalent) but was behind requireAdmin, i.e. reachable by any
workspace-level admin. On a multi-tenant host that's a customer, not the infra
operator. Gate it with requirePlatformAdmin (DOCKER_UPDATE_ENABLED still gates
it further). check-update stays requireAdmin — it's a read-only GHCR poll.
- ghcr-check.checkNow had no fetch timeout. Node's global fetch has no default
timeout, so a hung GHCR connection never settled — leaving `inFlight` set
forever (the finally never ran), which wedged the background poller AND hung
any awaited checkNow (/api/admin/check-update). Add a 10s AbortController
timeout on both requests so the try/catch/finally always fire.
All 405 server tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ScreenTinker <hello@screentinker.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets an operator (or an MDM) own updates instead of the app self-installing, which
on managed panels shows a self-install confirm dialog over customer content
(#155). Three layered controls:
- GLOBAL (server): config.otaEnabled from OTA_ENABLED (default on). When off,
/api/update/check returns update_available:false, reason:ota_disabled_global —
the whole instance stops offering updates.
- PER-DEVICE (server + dashboard): new devices.ota_enabled column (default 1). When
0, that device is never offered an update (reason:ota_disabled_device). A
"Self-update (OTA)" toggle in the device settings flips it via PUT /api/devices/:id.
- AUTO-DETECT (Android): UpdateChecker stands down entirely when a foreign device
owner (an MDM/DPC) manages the panel — detected via getActiveAdmins() + not being
device owner ourselves. Pure client-side, errs safe, needs no server change.
The two server gates are enforced server-side so they cover EVERY client version,
not just ones with the client-side stand-down. When OTA is off the device still
reports its version (dashboard sees state); the MDM/operator owns the actual update.
For an MDM-managed fleet (e.g. Pivot/MAXHUB), turn OTA off and let the MDM push the
APK — the install-dialog race disappears from every angle.
Tests: +2 (per-device gate + a real OTA_ENABLED=false server for the global gate);
full server suite 393 pass; Android compiles.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v4-exit-signal-phase3 tests eval a hardcoded LINE RANGE out of tizen/js/app.js
(harness(TIZEN, 663, 697)). The #162 stage-owner fix inserted ~19 lines above that
block, so the slice no longer captured the crash/pagehide handlers and the three B/wgt
tests failed. Re-point the range to 682-716 (block content unchanged, verified identical).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /:id built the items array but never called schedulesForItem, so the playlist
editor rendered "always plays" for items that have a live schedule. Because the
editor re-PUTs whatever it loaded and PUT .../schedules is a wholesale
DELETE+INSERT, an unchanged save on a mis-loaded item silently wiped the real
schedule. Mirror GET /:id/items:351 so the read path returns the blocks the
editor and player already agree on.
Adds render / round-trip / wipe-trap regression tests (subprocess HTTP harness).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The -patchN scheme (e.g. 1.9.2-patch3) parses as a semver prerelease, so decide()'s
superseded-prerelease guard refused to offer a newer stable core (1.9.3) to the existing fleet —
stranding every 1.9.2-patchN device on OTA (Force Update didn't help; the re-check re-returned
superseded-prerelease). isReleased() now counts -patchN as a shipped release, so those devices get
offered 1.9.3 via normal OTA, while GENUINE prereleases (-beta/-rc/-alpha) keep prerelease semantics
and newer cores are never downgraded. 6 new tests + 14 existing OTA tests green (388/388 suite).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI flaked on the 300k-row prune non-blocking assert: a healthy chunked prune hit a 417ms max
event-loop gap on a shared runner, over the strict 250ms bar (the same test passed on the prior
commit; the release bump changed no logic). These probes exist to catch a MULTI-SECOND freeze (the
pre-fix whole-table sort froze 40-48s) — not to enforce a sub-300ms latency SLA — so a strict bar is
fragile under runner contention/GC. Widen both to <1500ms: still << "seconds" (catches any real
regression) but robust on CI. No production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Server-side keystone: the server now honors the v4 liveness contract uniformly across the MIXED
fleet (v4 + old pre-v4 + disconnected), all three clients depending on it.
- UNIFORM heartbeat-ack: emitted from the single shared device:heartbeat handler (uniform by
construction; no per-client/per-path branch), BEFORE the auth guard so a known device's watchdog
stays armed. Harmless to old clients (they ignore it).
- RECONNECT-WINDOW ack-gap fix (ackableHeartbeat): ack a KNOWN device (authed socket OR a device_id
that resolves) even mid-reconnect; NOT anonymous/never-authenticated sockets (degrade-safe);
identity-agnostic. No state mutation before requireDeviceAuth (auth surface unchanged; device_ids
are uuidv4).
- DASHBOARD LIVENESS (deriveLiveness): server-derived, VERSION-AGNOSTIC Healthy/Degraded/Offline
from signals every client sends (socket presence, heartbeat age, reconnect frequency); no client
status-push.
- IDENTITY CAPTURE (capture-don't-act): client_type/client_version/platform/contract_version columns;
degrades to legacy/unknown for old clients; NEVER breaks register.
- A-BUCKET FIX (QA): recordReconnect + persistIdentity gated on !isPlaylistRefresh (a ~45-60s refresh
is not a reconnect/new identity — matches #134), and the identity write is change-detected — closing
the WAL write-amplification (A1) and the benign-refresh -> false-"Degraded" (A2) regressions.
New lib/liveness.js (pure helpers, unit-tested). 30 new tests (uniform ack, ack-gap, mixed fleet,
identity capture, cross-client conformance, refresh-gate reproduce-then-prove); 366/366 total.
OTA artifact-availability is a separate concern (out of scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Field-safe SERVER net. A device opening duplicate/rapid sockets (the APK duplicate-socket bug,
separate track) currently churns through evictions during the reconnect-throttle's 30s
post-restart WARM-UP (only the hard ceiling 20 applies then, so an 8-in-9s burst passes
undamped and each new socket evicts the prior). This makes the server absorb it: a thrashing
PAIRED device converges to ONE stable connection and stays online.
- lib/session-settle.js (decision only; bounded, swept): shouldHold(deviceId, incumbentAlive)
— true only when a socket was accepted for this device within SESSION_SETTLE_WINDOW_MS
(config, default 2500ms) AND the incumbent is alive. Warm-up-independent.
- deviceSocket register gate (just before evictPriorSocket): if a LIVE incumbent exists and
we're inside the window, SOFT-REFUSE the new socket (device:throttled reason=session_settle
+ disconnect) and keep the incumbent; else accept + evict + (re)arm the window.
- LIVENESS SAFEGUARD (load-bearing): only hold when the incumbent socket is actually in the
/device namespace — a dead/half-open incumbent is replaced, NEVER stranding the device (max
hold is the 2.5s window from the incumbent's accept, then any new socket is accepted).
- Soft refusal, NEVER a quarantine (reuses patch1's paired-safe philosophy); single-session
enforcement intact for a legitimate move; unpaired/abusive flapping still caught by the
existing limiters. O(1), no loop impact.
Tests (liveness first-class): live incumbent holds + DEAD incumbent replaced (not stranded);
storm of 6 sockets converges to ONE, stays online, not quarantined (during warm-up); single-
session move past the window replaces cleanly; unit decision + bounded sweep. The
evicted-socket-rearm test shrinks its settle window so it still exercises the eviction path.
Suite 336/336.
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
The flap-limiter could 30-min quarantine a PAIRED, legitimate device on reconnect churn.
Behind Bold's single SNAT IP a repeated edge flush -> every device reconnects -> trips flap
-> quarantined -> a recoverable blip becomes a SUSTAINED FLEET-WIDE LOCKOUT we caused.
check(key, now, {paired}) now skips (and clears) the quarantine escalation for a paired
device — it still gets the brief soft cooldown if it truly hammers, but never the long
lockout. The register gate computes paired = device_id && validateDeviceToken(...) (a
matching STORED token, false for missing/mismatch) so a spoofed device_id can't claim the
exemption; unpaired/anon flapping (attacker / unprovisioned hammering) still quarantines.
Tests: unpaired flapper still quarantined; paired never quarantined (soft cooldown only);
paired creds RELEASE an in-flight quarantine; N paired devices from one SNAT IP all admitted
on reconnect and never quarantined across repeated flush cycles.
Live error: "Uncaught (in promise) TypeError: PlayerMediaHealth.shouldShowIdle is not a
function" inside the device:paired socket handler.
WHAT WAS ACTUALLY WRONG: not a missing/misnamed definition — the module DOES export
shouldShowIdle (served + unit-tested). It's a VERSION SKEW: /player/* is served network-first
by the service worker, so a transient module-fetch failure falls back to the STALE cache (an
older player-media-health.js that predates shouldShowIdle) while index.html loads fresh with
the call. window.PlayerMediaHealth then exists but lacks the method, and the call site
guarded the OBJECT (`window.PlayerMediaHealth ? ...`) not the METHOD — so it threw, aborting
the rest of the device:paired handler (the showStatus after it was skipped).
FIX: guard the METHOD at both call sites (typeof X.method === 'function') so a stale/partial
module can never throw — it falls back to the safe inline default (!isPlaying for the idle
decision) and the handler runs to completion.
SIBLING (errors travel in pairs): the needsReattach call in the "Playlist unchanged" branch
had the SAME object-not-method guard. It was inside a try/catch so it couldn't throw uncaught,
but a stale module would silently skip the re-attach/hideStatus. Guarded it the same way.
No service-worker change needed: network-first already self-heals on the next good load; the
method-guard covers the transient/offline-fallback skew permanently.
Tests: player-media-health.test.js +1 module-surface test (both needsReattach and
shouldShowIdle are exported functions — catches the define-vs-call class). Inline player JS
syntax-checked; both guards verified present. Suite 319/319.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONFIRMED from a live console capture (restore cache -> video plays -> reconnect ->
"Playlist unchanged" -> screen falls to "Waiting for content..."). Audio survived because
only the "showing content" VIEW was covered, not the audio path.
ROOT CAUSE: the server re-emits device:paired on EVERY re-register of an already-paired
device (ws/deviceSocket.js:510) — i.e. on every reconnect, while content is already playing.
The player's device:paired handler called showStatus('Waiting for content...') UNCONDITIONALLY
(the "falls through to idle" sibling), putting the idle overlay OVER the live video. The
following device:playlist-update -> "Playlist unchanged" branch returned early and never
cleared it, so the idle screen stuck on top of playing content.
FIX (idle screen only when genuinely idle; unchanged is a strict no-op that keeps playback):
- lib/player-media-health.js: new shouldShowIdle(state) — idle ONLY when nothing is playing
AND there's genuinely no content. Already-playing (or content-present-about-to-render) is
never idle.
- device:paired handler: gate showStatus on shouldShowIdle({isPlaying, hasContent}) instead
of showing it unconditionally. On a reconnect while playing -> no-op.
- "Playlist unchanged" branch: when healthy playback is confirmed, hideStatus() to clear any
stale idle overlay a reconnect's device:paired may have put up — so the confirmation can
never leave "Waiting for content..." over live content. Still leaves the actual media
element exactly as-is (no teardown, no flicker).
- sw.js cache v10 -> v11.
SIBLING SCAN: device:paired was the only unconditional idle reset. The connect() idle
prompts (connecting / connecting_muted) were already guarded by !isPlaying; empty-playlist
and no-renderable idles are genuine.
Tests: player-media-health.test.js +2 (shouldShowIdle: playing never idle; idle only when
empty+not-playing). Inline player JS syntax-checked; module served + guard referenced on a
booted server. Suite 318/318.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT CAUSE (hypothesis A, pre-existing — NOT a beta7 regression; server/player/index.html
is untouched since v1.9.2-beta6): handlePlaylistUpdate's "Playlist unchanged" branch blindly
returned. The media re-attach (renderContent) lives ONLY in the content-changed branch, so
if the <video> surface was lost (element detached from the DOM while still decoding — video
gone, audio still playing) a no-new-content refresh never re-attached it. New-content
refreshes were fine because they re-render.
FIX (make the refresh idempotent for the media surface, no flicker on the healthy path):
- server/lib/player-media-health.js (new, UMD + unit-testable, mirrors schedule-eval.js):
needsReattach(state) — re-attach ONLY when playback should be happening but the current
item's surface is actually lost (video null / detached / ended / errored; non-video: no
mounted surface). A healthy attached+live video returns false, so a routine poll stays a
no-op (no re-render, no flicker). Served at /player/player-media-health.js from the single
source; loaded by the player.
- index.html no-change branch: extract the current item's DOM facts and, iff
PlayerMediaHealth.needsReattach, call playCurrentItem() to re-render the current item.
Wrapped so the health check can never break a refresh.
- teardownCurrentMedia: also release currentVideoEl even when it was DETACHED from the
container — a detached-but-playing <video> keeps emitting audio and the container-scoped
querySelectorAll can't find it. This kills the "ghost audio" on re-attach.
- sw.js cache bumped v9 -> v10 so players pick up the new index.html + module.
Tests: test/player-media-health.test.js (6) exercises the branch selection — healthy video
-> no re-attach; detached/null/ended/errored -> re-attach; idle -> never; non-video by
surface presence. Inline player JS syntax-checked; module served + referenced verified on a
booted server. Suite 316/316.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The billing:read scope + dual-path gate were built but there was no way to MINT a token
(and it must NOT go in the workspace-scoped, self-service API-Tokens UI). Adds a server-side,
owner-only CLI — no new UI, no network endpoint. Owner-only BY CONSTRUCTION: it's a
host-side script, so filesystem/shell access = the platform owner.
- server/lib/billing-token.js (testable): mintBillingToken/revokeBillingToken/
listBillingTokens. Reuses the EXACT existing token path — same secret (st_ + 32 bytes
base64url), same SHA-256 hashing (hashToken), same api_tokens columns — no second format.
Resolves the platform OWNER (oldest platform_admin/superadmin; #14 collapsed superadmin ->
platform_admin so that's the top tier) and binds to their workspace. api_tokens.user_id +
workspace_id are BOTH NOT NULL (no platform-level token exists); the workspace binding is
VESTIGIAL for billing (billing:read is off-ladder -> can't reach any workspace router;
billing is platform-global), documented in-file rather than loosening NOT NULL pre-release.
- scripts/mint-billing-token.js: thin CLI wrapper. --name mints and prints the secret ONCE
(+ id, + "run as owner on host" warning), --list, --revoke <id> (soft revoke, mirrors the
dashboard DELETE).
Tests (4, test/billing-token-mint.test.js): minted row is scope EXACTLY billing:read with a
matching SHA-256 hash and no read/write/full/agency scope; the token reads GET
/api/billing/usage (200) but is refused on /api/devices (403) and /api/admin (401) — scope
isolation; revocation -> 401; mint requires a name; revoke refuses a non-billing id. CLI
smoked live (mint/list/revoke). Suite 310/310.
SPEC-vs-REALITY (again): spec said bcrypt + JSON `scopes`; this codebase uses SHA-256 + a
single `scope` TEXT column. Built to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Least-privilege way to read GET /api/billing/usage without requiring platform admin.
Additive + isolated: reuses the existing api_tokens scope system (the off-ladder 'agency'
scope is the precedent) and does NOT touch the shared role/permission checks other
endpoints rely on.
- New off-ladder scope 'billing:read' (routes/tokens.js SCOPES). Like 'agency' it is NOT
on the read<write<full ladder, so tokenScopeGate rejects a billing token on every
PUBLIC_ROUTER and JWT-only routers reject any st_ token -> the scope grants billing-read
and NOTHING else.
- DUAL-PATH gate requireBillingRead (middleware/apiToken.js), written as an EXPLICIT OR:
authorize if (billing:read token) OR (platform-admin session). Admins keep read access
but are NOT required to; the token path doesn't lock out admins or vice versa. Billing
route now mounted with bearerAuth (token OR JWT front door) + requireBillingRead (was
requireAuth + requirePlatformAdmin).
- MINTING is platform-admin only (stricter than read/write/full/agency, which any
workspace member may mint) since a billing:read token grants GLOBAL billing-read. Note:
no finer "owner" tier exists here (#14 collapsed superadmin->platform_admin), so
PLATFORM_ROLES is the top level required.
Tests (5, test/billing-authz.test.js): dual-path positive (token AND admin session both
200) + negative (user 403 / anon 401); scope isolation (billing token 403 on /api/devices,
401 on /api/admin; read token 200 on devices but 403 on billing); minting owner-only
(user + ordinary-admin 403, platform-admin 201); revocation -> 401. Existing token
firewall/partition suite (api.test.js) + billing-endpoint tests unchanged & green. Reused
the exact SHA-256 token-verification path (no bcrypt/new mechanism). Suite 306/306.
NOTE: spec described bcrypt + JSON `scopes` + an analytics:read precedent; this codebase
actually uses SHA-256 + a single `scope` TEXT column + 'agency' as the off-ladder
precedent. Implemented faithfully to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The storm harness asserted a tick COUNT (environment-timing-sensitive) that could flake
in CI. Kept the real invariant (max event-loop gap < 300ms) and loosened the tick
assertion to >=2 (enough to have measured a real gap), which cannot flake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the new internal states so we can SEE the limiters biting during the alpha soak
instead of grepping logs. /api/status now carries debug: {
flap: {buckets, quarantined},
ota_download: {inFlight, servedThisWindow, shedThisWindow, windowCount},
maintenance: {deleted, ms, at, running}, // last status-log prune
log_coalescer_buffer,
}. Aggregate counts only (no device ids/secrets), cheap in-memory reads. stats()
added to flap-limiter + ota-download-guard (singleton prod state), getMaintenanceStats
from database. Asserted in the booted /api/status test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coalesced loop-lag summary carried an arbitrary sample's p99. Now record() tracks
the MAX (peak) over the window and the summary emits it — the peak is the number that
matters during an incident: '[loop-lag] band=critical (x47 in 30s, peak 1502ms)'. Band
CHANGES still log immediately, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s
while the table is still large (chunked startup prune trickling in the background), and
the backlog drains to the cap with the server responsive throughout. The old whole-table
sort froze boot ~40s. Fallout doc gets the P2 findings section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigated the fleet-wide simultaneous reconnect concern. buildPlaylistPayload is
synchronous but cheap: measured avg 0.078ms / max 0.70ms per call with a 200-item
snapshot; 230 devices = ~18ms CPU spread across 230 separate handler invocations (the
loop yields between them), never one block. Per-call << 50ms invariant. No change
needed; documented with the measured number.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block is a real lever now, so cover the authz path. Booted server + JWT; device and a
viewer membership seeded via the DB file. Proves POST /api/devices/:id/{block,unblock}:
owner (workspace_admin) -> 200; unauthenticated -> 401; cross-workspace user -> 403;
workspace_viewer -> 403 (read-only), with the DB blocked column asserted at each step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fixed OTA_DOWNLOAD_MAX_CONCURRENT=10 throttled a legitimate coordinated rollout even
on a perfectly healthy server, and a shed 503 costs a client a full ~30-min re-check
cycle. Made the download guard's concurrency + rate caps band-aware:
- normal -> serve FREELY (no cap): a whole-fleet rollout isn't staggered when healthy
- elevated -> the configured caps engage (early backpressure)
- critical -> shed 503 (the real protection, unchanged)
Kill switch OTA_DOWNLOAD_GUARD_ENABLED=false disables it entirely.
Tests updated: normal serves 50/50 with 0 shed; caps + shed now asserted under elevated;
storm harness OTA flood runs under elevated (the loaded state). Suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolveIdentity runs on every register (block + flap gates). It already returned on
device_id before any DB access; memoized the device_fingerprints statement (prepared
once, lazily) and documented the invariant. Test asserts a device_id-present resolve
prepares/runs ZERO device_fingerprints queries; a device_id-absent resolve does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flap limiter's auto-quarantine used to run `UPDATE devices SET blocked = 1` — a
PERMANENT, human-cleared block on an automatic trigger. A stuck-then-recovered device
stayed dark until someone noticed.
- Removed the auto-write from ws/deviceSocket.js. devices.blocked is now written ONLY by
an operator (dashboard endpoint / direct SQLite).
- Quarantine moved into lib/flap-limiter.js as IN-MEMORY, TIME-LIMITED state: after
connectRateQuarantineTrips trips in a window the identity is quarantinedUntil = now +
connectRateQuarantineMs (new, default 30m); check() then refuses cheaply with
reason:'quarantined' and AUTO-CLEARS when the window passes. Safe in-memory now that
Item A ended the restart loop, and a self-healing auto-action must not survive as a DB row.
- Log quarantine START once; repeat refusals go through the coalescer. Stale
"-> blocked=1" comments updated.
- connectRateQuarantineTrips=0 still disables it.
Tests: quarantine engages after N trips, refuses cheaply during the window, auto-clears
after connectRateQuarantineMs; and an integration flapper is quarantined while
devices.blocked stays 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Combines all four spiral inputs at once: a pre-bloated device_status_log (300k rows), a
maintenance sweep running over it (the old whole-table sort froze 40-48s), a hard
flapper, and an OTA download flood from a single SNAT IP. Asserts:
- the event loop never enters a multi-second freeze (max 10ms-ticker gap < 300ms,
vs 40-48s pre-fix) and the ticker keeps firing throughout,
- the chunked sweep drains the 300k backlog to the per-device cap,
- every limiter still bites under load: the flapper is refused, the OTA flood is shed
past the global window cap (served capped, never per-IP).
Suite 267/267.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Don't let telemetry/logging cook the loop under a storm.
- lib/log-coalescer.js: dedup+count high-frequency lines, flush ONE summarized line per
key per window ("[loop-lag] band=critical (x47 in 30s)"). Bounded buffer (auto-flush
at MAX_KEYS). Applied to the loop-lag "still loaded" line (band CHANGES stay immediate),
the per-request OTA check line, and "Device reconnected".
- loop-lag: event_loop_lag rows are BUFFERED and batch-inserted on a flush interval
(was a synchronous INSERT per sample); the buffer is bounded (drop-oldest). Its
retention prune now rides the Item-A chunkedDelete so this table can never repeat the
status_log bloat-then-freeze. /api/status still reads in-memory current (real-time
band unaffected).
- Bounded the previously un-evicted per-device Maps: content-ack limiter gets an idle
sweep (started in server.js); status-log-writer.lastWritten is capped (drop-oldest;
it only suppresses a redundant consecutive row, so eviction is safe).
Tests: N identical lines -> one counted line; single line verbatim; coalescer buffer
bounded under a distinct-key flood; content-ack Map swept of idle buckets.
loop-lag-integration updated for the batched-insert cadence. Suite 266/266.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Enforcement (deviceSocket): resolve identity ONCE via the SNAT-safe chain and check
blocked against the RESOLVED device_id (device_id directly OR fingerprint->device_id),
so a blocked device that reconnects WITHOUT a device_id is still caught — the old
"if (device_id)" gate let a device_id-less reconnect slip past. Still the first gate,
before flap/throttle/DB/playlist. Nulling the token still does NOT block (it
re-provisions) — the blocked column is the lever.
- Dashboard toggle: POST /api/devices/:id/{block,unblock} (write-gated + workspace-scoped
via checkDeviceOwnership) writes devices.blocked; takes effect on the device's NEXT
register with no restart. api.js + a Block/Unblock button in device-detail.js.
- Outage procedure documented in-code: direct SQLite
"UPDATE devices SET blocked = 1 WHERE id = <id>" works with the dashboard down.
Tests: blocked refused at handshake with no playlist build; device_id-less reconnect
with a mapped fingerprint still refused; unblock effective on next register, no restart.
Suite 262/262.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fleet SNATs to one IP, so nothing on the OTA path may key on IP.
- /api/update/check: EARLY-RETURN before any filesystem call when the breaker won't
offer (rate-backoff / up-to-date / phantom / client-newer). A looping client that
gets rate-backoff now does ZERO fs — the flood can't become a statSync flood.
- lib/apk-cache.js: resolve APK path/size/mtime once at boot + refresh on an interval;
the check/download endpoints read cached metadata (get() does no fs, proven by test).
- lib/ota-download-guard.js + /download/apk: GLOBAL concurrency + rate caps + critical-
band shed (503 Retry-After), NEVER per-IP. Replaces the per-IP-per-10min log throttle
(which hid the flood under SNAT) with a per-window served/shed aggregate so a download
flood is VISIBLE. Bounded single rolling-state object; in-flight released on finish/close.
- Breaker unchanged; no IP limiting or device_id requirement added (legacy field clients
send no device_id on OTA checks — must keep working).
Tests: apk-cache get() = 0 statSync over 1000 reads; download guard sheds past global
concurrency + per-window rate + critical band; admit() has no IP parameter. Suite 259/259.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #142 burst throttle (5/10s) misses a device flapping every 3-5s (~2-3/10s) — yet
each cycle is an expensive register+build+acks and one status_log row (the spiral
trigger). Now that Item A ends the restart loop that used to wipe in-memory throttle
state every ~40s, an in-memory sustained limiter can finally bite.
- lib/device-identity.js: SNAT-safe identity resolution — device_id -> fingerprint
(map via device_fingerprints -> device_id, else raw fp) -> device_token -> ONE bounded
global anon bucket. NEVER IP (the fleet SNATs to 10.10.10.1). An unidentifiable client
is still bucketed (collectively) so an anon flood is capped, never unthrottled.
- lib/flap-limiter.js: per-identity connect-frequency over a long window
(CONNECT_RATE_WINDOW_MS=5min, CONNECT_RATE_MAX=20; anon bucket cap 60). Over the rate
-> refuse + disconnect (cheap). Bounded by an idle sweep (anon bucket never swept).
Optional auto-quarantine: a device_id-resolved hard flapper -> blocked=1.
- Wired at the device:register gate BEFORE fingerprint tracking/throttle/DB/build,
skipping same-socket playlist refreshes. Sweep started in server.js.
Tests: 4s-flapper refused after the window max; 60s-normal never; two device_ids
independent (never IP); device_id-less bucketed by fingerprint; neither id nor
fingerprint capped via global anon; idle sweep preserves the anon bucket. Suite 254/254.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found in the alpha load test: client-chosen pairing codes collide by birthday
paradox, the provisioning INSERT hit UNIQUE(devices.pairing_code), the SqliteError
threw out of the (synchronous) socket handler -> uncaughtException -> logFatalAndExit
-> the WHOLE server exited and every device dropped. The colliding flood crash-LOOPED
the container (2 restarts).
Two layers, same "one device can't take down the fleet" theme as #142/#143/#144:
1. Narrow (deviceSocket.js): wrap the device:register provisioning INSERT in
try/catch — a UNIQUE pairing_code collision (or ANY db error) rejects THAT
registration (device:auth-error -> client retries) instead of throwing.
currentDeviceId/authenticated now set only AFTER the row exists (no half-auth
socket on failure).
2. Broader (lib/safe-socket.js): protectSocket() overrides socket.on per connection
so any handler throw is caught, logged (event + id + stack), the socket told, and
DISCONNECTED — per-CONNECTION fail-fast, not whole-PROCESS. We don't keep serving a
connection from possibly-half-mutated state (honors the existing fail-fast intent),
we just contain it to "one device reconnects" (a non-event after beta5). Wired into
both the /device and /dashboard connection handlers; auto-covers future handlers.
Audited first: no handler throws as control flow, so blanket-wrapping is safe.
Tests (mutation-verified, fail without their fix):
- register-insert-crash.test.js: a pairing_code collision AND a general bind error
each reject-one-device with no uncaughtException; server keeps serving.
- socket-handler-isolation.test.js: a throwing handler disconnects only that socket;
the server + other sockets stay alive.
Full suite 243/243.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/api/update/check offered the update whenever client !== latest (raw string
inequality, not semver) with no backoff. A device that can't APPLY the update
(broken OTA client 1.7.12, signing/Fire OS) keeps reporting the same version and is
told update_available=true on every poll; a fast poll loop saturates the event loop
(prod loop-lag 49s). All requests share one NAT IP, so IP-keying is useless.
server-only breaker (lib/ota-breaker.js), two independent axes:
- RATE breaker (primary, immediate): a key checking >THRESHOLD (3) times within
WINDOW (60s) is looping -> throttle update_available with exponential backoff
(30s->2m->8m->cap 30m). Healthy devices poll ~12 min and never approach this, so
rollout/stragglers are inherently safe -- NO grace-for-flood timer; slow == safe.
- PHANTOM guard (immediate): unrecognized version, or a prerelease of an OLDER core
(superseded old-minor beta e.g. 1.9.1-beta4), gets no-offer on the first check. A
RECENT real older version (beta3 vs latest beta4; stable 1.7.12) stays offerable.
- Never offers a downgrade (client >= latest -> no offer).
KEYING (#144 option 3): keyed on device_id when present, else reported version.
- server.js:581 accepts + logs ?device_id=, passes it to the breaker.
- UpdateChecker.kt:122 appends &device_id=<config.deviceId> (existing registered id;
omitted until provisioned). One-line client change.
beta4+ clients get precise per-device throttling; stuck legacy clients sending only
?version= are caught by the version-keyed + rate + phantom logic. Response gains
additive `reason` + `retry_after_seconds` (old clients ignore).
BOUNDED STATE: a periodic sweep (startSweep, wired in server.js) evicts buckets idle
> IDLE_RESET_MS so the keyed Map can't grow unbounded (churned device_ids); not
reset-on-access only.
SCOPE (deliberate): this targets the FAST flood + phantoms. The slow #144 drip
(stable 1.7.12 polling ~every 12 min, ~20/hr) stays below >3/60s and is NOT
throttled -- catching it needs #144 option-3 "skip-this-version after N cycles",
which is intentionally NOT in this build.
NOTE: carries a CLIENT/APK change -> versionCode must increment at the beta4 bump and
the release keystore is required for the APK. The device_id path only helps devices
that can install beta4+; the stuck legacy fleet is covered by the version-keyed path.
Tests: unit (lib/ota-breaker, injected time) a-f + comparator + escalation + sweep +
slow-drip-scope; HTTP integration (real endpoint, device_id passthrough). Full suite
green serial AND parallel (234). OTA-only delta -- reconnect/reclaim/shed/content-ack/
block untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bold: screens sit on the Connect page showing the server URL = paired server-side
but never told, so the app never starts playing.
Flow / gap (Step A):
- CLIENT leaves the Connect page ONLY on the 'device:paired' event — web player
(player/index.html) hides the setup screen; Android ProvisioningActivity.onPaired
launches MainActivity + finish(). That event is the sole signal.
- SERVER pushes 'device:paired' to the device's room from POST /api/provision/pair
(server.js) at pair time — but ONLY reaches a LIVE socket then. The normal
device_id reconnect path emitted device:registered + device:playlist-update but
NOT device:paired. So a screen paired while disconnected, or that reconnects after
pairing (exactly the screens cycling on the Connect page), is paired server-side
(user_id set, receiving playlists) yet never gets device:paired -> stuck on Connect.
Fix (server-only, uses the EXISTING client listener — no client update needed, which
matters because we can't push a client update to stuck screens): on the device_id
reconnect, if the device is paired (user_id set), re-emit 'device:paired'
{device_id, name}. Push-on-pair (server.js) already covers the live-at-pair-time
case; this covers paired-then-reconnect. A paired screen now leaves Connect and
plays on its next reconnect with no client change and no manual re-pair.
Tests (port 3989, real flow): provision -> pair via /api/provision/pair (socket
closed) -> reconnect RECEIVES device:paired (+name +playlist) — the stuck-screen
repro; an unpaired device gets NO device:paired (stays on the pairing flow); the fix
reuses the existing device:paired event (no new protocol). Full suite green serial
AND parallel (220); dbac699 / 404c330 / e734281 intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bold beta1: three devices spam "Fingerprint reclaim rejected ... device active
(status=offline, ~2500s since heartbeat, liveConn=false)" twice/~2s indefinitely —
contradictory: gone by every signal yet treated as active.
Root cause (NOT a missing clear — corrected the hypothesis). The reject condition
was `liveConn || status==='online' || secondsSince < RECLAIM_GRACE_SECONDS(24h)`.
For the observed devices liveConn=false and status=offline, so the ONLY true term
is `secondsSince < 24h` — an effective 24h CALENDAR grace, not a stale flag. Audited
the clears: liveConn (deviceConnections) is removed on the debounced disconnect
(heartbeat.removeConnection) AND the offline_timeout sweep (deviceConnections.delete);
status is set 'offline' on both. liveConn=false + status=offline PROVE the clears
ran — there is nothing stale to clear. The 24h time gate (mislabeled "device active")
blocked a legitimately-gone device from reclaiming for up to 24h, so it retried
every ~2s forever-in-practice. The "twice per ~2s" is two reclaim ATTEMPTS per cycle
(client reconnect + re-pair-on-auth-error), each hitting the single console.warn —
not double-logging in one attempt.
Fix:
- Decide "still alive" from RUNTIME signals: `!!liveConn || secondsSince <
reclaimSettleSeconds`. A device with no live socket and a heartbeat older than the
settle window is gone -> reclaimable. A live (or just-seen) device is still
rejected, so reclaim-abuse protection holds. NOT just ignoring "active" — it fixes
WHY it was stuck (the 24h gate). RECLAIM_SETTLE_SECONDS default 300 (was 24h).
SECURITY TRADEOFF flagged in config: shortens the anti-fingerprint-theft window;
raise to re-tighten. Tuning guess to validate vs Bold.
- Log throttle: the deferral logs at most once per device per RECLAIM_REJECT_LOG_
WINDOW_MS (default 60s) — collapses the double-log + the per-2s flood (same
discipline as the content-ack shed log). Cleared when a reclaim proceeds.
Recovery of the 3 wedged devices (2febcaa9, 1984694c, 139159eb): they SELF-HEAL on
their next reclaim attempt (~2s) once this ships — their heartbeats are ~2500s stale
(>300s settle) and liveConn=false, so the reclaim now succeeds. No operator SQL needed.
Tests (port 3988): gone device reclaims; live device still rejected; clear-on-leave
(disconnect clears liveConn -> stale device reclaims); deferral log <=1 per window.
Full suite green serial+parallel (217). reconnect-throttle.js, the dbac699 content-ack
limiter, and the 404c330 block/auth code untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Highest-priority #143 item (operator finding from Bold): nulling a device's token
did NOT lock it out — device 75c2a08a immediately reconnected and saturated the
loop. Two distinct defects:
1. Auth short-circuit (the cause). device:register used
if (device.device_token && !validateDeviceToken(...)) { reject }
so a NULL/empty STORED token made the guard falsy -> validation SKIPPED, and the
next block even MINTED a fresh token and persisted it. Nulling a token thus
RE-PROVISIONED the device instead of locking it out. Fix: drop the
`device.device_token &&` guard -> `if (!validateDeviceToken(device_id, device_token))`
(validateDeviceToken already returns false for null-stored/missing/mismatch), and
remove the legacy "mint a token for a null-token device" path (the re-provision
vector). An already-provisioned device (every row, incl. 'provisioning', is created
WITH a token) presenting null/empty/invalid is now REJECTED + disconnected.
The first-pairing seam is unaffected: a brand-new device has NO device_id and goes
through the pairing_code branch (which mints id+token) — a different code path.
2. No server-side kill switch. Added a `blocked` column (devices.blocked INTEGER
NOT NULL DEFAULT 0; schema.sql + a database.js migration). The block is the FIRST
gate at the top of device:register — before the fingerprint block, the reconnect
throttle, any DB writes, or playlist build — so a blocked device's socket is
refused immediately (auth-error 'Device blocked' + disconnect, zero further work).
It does NOT rely on null-token (the thing that failed). The row is re-read every
register, so a DIRECT SQLite edit takes effect on the device's NEXT reconnect with
NO server restart. Operator statements (dashboard-down, hand-edit):
block: UPDATE devices SET blocked = 1 WHERE id = '<device_id>';
unblock: UPDATE devices SET blocked = 0 WHERE id = '<device_id>';
Tests (port 3987): nulled-token provisioned device is REJECTED (75c2a08a repro);
blocked=1 refused at the first gate (no register/playlist); unblock reconnects;
first-pairing still works; normal valid-token device unaffected. Full suite green
serial AND parallel (213); reconnect-throttle.js + the dbac699 content-ack limiter
untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#142's content-ack dedup is insufficient: a device cycling 2-4 content IDs makes
every ack look unique so dedup never fires, while aggregate volume from ~30 devices
saturates the event loop (the #142 reconnect throttle kept the server responsive,
which is how this was even observable).
Folded ONE control on the content-ack path (no competing limiters; reconnect-
throttle.js untouched) in lib/content-ack-limiter.js:
- Step 1 — per-device RATE budget: caps TOTAL non-duplicate acks per device per
window regardless of differing content_id (the case dedup misses). Over budget =
DROP silently (the per-ack log+emit is the cost); log ONCE per device per window
when shedding starts. Keeps the #142 dedup (dedup'd repeats don't consume budget).
Per-device, in-memory, resets on restart (modeled on lastPlayLogAt; does NOT reuse
reconnect-throttle's ban-semantics bucket).
Env (TUNING GUESSES, validate vs Bold's fleet): CONTENT_ACK_MAX_PER_WINDOW=20,
CONTENT_ACK_RATE_WINDOW_MS=10000 (=2/s, above legit ~<=1/s, below the flood).
- Step 2 — global pressure valve: reuses the #142 loop-lag band (+ its hysteresis,
no second control loop). Under CRITICAL band, shed content-acks even for an
in-budget device; reconnects + dashboard/HTTP are ALWAYS processed; a healthy
device in a non-critical band is never touched by the valve. Valve open/close
logged once at the band edge in services/loop-lag.js (not per shed message).
Tests (unique ports 3985/3986, not the 3982/3983/3984 set):
- unit: the #143 regression (cycling ids evading dedup IS rate-limited), under/over
budget, dedup still works + doesn't consume budget, valve sheds in-budget under
critical while normal is untouched, rate precedence, window reset, per-device
isolation.
- integration: socket flood is capped to budget with a single shed-start log;
under-budget passes every ack; valve OPEN sheds content-acks while a reconnect +
/api/status still succeed.
Full suite green serial AND parallel (208 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
services/heartbeat.js deleted unclaimed provisioning devices with
created_at < now - (365 * 86400) — a YEAR — while its own comment said "older
than 24 hours". So socket-register pairing junk lingered ~365x longer than
intended. Change the window to 24 * 3600 to match the comment.
Correctness fix only — does NOT touch the pre-auth register path or add a rate
limiter (that pre-auth hardening is a separate security issue, out of this cut).
Extracted the sweep into pruneProvisioningDevices() (still in heartbeat.js, called
from the same interval) so it is unit-testable. Test asserts a >24h unclaimed
provisioning row is swept while a <24h row, an imported row (user_id set), and a
non-provisioning row are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
device:content-ack logged + emitted every message, so a device repeatedly
reporting the same "content <id>: ready" (observed from an older app version)
added avoidable load per message.
- Suppress identical (device_id, content_id, status) reports within
config.contentAckDedupMs (default 10s), modeled on the lastPlayLogAt throttle.
A status change has a different key and passes immediately; a fresh report after
the window passes too. In-memory, resets on restart. The handler does no DB
writes, so this is purely shedding redundant log+emit work.
test: integration over a real authenticated device socket — a burst of identical
"ready" collapses to one log/emit, a "ready" after the window passes, and a status
change is never deduped. Unique PORT (3984).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-device insert-time prune (deviceSocket.js) only ever touches a device
that is actively inserting, so it misses two paths: removed/idle devices whose
rows linger forever, and heartbeat.js's offline_timeout insert that bypasses
logDeviceStatus entirely. The reporter's 1.2M-row bloat accumulated UNDER a 7-day
per-device prune for exactly this reason.
- pruneStatusLog() (db/database.js): a GLOBAL time-range sweep across ALL devices,
modeled on the play_logs prune. Run once on startup (recovers a bloated table
right after deploy) and on the heartbeat interval (services/heartbeat.js).
- STATUS_LOG_RETENTION_DAYS env, default 3 (lower than the old hardcoded 7d; the
dashboard only shows a 24h uptime window, so 2-3d is ample for diagnostics).
- Deliberately NO per-device row cap: Step 3's throttle already bounds how fast a
storming device can generate status rows, so a cap would add sweep complexity
for little gain (noted for later if needed).
- NO VACUUM / auto_vacuum here (kept off the hot path); space reclaim is left as a
separate decision (see report).
test: deterministic in-process unit test proves the sweep deletes over-retention
rows across all devices — including a device absent from the devices table and an
offline_timeout row — while keeping recent rows; idempotent on an empty table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gates genuine reconnects PER DEVICE before the heavy register work (DB writes +
playlist build) runs, so a single flapping device can no longer saturate the
event loop and take down the server.
- Actuator is per-device, keyed on device_id (modeled on lastPlayLogAt). A device
is flagged only when it exceeds reconnectBaseMax genuine reconnects per window.
Same-socket playlist refreshes (isPlaylistRefresh) are exempt.
- Load-awareness is BANDED (normal/elevated/critical from the step-2 lag signal),
not a continuous controller. The band only MULTIPLIES an already-flagged
device's backoff; global lag never gates a healthy device.
- Hysteresis: escalate immediately while storming (tighten fast); decay one level
per reconnectReleaseMs of calm (release slow).
- HARD CEILING per device, independent of band and warm-up — a slow-ramp attacker
can't train through it.
- COLD START: for reconnectWarmupMs after boot, force the normal band and apply
only the hard ceiling, so a full-fleet reconnect after a deploy doesn't throttle
healthy screens. State is in-memory, resets on restart.
- Observability: every throttle engagement logs device, band, observed vs allowed
rate, and backoff. Throttled device gets device:throttled + a deferred disconnect.
Tests (api.test.js style):
- unit: healthy-never-throttled, storm-throttled-with-growing-backoff, band
multiplies backoff, hard-ceiling-even-in-warmup, warm-up leniency, neighbor
isolation, slow release.
- integration GATE (the required one): full-fleet reconnect right after restart
throttles NO healthy device; a single device storming IS throttled; a neighbor
stays unaffected while another storms.
- also fixes pre-existing test PORT collisions (my new integration files clashed
with totp.test.js:3979 and totp-keyrotation.test.js:3980 -> moved to 3982/3983);
full suite now green serially AND in parallel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuously samples event-loop delay via perf_hooks.monitorEventLoopDelay()
(C++-backed histogram; cheap). Each window persists mean/p50/p99/max to a new
event_loop_lag table and recomputes a coarse load band (normal/elevated/critical)
from the window p99. Standalone value: current lag is exposed on /api/status and
band changes are logged, so site lag is diagnosable independent of throttling.
The band feeds the #142 reconnect throttle (next commit) but ships first as its
own subsystem.
- event_loop_lag is bounded from day one: indexed on sampled_at + scheduled prune
(LAG_TELEMETRY_RETENTION_DAYS, small default) modeled on the play_logs prune.
Deliberately NOT another unbounded-growth table.
- Band transitions are asymmetric: jump up immediately (tighten fast), release one
level at a time after N calm samples below a deadband (release slow, no flap).
Pure nextBand() function, unit-tested deterministically.
- config: LAG_SAMPLE_INTERVAL_MS, LAG_RESOLUTION_MS, LAG_TELEMETRY_RETENTION_DAYS,
LAG_PRUNE_INTERVAL_MS, LAG_ELEVATED_MS, LAG_CRITICAL_MS, LAG_RELEASE_SAMPLES.
- tests: band-transition unit tests; integration proves sampling persists, stays
bounded under the prune, and surfaces on /api/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A mute toggle wrote the draft playlist_items + emitted a live device:mute-changed but only markDraft()'d — it never updated playlists.published_snapshot, the copy the device actually plays. So the device's item.muted stayed 0 and every loop/reload re-applied full volume: dashboard icon red but audio kept playing (Android; web's native <video> loop masked it). emitMuteChanged now surgically patches the matching item's muted (0/1) inside the published_snapshot and re-pushes the playlist, so loops re-apply the correct flag. Surgical patch (not publishPlaylist) so a mute toggle can't prematurely publish other draft edits or flip publish state. Adds a regression test that fails without the patch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the cache/backoff loop fix (aa23cf0): make a device that can't
self-install visible to operators, and fix the signature-verify bug that kept the
whole #139 fix from engaging on the actual Fire OS target.
Dashboard surface (Phase 2):
- devices gains ota_status / ota_target_version / ota_attempts / ota_updated_at
via the idempotent ALTER TABLE ADD COLUMN migration (non-destructive,
default-backfilled, idempotent on re-run).
- The device reports ota_status (OtaThrottle.statusFor -> none | pending |
manual_update_required) in device_info; the server persists it on register
(the reconnect backstop). devices d.* already surfaces it to the dashboard.
- Dashboard shows a non-blocking amber badge when manual_update_required
("Update available (vX) - install failed N times, manual update required");
i18n key in en.js (non-en inherits via the en fallback). Server suite +1 test.
Event-driven status (Option B):
- New device:ota-status WS message, emitted on STATE TRANSITIONS only
(enter-backoff -> manual_update_required, clear -> none), so the badge updates
promptly without waiting for a reconnect and without per-poll/heartbeat chatter.
Server handler persists the same fields; an unknown/forged device_id is a safe
no-op. The register-path persist stays as the reconnect backstop.
Signature-verify fix (the critical piece):
verifyApkSignature read the downloaded APK's signer via
getPackageArchiveInfo(GET_SIGNING_CERTIFICATES).signingInfo, but that field is
null for ARCHIVE files on API 28/29 (populated only from API 30). On Fire OS 8
(Android 9 / API 28) - the actual deployment target - this returned 0 certs from
a correctly-signed APK, so every OTA was refused as "tampered," the cache was
deleted, and the full APK re-downloaded every check cycle. This was the real
cause of the #139 re-download loop, NOT a silent-install failure: the cache and
backoff added in this branch sit behind this verify gate and never engaged on
the target.
Fix: below API 30, read the archive's signer via the legacy GET_SIGNATURES +
.signatures (its v1/JAR cert, which IS populated on 28/29). Keep
GET_SIGNING_CERTIFICATES + signingInfo for API >= 30 and for the installed-app
read (which works on 28+). The archive's signer is still extracted and compared
to the installed app's signer; a mismatch or zero-cert APK is still rejected.
This reads the cert correctly on old APIs - it does not weaken verification.
Verified on emulators:
- API 28: verify now passes for a legit APK (was: 0 certs, refused). Full backoff
then engages - 8.5MB pulled once, cache-hit on retries, backoff after 3,
manual_update_required emitted once; clears on successful update.
- API 28 negative: a re-signed (different-key) APK is still refused on cert
MISMATCH - no hole opened.
- API 30: unchanged path still passes (no regression).
- server suite 173/173, OtaThrottleTest 7/7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent multi-zone bugs, plus operator-facing warnings, i18n, and
regression tests guarding the data contracts.
Bug 1 — per-item mute was a no-op end to end:
- GET /api/devices/:id dropped the `muted` column from its assignments SELECT,
so the dashboard toggle never reflected state (the muted=false case in
particular). Column restored to the device payload.
- Android player now honours the per-item mute flag for YouTube (initial state
+ live via the IFrame JS API).
Bug 2 — items whose zone_id belongs to a different layout were silently dropped:
- Player fallback (web + Android): an orphaned zone_id is recovered into the
largest zone instead of vanishing, with telemetry.
- server/lib/zone-validate.js is the single source of truth for the orphan rule
(zone not in the device's active layout); used by the device payload
(per-item `orphan` flag + `active_layout_zones`) and the device list
(`orphan_count`).
- Assign-time hardening: a stale zone_id (not in the device's active layout) is
cleared to null on POST/PUT rather than persisted as a new orphan.
- scripts/find-orphan-zone-items.js: read-only sweep for existing orphans.
Dashboard warnings (operator-facing, never on the live player):
- Per-item badge + reassign affordance, device-list glance, preview banner.
- Graceful degradation: the zone selector falls back to /api/layouts/:id so it
can't vanish on a stale payload.
i18n: orphan-zone strings added to en/es/fr/de/pt/it (hi falls back by design;
count strings interpolate through tn()).
Tests: server/test/device-zone-contract.test.js adds 5 regression tests for the
data contracts above (muted true/false round-trip, active_layout_zones, orphan
flag + count, orphan-clears-on-reassign, assign-time clearing). 172/172 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
