The patch4 server derives 3-state liveness (healthy / degraded-reconnecting / offline) and emits it as
data.liveness on dashboard:device-status, but the frontend only consumed binary online/offline — the
signal was thrown away. Add a shared livenessBadge() helper (utils.js) consumed by both the dashboard
device list and the device-detail view (initial render + live statusHandler). Degrades to the binary
status when liveness is absent (old payload / plain reconnect+disconnect emits / DB device object) so
nothing renders blank; unknown/no-data -> offline default. CSS: healthy=green, degraded=amber+pulse
(reads as reconnecting), offline=red — reusing the existing --success/--warning/--danger tokens. Labels
in en.js (all locales fall back to en). Frontend-only; server derivation unchanged. 13/13 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard CSP (script-src 'self') blocked Cloudflare's Web Analytics beacon. Add the two exact
entries the beacon needs (both required — script-only loads but silently can't report):
- script-src: https://static.cloudflareinsights.com (beacon script loads)
- connect-src: https://cloudflareinsights.com (beacon POSTs analytics back)
Exact domains, no wildcards. connect-src already had 'wss:'/'ws:' (socket.io) + 'https:' — those stay,
so the dashboard socket is unaffected; the explicit CF domain documents intent and survives any future
tightening of the broad 'https:'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #150 re-adopt commit (74e7062) left device-detail.js unparseable and, once parsed,
unexecutable — so the entire device-detail view's JS was dead (settings, #150 re-adopt UI, delete):
- SyntaxError at 768: `await api.getContent()` at the top level of the non-async setupActions()
("Unexpected reserved word") -> the whole module fails to parse. Fixed with the .then() pattern
already used by the sibling playlist picker, keeping setupActions synchronous so every listener
below it (save, #150 re-adopt, delete) still registers immediately (making it async would defer
them behind the fetch).
- Stray `async` orphaned on its own line (was line 648) before showReAdoptModal's doc comment:
parses, but executes as the bare identifier statement `async;` -> ReferenceError at module load,
which would keep the view dead even after the parse fix. Removed it.
Also add <meta name="mobile-web-app-capable"> beside the apple- one (clears the deprecation warning).
Full frontend ES-module parse-scan clean; both bugs were confined to this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Server-side keystone: the server now honors the v4 liveness contract uniformly across the MIXED
fleet (v4 + old pre-v4 + disconnected), all three clients depending on it.
- UNIFORM heartbeat-ack: emitted from the single shared device:heartbeat handler (uniform by
construction; no per-client/per-path branch), BEFORE the auth guard so a known device's watchdog
stays armed. Harmless to old clients (they ignore it).
- RECONNECT-WINDOW ack-gap fix (ackableHeartbeat): ack a KNOWN device (authed socket OR a device_id
that resolves) even mid-reconnect; NOT anonymous/never-authenticated sockets (degrade-safe);
identity-agnostic. No state mutation before requireDeviceAuth (auth surface unchanged; device_ids
are uuidv4).
- DASHBOARD LIVENESS (deriveLiveness): server-derived, VERSION-AGNOSTIC Healthy/Degraded/Offline
from signals every client sends (socket presence, heartbeat age, reconnect frequency); no client
status-push.
- IDENTITY CAPTURE (capture-don't-act): client_type/client_version/platform/contract_version columns;
degrades to legacy/unknown for old clients; NEVER breaks register.
- A-BUCKET FIX (QA): recordReconnect + persistIdentity gated on !isPlaylistRefresh (a ~45-60s refresh
is not a reconnect/new identity — matches #134), and the identity write is change-detected — closing
the WAL write-amplification (A1) and the benign-refresh -> false-"Degraded" (A2) regressions.
New lib/liveness.js (pure helpers, unit-tested). 30 new tests (uniform ack, ack-gap, mixed fleet,
identity capture, cross-client conformance, refresh-gate reproduce-then-prove); 366/366 total.
OTA artifact-availability is a separate concern (out of scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/player v4 — client-only (served web player). Brings the browser player onto the locked v4
liveness contract, IDENTICAL on the wire to the now-v4 APK and .wgt:
- v4 liveness watchdog: consume device:heartbeat-ack, lastServerMessageAt refreshes on ANY inbound
(onAny + engine ping) while ARMING gates on the ack (degrade-safe); 45s±10s jittered threshold;
backoff 1s/30s/±0.2 on the socket.io Manager; NO status/health poll; teardown-before-reopen (#148).
- Browser-specific half-open triggers (the /player-unique part): Page Visibility, sleep/resume
(pageshow/bfcache), network change (online) — all drive the SAME #148 teardown-first reconnect.
- THROTTLE-AWARE: silence computed by timestamp (not timer-fire-count); watchdog does NOT act while
the tab is HIDDEN (throttled-timer gap is expected); on becoming visible, verifyLivenessSoon()
resets the grace and reconnects ONLY if genuinely dead — never spuriously tears down a live socket.
- v4 client identity block on register (client_type=player / client_version / platform / v4).
Verified: arm-after-ack both directions, v4 values MATCH .wgt exactly, browser-trigger no-duplicate-
socket, throttle-aware reproduce-then-prove (real extracted checkLiveness/verifyLivenessSoon),
foreground-recovery one-socket. Depends on the server device:heartbeat-ack (core pass) — degrade-safe
until then.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tizen .wgt v4-conformance delta — client-only, targeted (the lifecycle hardening was
already merged; this only closes the v4 gaps the .wgt predated):
- Arm-after-ack (the behavioral change): the watchdog arms ONLY after a
device:heartbeat-ack, not on any inbound/engine ping — so an ack-less/old server
never arms it (degrade-safe). markAlive still refreshes lastServerMessageAt on any
inbound for the silence check; arming gates on the ack.
- Threshold: fixed 35s -> v4 canonical 45s ± up to 10s jitter (re-jittered per connect).
- Backoff params: 1s start / 30s cap / ±20% jitter (was 2s/10s/±50%), exp-double kept.
- v4 client identity block on register (client_type=wgt / client_version / platform /
contract_version=v4), canonical snake_case matching the APK.
- No-poll confirmed. Preserved untouched: keep-awake re-assert, resume handler, #148
teardown-before-reopen, 4th-beat refresh, unpaired 3s backoff.
Verified: arm-after-ack both directions (engine-ping-no-ack -> NOT armed; ack -> arms +
catches half-open), v4 runtime values, #148 one-socket-one-register, server suite green.
Depends on the server device:heartbeat-ack (core pass) — degrade-safe until then.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Client-only, no server change. Implemented in verified clusters:
- H1 (config-proof, no heartbeat-ack): derive the liveness window from the server-negotiated
pingInterval (version-robust read) + arm the watchdog only after a real inbound signal, so it
degrades safe against any server and a raised PING_INTERVAL can't false-fire it into a storm.
- A5: monotonic clock (performance.now) for watchdog/resume deltas — NTP/RTC jumps can't false-fire
or blind the watchdog.
- H4 (leak was verified ABSENT): timer/teardown hygiene — tracked register-retry + teardownSession()
on reset/BACK (stop heartbeat/stream/player-loop/pending-register); all start*() are stop-first.
- A1: single-item playlist retries a broken item (was a permanent black screen while heartbeat green).
- A6: reconnect randomizationFactor 0.5 (no fleet thundering-herd) + timeout 10s->20s (parity).
- A2 (minimal): cache last renderable playlist-update to localStorage, replay on cold-start/offline;
cleared on unpair/reset/auth-error.
- B3: input hardening (non-array assignments/zones guarded, duration_sec numeric-coerced).
- A3: log keep-awake API availability so Bold can VERIFY the flap fix on real hardware.
#148 double-connect discipline re-proven after socket-touching changes.
Replace the resume-only hide-duration heuristic as the AUTHORITATIVE half-open detector with a
real server-silence watchdog, so the .wgt self-heals a dead-but-connected socket from ANY cause
(network drop, NAT idle timeout, transport death while foregrounded), not just resume.
- Central receive-path liveness: markAlive() refreshes lastServerMsgAt on EVERY inbound server
message — app events via socket.onAny, and the server's ~15s engine ping via socket.io 'ping'
(both client-only signals the server already sends; no server change; heartbeats get no ack).
- Watchdog (10s cadence): if socket.connected && authenticated && silent > 35s (2+ missed pings,
under engine.io's own ~45s close), treat as half-open and reconnect via the teardown-first
connect() -> exactly one socket, #118 re-registers once.
- #148 discipline: fires ONLY while socket.connected===true (the state socket.io can't see), so
it never races socket.io's own down-socket auto-reconnect; connect() resets liveness so the
watchdog and the resume fast-path can't double-fire. Resume path kept as the fast suspend path.
- Timers cleared on exit.
Client-only; keep-awake+lifecycle remain the leading flap candidate, NOT a confirmed cause.
Client-only. Brings the standalone Tizen .wgt player toward APK//player parity:
- A: re-assert keepAwake() on a 30s interval (power lock / screensaver-off can be released
when the TV backgrounds the app); cleared on exit.
- B: visibilitychange/resume handler. On resume re-asserts keep-awake and, ONLY for the
half-open case socket.io cannot detect (connected===true after a suspend-length hide),
owns a clean teardown-before-reopen via connect() (exactly one socket, #118 re-registers
once). Defers to socket.io's auto-reconnect when the socket is already disconnected — the
two are mutually-exclusive states so no manual reconnect races socket.io. No manual re-register.
- C: 4th-beat now re-emits device:register (real fallback playlist refresh) instead of a
duplicate device:heartbeat.
- D: APP_VERSION_FALLBACK 1.9.1 -> 1.9.2 (repo hygiene; build-wgt.sh stamps at build).
- F: device:unpaired now backs off 3s before re-registering (symmetric with auth-error),
so MDM re-pair churn can't tight-loop.
Keep-awake (A+B) is the LEADING flap candidate, NOT a confirmed cause. Offline caching (E)
deliberately excluded. No server changes; no bump/tag/build.
Fallback for when the automatic fingerprint-match restore can't fire (factory reset / new
hardware / changed fingerprint). From a device's detail view (UX b): 'Restore from removed
device…' opens a picker of the workspace's removed-device snapshots (GET /devices/removed),
showing device_name + last_seen/removed_at + restore summary (orientation/timezone/playlist),
a Blocked badge, and an Apply action (POST /devices/:id/re-adopt) with a confirm — including an
explicit warning that applying a blocked snapshot re-blocks the target. Refreshes the device
view on success; handles 404/403/400; empty state. Fingerprint shown truncated on-hover only.
Frontend only. Local, no bump/tag.
Delete+re-pair mints a new device row whose INSERT omits every setting, silently resetting
orientation/name/playlist/etc to defaults (Bold MDM churn). Add a fingerprint-keyed
device_settings table (no FK to devices -> survives the cascade): snapshot on DELETE, auto-
restore on fingerprint-match re-pair (relinking the fp to the new id), operator re-adopt API
(GET /devices/removed + POST /devices/:id/re-adopt) for the changed-fingerprint case. Purge on
workspace/user/org deletion (no cross-tenant bleed). Orientation enum-validated on PUT + restore.
blocked preserved (re-enforced by the register kill-switch). Wall membership deferred (TODO).
Backend only — frontend re-adopt UI NOT built (awaiting API review). Local only, no bump/tag.
Close the disk-fill trap: with wal_autocheckpoint=0 a dead worker means nothing checkpoints.
Controller now respawns an unexpectedly-dead worker (bounded: RespawnMax/RespawnWindowMs +
backoff); on exhaustion it re-arms a conservative inline autocheckpoint (FallbackPages) on the
main connection + reclaims the backlog, logging loudly. Clean stopWalCheckpointer() teardown is
distinguished via a 'stopping' flag so SIGTERM never triggers respawn. Env-gated worker
fault-injection (WAL_CKPT_FAIL_START) for tests. Local only — no bump/tag.
Disable wal_autocheckpoint on the main connection; run PASSIVE checkpoints from a
worker_threads worker with its OWN better-sqlite3 handle, escalating to TRUNCATE on a
size high-water or PASSIVE-starvation. Removes the synchronous fsync-heavy checkpoint
from the event loop. Config: walCheckpointIntervalMs/HighWaterMB/StarvationRuns.
Local only — no bump/tag.
The player opened duplicate/rapid WebSocket connections for the same device_id: connect() was
unconditional (disconnect + forceNew socket) and reachable from every lifecycle entry point
(boot, service start, MainActivity/ProvisioningActivity bind, foreground re-bind, START_STICKY).
A ROM that re-binds on foreground (MAXHUB PROC_STATE_TOP, isBindService:true) therefore
re-invoked connect() repeatedly -> a burst of sockets, each evicted by the next (the 8-in-9s
storm). Fire TV never re-binds like that, so it never reproduced.
- ConnectionGuard (new, pure/testable — service is the shell, per the OtaThrottle pattern):
shouldOpenNewSocket(hasSocket, sameUrl, socketActive) — reuse a live/self-healing socket to
the same url; open a new one only when none is usable.
- WebSocketService: connect() is now idempotent (@Synchronized + ConnectionGuard) — every entry
point reuses the one socket, never opens a duplicate; body split into openSocket(). socketActive
/ currentUrl track the single socket.
- Single owner: onStartCommand now calls connect() so the SERVICE owns the one connection
(idempotent across START_STICKY restarts), not whichever activity binds.
- Reconnect discipline: on io server/client disconnect (which Socket.IO does NOT auto-reconnect)
mark the socket inert and schedule exactly ONE backed-off re-open — never a blind re-open loop;
a transport drop keeps socketActive=true so Socket.IO's own reconnect is reused.
Test: ConnectionGuardTest (5, incl. 8-rapid-binds-all-reuse). :app:testDebugUnitTest green
(ConnectionGuard 5, OtaThrottle 7, ScheduleEval 1). NOT bumped/signed/released — Dan builds+signs
with the BMG keystore; 1.9.2-patch2 (server net) covers un-updated devices.
Field-safe SERVER net. A device opening duplicate/rapid sockets (the APK duplicate-socket bug,
separate track) currently churns through evictions during the reconnect-throttle's 30s
post-restart WARM-UP (only the hard ceiling 20 applies then, so an 8-in-9s burst passes
undamped and each new socket evicts the prior). This makes the server absorb it: a thrashing
PAIRED device converges to ONE stable connection and stays online.
- lib/session-settle.js (decision only; bounded, swept): shouldHold(deviceId, incumbentAlive)
— true only when a socket was accepted for this device within SESSION_SETTLE_WINDOW_MS
(config, default 2500ms) AND the incumbent is alive. Warm-up-independent.
- deviceSocket register gate (just before evictPriorSocket): if a LIVE incumbent exists and
we're inside the window, SOFT-REFUSE the new socket (device:throttled reason=session_settle
+ disconnect) and keep the incumbent; else accept + evict + (re)arm the window.
- LIVENESS SAFEGUARD (load-bearing): only hold when the incumbent socket is actually in the
/device namespace — a dead/half-open incumbent is replaced, NEVER stranding the device (max
hold is the 2.5s window from the incumbent's accept, then any new socket is accepted).
- Soft refusal, NEVER a quarantine (reuses patch1's paired-safe philosophy); single-session
enforcement intact for a legitimate move; unpaired/abusive flapping still caught by the
existing limiters. O(1), no loop impact.
Tests (liveness first-class): live incumbent holds + DEAD incumbent replaced (not stranded);
storm of 6 sockets converges to ONE, stays online, not quarantined (during warm-up); single-
session move past the window replaces cleanly; unit decision + bounded sweep. The
evicted-socket-rearm test shrinks its settle window so it still exercises the eviction path.
Suite 336/336.
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
The flap-limiter could 30-min quarantine a PAIRED, legitimate device on reconnect churn.
Behind Bold's single SNAT IP a repeated edge flush -> every device reconnects -> trips flap
-> quarantined -> a recoverable blip becomes a SUSTAINED FLEET-WIDE LOCKOUT we caused.
check(key, now, {paired}) now skips (and clears) the quarantine escalation for a paired
device — it still gets the brief soft cooldown if it truly hammers, but never the long
lockout. The register gate computes paired = device_id && validateDeviceToken(...) (a
matching STORED token, false for missing/mismatch) so a spoofed device_id can't claim the
exemption; unpaired/anon flapping (attacker / unprovisioned hammering) still quarantines.
Tests: unpaired flapper still quarantined; paired never quarantined (soft cooldown only);
paired creds RELEASE an in-flight quarantine; N paired devices from one SNAT IP all admitted
on reconnect and never quarantined across repeated flush cycles.
Live error: "Uncaught (in promise) TypeError: PlayerMediaHealth.shouldShowIdle is not a
function" inside the device:paired socket handler.
WHAT WAS ACTUALLY WRONG: not a missing/misnamed definition — the module DOES export
shouldShowIdle (served + unit-tested). It's a VERSION SKEW: /player/* is served network-first
by the service worker, so a transient module-fetch failure falls back to the STALE cache (an
older player-media-health.js that predates shouldShowIdle) while index.html loads fresh with
the call. window.PlayerMediaHealth then exists but lacks the method, and the call site
guarded the OBJECT (`window.PlayerMediaHealth ? ...`) not the METHOD — so it threw, aborting
the rest of the device:paired handler (the showStatus after it was skipped).
FIX: guard the METHOD at both call sites (typeof X.method === 'function') so a stale/partial
module can never throw — it falls back to the safe inline default (!isPlaying for the idle
decision) and the handler runs to completion.
SIBLING (errors travel in pairs): the needsReattach call in the "Playlist unchanged" branch
had the SAME object-not-method guard. It was inside a try/catch so it couldn't throw uncaught,
but a stale module would silently skip the re-attach/hideStatus. Guarded it the same way.
No service-worker change needed: network-first already self-heals on the next good load; the
method-guard covers the transient/offline-fallback skew permanently.
Tests: player-media-health.test.js +1 module-surface test (both needsReattach and
shouldShowIdle are exported functions — catches the define-vs-call class). Inline player JS
syntax-checked; both guards verified present. Suite 319/319.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONFIRMED from a live console capture (restore cache -> video plays -> reconnect ->
"Playlist unchanged" -> screen falls to "Waiting for content..."). Audio survived because
only the "showing content" VIEW was covered, not the audio path.
ROOT CAUSE: the server re-emits device:paired on EVERY re-register of an already-paired
device (ws/deviceSocket.js:510) — i.e. on every reconnect, while content is already playing.
The player's device:paired handler called showStatus('Waiting for content...') UNCONDITIONALLY
(the "falls through to idle" sibling), putting the idle overlay OVER the live video. The
following device:playlist-update -> "Playlist unchanged" branch returned early and never
cleared it, so the idle screen stuck on top of playing content.
FIX (idle screen only when genuinely idle; unchanged is a strict no-op that keeps playback):
- lib/player-media-health.js: new shouldShowIdle(state) — idle ONLY when nothing is playing
AND there's genuinely no content. Already-playing (or content-present-about-to-render) is
never idle.
- device:paired handler: gate showStatus on shouldShowIdle({isPlaying, hasContent}) instead
of showing it unconditionally. On a reconnect while playing -> no-op.
- "Playlist unchanged" branch: when healthy playback is confirmed, hideStatus() to clear any
stale idle overlay a reconnect's device:paired may have put up — so the confirmation can
never leave "Waiting for content..." over live content. Still leaves the actual media
element exactly as-is (no teardown, no flicker).
- sw.js cache v10 -> v11.
SIBLING SCAN: device:paired was the only unconditional idle reset. The connect() idle
prompts (connecting / connecting_muted) were already guarded by !isPlaying; empty-playlist
and no-renderable idles are genuine.
Tests: player-media-health.test.js +2 (shouldShowIdle: playing never idle; idle only when
empty+not-playing). Inline player JS syntax-checked; module served + guard referenced on a
booted server. Suite 318/318.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT CAUSE (hypothesis A, pre-existing — NOT a beta7 regression; server/player/index.html
is untouched since v1.9.2-beta6): handlePlaylistUpdate's "Playlist unchanged" branch blindly
returned. The media re-attach (renderContent) lives ONLY in the content-changed branch, so
if the <video> surface was lost (element detached from the DOM while still decoding — video
gone, audio still playing) a no-new-content refresh never re-attached it. New-content
refreshes were fine because they re-render.
FIX (make the refresh idempotent for the media surface, no flicker on the healthy path):
- server/lib/player-media-health.js (new, UMD + unit-testable, mirrors schedule-eval.js):
needsReattach(state) — re-attach ONLY when playback should be happening but the current
item's surface is actually lost (video null / detached / ended / errored; non-video: no
mounted surface). A healthy attached+live video returns false, so a routine poll stays a
no-op (no re-render, no flicker). Served at /player/player-media-health.js from the single
source; loaded by the player.
- index.html no-change branch: extract the current item's DOM facts and, iff
PlayerMediaHealth.needsReattach, call playCurrentItem() to re-render the current item.
Wrapped so the health check can never break a refresh.
- teardownCurrentMedia: also release currentVideoEl even when it was DETACHED from the
container — a detached-but-playing <video> keeps emitting audio and the container-scoped
querySelectorAll can't find it. This kills the "ghost audio" on re-attach.
- sw.js cache bumped v9 -> v10 so players pick up the new index.html + module.
Tests: test/player-media-health.test.js (6) exercises the branch selection — healthy video
-> no re-attach; detached/null/ended/errored -> re-attach; idle -> never; non-video by
surface presence. Inline player JS syntax-checked; module served + referenced verified on a
booted server. Suite 316/316.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The billing:read scope + dual-path gate were built but there was no way to MINT a token
(and it must NOT go in the workspace-scoped, self-service API-Tokens UI). Adds a server-side,
owner-only CLI — no new UI, no network endpoint. Owner-only BY CONSTRUCTION: it's a
host-side script, so filesystem/shell access = the platform owner.
- server/lib/billing-token.js (testable): mintBillingToken/revokeBillingToken/
listBillingTokens. Reuses the EXACT existing token path — same secret (st_ + 32 bytes
base64url), same SHA-256 hashing (hashToken), same api_tokens columns — no second format.
Resolves the platform OWNER (oldest platform_admin/superadmin; #14 collapsed superadmin ->
platform_admin so that's the top tier) and binds to their workspace. api_tokens.user_id +
workspace_id are BOTH NOT NULL (no platform-level token exists); the workspace binding is
VESTIGIAL for billing (billing:read is off-ladder -> can't reach any workspace router;
billing is platform-global), documented in-file rather than loosening NOT NULL pre-release.
- scripts/mint-billing-token.js: thin CLI wrapper. --name mints and prints the secret ONCE
(+ id, + "run as owner on host" warning), --list, --revoke <id> (soft revoke, mirrors the
dashboard DELETE).
Tests (4, test/billing-token-mint.test.js): minted row is scope EXACTLY billing:read with a
matching SHA-256 hash and no read/write/full/agency scope; the token reads GET
/api/billing/usage (200) but is refused on /api/devices (403) and /api/admin (401) — scope
isolation; revocation -> 401; mint requires a name; revoke refuses a non-billing id. CLI
smoked live (mint/list/revoke). Suite 310/310.
SPEC-vs-REALITY (again): spec said bcrypt + JSON `scopes`; this codebase uses SHA-256 + a
single `scope` TEXT column. Built to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Least-privilege way to read GET /api/billing/usage without requiring platform admin.
Additive + isolated: reuses the existing api_tokens scope system (the off-ladder 'agency'
scope is the precedent) and does NOT touch the shared role/permission checks other
endpoints rely on.
- New off-ladder scope 'billing:read' (routes/tokens.js SCOPES). Like 'agency' it is NOT
on the read<write<full ladder, so tokenScopeGate rejects a billing token on every
PUBLIC_ROUTER and JWT-only routers reject any st_ token -> the scope grants billing-read
and NOTHING else.
- DUAL-PATH gate requireBillingRead (middleware/apiToken.js), written as an EXPLICIT OR:
authorize if (billing:read token) OR (platform-admin session). Admins keep read access
but are NOT required to; the token path doesn't lock out admins or vice versa. Billing
route now mounted with bearerAuth (token OR JWT front door) + requireBillingRead (was
requireAuth + requirePlatformAdmin).
- MINTING is platform-admin only (stricter than read/write/full/agency, which any
workspace member may mint) since a billing:read token grants GLOBAL billing-read. Note:
no finer "owner" tier exists here (#14 collapsed superadmin->platform_admin), so
PLATFORM_ROLES is the top level required.
Tests (5, test/billing-authz.test.js): dual-path positive (token AND admin session both
200) + negative (user 403 / anon 401); scope isolation (billing token 403 on /api/devices,
401 on /api/admin; read token 200 on devices but 403 on billing); minting owner-only
(user + ordinary-admin 403, platform-admin 201); revocation -> 401. Existing token
firewall/partition suite (api.test.js) + billing-endpoint tests unchanged & green. Reused
the exact SHA-256 token-verification path (no bcrypt/new mechanism). Suite 306/306.
NOTE: spec described bcrypt + JSON `scopes` + an analytics:read precedent; this codebase
actually uses SHA-256 + a single `scope` TEXT column + 'agency' as the off-ladder
precedent. Implemented faithfully to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updated the item-B section for the in-memory time-limited auto-quarantine (no DB block,
auto-clears) and the item-C section for band-aware downloads (serve freely when healthy,
caps only under load). Both reference the new /api/status debug observability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The storm harness asserted a tick COUNT (environment-timing-sensitive) that could flake
in CI. Kept the real invariant (max event-loop gap < 300ms) and loosened the tick
assertion to >=2 (enough to have measured a real gap), which cannot flake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the new internal states so we can SEE the limiters biting during the alpha soak
instead of grepping logs. /api/status now carries debug: {
flap: {buckets, quarantined},
ota_download: {inFlight, servedThisWindow, shedThisWindow, windowCount},
maintenance: {deleted, ms, at, running}, // last status-log prune
log_coalescer_buffer,
}. Aggregate counts only (no device ids/secrets), cheap in-memory reads. stats()
added to flap-limiter + ota-download-guard (singleton prod state), getMaintenanceStats
from database. Asserted in the booted /api/status test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coalesced loop-lag summary carried an arbitrary sample's p99. Now record() tracks
the MAX (peak) over the window and the summary emits it — the peak is the number that
matters during an incident: '[loop-lag] band=critical (x47 in 30s, peak 1502ms)'. Band
CHANGES still log immediately, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s
while the table is still large (chunked startup prune trickling in the background), and
the backlog drains to the cap with the server responsive throughout. The old whole-table
sort froze boot ~40s. Fallout doc gets the P2 findings section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigated the fleet-wide simultaneous reconnect concern. buildPlaylistPayload is
synchronous but cheap: measured avg 0.078ms / max 0.70ms per call with a 200-item
snapshot; 230 devices = ~18ms CPU spread across 230 separate handler invocations (the
loop yields between them), never one block. Per-call << 50ms invariant. No change
needed; documented with the measured number.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block is a real lever now, so cover the authz path. Booted server + JWT; device and a
viewer membership seeded via the DB file. Proves POST /api/devices/:id/{block,unblock}:
owner (workspace_admin) -> 200; unauthenticated -> 401; cross-workspace user -> 403;
workspace_viewer -> 403 (read-only), with the DB blocked column asserted at each step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fixed OTA_DOWNLOAD_MAX_CONCURRENT=10 throttled a legitimate coordinated rollout even
on a perfectly healthy server, and a shed 503 costs a client a full ~30-min re-check
cycle. Made the download guard's concurrency + rate caps band-aware:
- normal -> serve FREELY (no cap): a whole-fleet rollout isn't staggered when healthy
- elevated -> the configured caps engage (early backpressure)
- critical -> shed 503 (the real protection, unchanged)
Kill switch OTA_DOWNLOAD_GUARD_ENABLED=false disables it entirely.
Tests updated: normal serves 50/50 with 0 shed; caps + shed now asserted under elevated;
storm harness OTA flood runs under elevated (the loaded state). Suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
