A duration_sec=0 assignment (especially a widget) made the player schedule a 0ms
auto-advance, self-looping and black-screening the TV. #198 fixed the Android
client; this hardens the source so a 0 can't be stored or served in the first
place. assignments.js accepted an explicit 0 on the POST/PUT/copy write paths —
the `= 10` destructure default only covers an ABSENT field, not an explicit 0.
- Add normalizeDuration() and apply it on all assignment write paths so any
missing/invalid/<1 duration is floored to the 10s default.
- Add an idempotent migration repairing existing playlist_items rows with
duration_sec IS NULL OR < 1 (fixes the live widget on existing DBs).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vertical auto-scroll is a CSS keyframe that translates the track by
cycleH = baseH + GAP_PX and loops linear infinite. GAP_PX was 100 but the actual
.gap element between the content and its seamless clone is 120px, so every cycle
the reset landed 20px off — a visible jump/stutter once per loop.
Set GAP_PX = 120 to match the .gap CSS, and drive each gap element's height from
GAP_PX inline so the scroll math and the rendered gap can never drift again.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to #195. The CORP fix there landed on routes/content.js `/:id/file`, but that
handler is SHADOWED: server.js registers a public `app.get('/api/content/:id/file')`
(and `/thumbnail`) BEFORE the auth-gated content router, and that public route (gated by
playlist/widget reference) is what actually serves widget logo/background images. So the
header never changed on the wire — origin still returned CORP: same-origin and the player's
sandboxed (opaque-origin) widget iframe kept getting NS_ERROR_DOM_CORP_FAILED / 0 bytes.
Set Access-Control-Allow-Origin: * + Cross-Origin-Resource-Policy: cross-origin on the real
public routes in server.js: /file, /thumbnail (local), and the remote-thumbnail proxy.
Revert the now-dead content.js edit so the fix lives only where the bytes are served.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(widgets): bulk import for the directory board (JSON / CSV / TSV / text)
Adds an "Import from JSON / CSV" button to the directory-board editor. Paste JSON
(the { company, tenantsByFloor, advertisements, backgroundImages } shape plus
categories[]/floors[]/flat-array/bare-floor-map variants), a CSV/TSV/pipe/semicolon
table (with or without a header — vacant/yes/1 => available, quoted fields), or a
sectioned "room name" text list, and it auto-fills title, footer, floors->categories,
rooms/names/details/availability, and background-image URLs. "Replace / append" toggle.
Tolerant key matching (room/suite/unit/id, name/tenant/company, details/subtitle, …);
warns on things it can't use (bare-filename background images, headerless columns).
parseDirectoryImport is pure and was unit-tested in node across every format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(widgets): directory board — logo replaces title, and images load on the player
Two on-screen bugs on the directory board:
1. A logo did not remove the title text — both rendered, stacking the wordmark over
the name. renderDirectoryBoard (and the directory-search header) now gate the title
h1 behind !logoSrc, so a logo replaces the title. New render test guards it.
2. Logo + background images did not show on the player (NS_ERROR_DOM_CORP_FAILED,
0 bytes). The player embeds widgets in a sandbox="allow-scripts" (opaque-origin)
iframe, so /api/content image requests are cross-origin, and the helmet default
Cross-Origin-Resource-Policy: same-origin blocks them. Set CORP: cross-origin (+
ACAO:*) on the content file + thumbnail routes, matching the existing /uploads/content
static route. Content already serves publicly, so no new exposure. Verified in a real
sandboxed iframe: same-origin blocks, cross-origin loads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Tizen .wgt player black-flashed between IMAGE items on slow decode HW
(Samsung OM55B / SSSP). Root cause: playCurrent() called clearStage() before
renderImage() set img.src, so the stage was empty (black) until the new image
decoded. Images had no decode-gated double-buffer — video gained one in #167
(938a43a), which made the always-present image flash conspicuous by contrast.
renderImage has been byte-identical since the first Tizen commit, so there was
nothing to revert; the buffer had to be ADDED, mirroring the video path.
- preloadImage()/_takePreloadImage()/_releasePreloadImage(): one-ahead,
image-only decode-gated buffer modeled on preloadVideo()/_takePreload().
Detached <img>, src set, warmed via HTMLImageElement.decode() (feature-
detected — onload/complete fallback for Tizen 5.0 / SSSP6). Warmed when the
current image begins its dwell and from the group-sync boundary tick.
- renderImage now SWAPS: take the pre-decoded <img> (or decode a fresh one) and
only THEN clearStage()+append, in one synchronous block — the compositor
never sees an empty stage. Never clear-then-load on the image path.
- Scoped to images only: playCurrent() skips the up-front clearStage() solely
for image targets (same branch order as the dispatch); video/youtube/widget
keep their pre-dispatch clear untouched. onerror and decode() rejection route
to skipSoon(); stale-index guard blocks mounting a stale decode over the
current item after next()/gotoIndex/load(); one-ahead with stale release on
index move, load(), stop(), and group-sync exit. #A1 single-item heal intact.
Tests:
- server/test/tizen-image-blackflash.test.js (new, 4/4): loads the real
player.js in a vm context with a test-controllable decode() Promise and proves
the invariant — across image->image the #stage is NEVER without a mounted
<img> (old element held until the new image's decode()/onload resolves, then
swap). Covers decode() supported, decode() absent (onload fallback), and
broken-image (decode-reject / onerror) -> skipSoon. Proven to VIOLATE on the
old clear-then-load ordering and HOLD on the fix.
- server/test/pip-overlay.test.js: the decode-gate makes image mount async,
which broke its older shim (no decode()/onload/complete). Teach the shim
element complete/naturalWidth so renderImage takes its synchronous
complete-fallback branch and mounts. Test-only.
Full server suite 486/486. node -c clean. Headless proves the DOM ordering
invariant (the flash's precondition); final black-frame sign-off needs a real
OM55B panel (manual steps in the test-file header).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A fresh unclaimed player that reconnects (same fingerprint) INSIDE the server's
~5s deferred-offline grace hit a false 'active on another connection' reclaim
reject, then collided on UNIQUE(devices.pairing_code) on the fall-through INSERT
and wedged unclaimed with no content. Real trial customer (web player) hit it.
server/ws/deviceSocket.js:
- Fix A (guard): gate the liveConn reclaim reject on !inDeferredOffline
(pendingOfflines.has(id)). A device mid-deferred-offline is a zombie, not live,
so a same-fingerprint reconnect is a legit reconnect, not a hijack. A genuinely
live socket (never disconnected -> no pending-offline) still rejects a cloned
fingerprint -> anti-hijack boundary preserved (documented).
- Fix B (idempotency): when the unclaimed old row holds the SAME pairing_code the
reconnecting player presents, ADOPT/refresh it (mirror the claimed-reclaim path,
but no device:paired) instead of INSERT-colliding. Differing-code case unchanged.
- deferOffline is NOT shrunk (it exists to prevent transient-blip flapping).
server/player/index.html:
- The cold-boot flap source: an unfiltered pageshow handler ran verifyLivenessSoon()
on every load, opening+registering a socket early, which the boot connect() then
tore down and rebuilt (connect->register->disconnect->reconnect). Guard it with
ev.persisted (mirror the pagehide guard) so only real bfcache restores trigger it.
server/test/pairing-race.test.js:
- Forces the race against the real socket server (log-gated reconnect inside the
deferred-offline window), asserts no false reject / no UNIQUE collision / single
claimable row; + a hijack case asserting a cloned fingerprint on a genuinely-live
display is still rejected. Web- and android-shaped fingerprints. Fails 2/4 on
pre-fix code, 4/4 with the fix.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Vertical Shorts were played in a player forced to 100%x100% on a landscape
frame, so they looked wrong (pillarboxed/small). Option A: detect vertical at
ingest, persist it, and have every player honor it.
- Ingest (routes/content.js): detect a Short from the /shorts/ URL form OR
portrait oEmbed dims (oEmbed now queried with the ORIGINAL url so /shorts/
reports its true dimensions), and persist it as st_aspect=vertical on the
stored embed URL. That's the only signal players get (remote_url), so it must
be captured at ingest, not re-derived per loop. YouTube ignores the unknown
param; players read the video id, not the full URL, to build the embed.
- Players read st_aspect=vertical and center a 9:16 box (fills a portrait screen,
pillarboxes cleanly on landscape) instead of 100%x100%:
web (player/index.html), Android (WebViewSupport.youtubeEmbedHtml), Tizen
(player.js single-zone + zone paths). Dashboard library uses a static thumbnail,
so it's unaffected.
Not doing Option B (yt-dlp): runtime dep + storage/bandwidth + maintenance +
YouTube ToS; embed-disabled Shorts already skip gracefully.
Tests: youtube-shorts.test.js (4) — /shorts/ and portrait-dims tag vertical,
landscape stays untagged, /shorts/ tags even if oEmbed fails. Android compiles;
web player inline JS + Tizen player.js parse.
Note: pre-existing Shorts added before this aren't retagged (would need an oEmbed
backfill) — re-add to fix, or a follow-up migration.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(widgets): add directory-search widget
An interactive, walk-up search view of an existing directory-board. It
references a source board by id (no data copy), so a venue can run the
scrolling board on a main screen and a search view on a tablet, letting
people find an entry instantly.
Server (routes/widgets.js):
- register 'directory-search' + renderDirectorySearch(): resolves the source
board, inlines its categories as one \u003c-guarded JSON blob, renders all
text via textContent (XSS-safe), live case-insensitive filter over
identifier/name/subtitle (debounced), grouped results, available styling,
optional touch on-screen QWERTY keyboard that drives the same filter path.
- missing / non-directory-board source -> friendly full-page fallback, not a 500.
- live-sync while open is out of scope; left a // TODO for a poll hook.
Frontend editor (views/widgets.js): type + magnifier icon, source-board
dropdown (from loaded widgets, filtered to directory-board), title, logo
(reuses the board's picker), placeholder text, theme, on-screen-keyboard toggle;
getConfigFromForm case. i18n: widget.dirsearch.* + type keys in en/es/it/de/pt/fr.
docs: openapi widget_type enum. Tests: server/test/directory-search.test.js.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(widgets): live-sync for directory-search (poll source board, no reload)
Reflect directory-board edits on an open directory-search page without a reload.
- New public GET /api/widgets/:id/data.json returns { categories } for a
directory-board (404 for missing/wrong-type so the page keeps last-good data
on a transient miss). CORS-open (ACAO:*) + no-store so a null-origin sandboxed
widget iframe can read it; exposes only data already public via /render.
Exempted from CSP + auth in server.js alongside /render.
- directory-search page inlines its source_widget_id and polls the board's
data.json every 30s via a relative URL (works behind a proxy/base path and
from a null-origin iframe). Only rebuilds + rerenders when the data actually
changed, so a mid-search view isn't disturbed; skips while document.hidden;
keeps last-good data on any fetch error. Flatten logic factored into
buildFlat() and reused by the poll.
Tests: data.json feed (categories, CORS header, 404s) + poll wiring in
directory-search.test.js (13/13). Verified live in a browser (page.clock
fast-forward): editing the board updates the search page with no reload.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(android): let player WebViews take touch focus for interactive widgets
directory-search is served through the existing generic widget path
(loadUrl <server>/api/widgets/:id/render), so it already renders on Android
with JS + DOM storage + mixed-content enabled, same-origin (so its live-sync
fetch of the source board's data.json works), and no touch blocking.
Add isFocusable/isFocusableInTouchMode to the shared WebView config so the
search field reliably takes a tap/cursor inside the kiosk lock-task WebView.
Harmless for passive widgets (board/YouTube have no focusable inputs); the
widget's own on-screen keyboard still drives the filter when the system IME
is suppressed. Verified with :app:compileDebugKotlin.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bold Media Group's fleet broke on the 1.9.3->1.9.6 upgrade. Their MDM does an
uninstall/reinstall (app data wiped), so the player registers with
{ pairing_code, fingerprint } and NO device_id and shows a pairing code — but the
dashboard reported "code does not exist". Deleting the device_fingerprints row fixed
it, which pinpointed the fingerprint-reclaim guard in server/ws/deviceSocket.js.
Root cause: the reclaim guard was
`stillAlive = !!liveConn || secondsSince < reclaimSettleSeconds; if (stillAlive) reject`.
On an in-place reinstall the old row heartbeat seconds ago, so `secondsSince < 300` is
ALWAYS true -> it emitted device:auth-error and returned BEFORE the pairing_code INSERT,
so the code the player displayed never existed server-side.
The settle window's real purpose was to REMATCH an existing fingerprint back to its
device row on reinstall — not to force a fresh re-pair. So the fix keys off claim status,
not the timer (server-only; no APK change — reviewed and confirmed unnecessary):
- Reject ONLY when the old row has a genuinely LIVE socket (liveConn) — the real anti-
hijack boundary. Unchanged.
- CLAIMED old row (user_id set) -> RECLAIM it regardless of the settle window: reuse the
row, rotate the token, emit device:registered{online} + device:paired. The panel returns
straight to paired (no operator re-pair, no orphaned duplicate row), preserving name /
claim / playlist / content. device:paired drives the app off the pairing screen, so the
fresh code it showed is irrelevant.
- UNCLAIMED old row -> fall through to the pairing_code path and PROVISION FRESH with the
shown code (reclaiming would leave a stale/null code -> "code does not exist"). #150
relinks the fingerprint to the new row.
`reclaimSettleSeconds` is now vestigial for this path. Trade-off: a fingerprint-only reclaim
of a CLAIMED-but-offline device is no longer delayed ~300s — not a new attack class (the old
code already granted it once the window elapsed); liveConn remains the hard boundary. Truly
closing that window without a re-pair needs client keystore attestation (a future APK).
Also fixes a latent crash this newly exercises: middleware/subscription.js getUserPlan()
dereferenced an undefined user in its else branch ("Cannot set properties of undefined
(setting 'trial_active')") when the user/plan JOIN missed. Under the claimed-reclaim path
that ran checkDeviceAccess->getUserPlan, the throw was swallowed by the reclaim try/catch and
silently dropped the device to provision-fresh. Guard: `if (!user) return null`.
Tests (server/test/fingerprint-reclaim.test.js):
- NEW: a CLAIMED reinstall reclaims the SAME row, emits device:paired, creates no duplicate,
keeps the fingerprint linked — regardless of the settle window (the Bold repro, fixed right).
- NEW: recent heartbeat + no live socket, UNCLAIMED -> provisions fresh with the shown code.
- NEW: a LIVE old socket still rejects and creates no new row (security preserved).
- Updated the #143 gone-device test to expect provision-fresh for an unclaimed row, and the
log-noise assertion to the "reclaim rejected" message.
465/465 server tests pass. Server-only: NOT deployed, no version bump, Android untouched.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Adds a pluggable email transport so self-hosters without Azure/M365 can send
mail through any standard SMTP server (Postfix, Gmail, Mailgun, SendGrid, corp
relay). Graph stays the default; behavior is byte-for-byte unchanged when
EMAIL_TRANSPORT is unset or "graph".
- config: EMAIL_TRANSPORT ("graph"|"smtp", default graph) + SMTP_HOST/PORT/
SECURE/USER/PASSWORD/FROM.
- services/email.js: branch by transport behind the SAME public sendEmail()/
isConfigured() surface. SMTP via nodemailer (lazy-required, like MSAL).
Shared across both transports: the "[ScreenTinker] " subject prefix (unless
rawSubject), the GRAPH_DEV_RESTRICT_TO allow-list, html-from-text derivation,
and the never-throws contract (failures log + return sent:false). SMTP_SECURE
true=implicit TLS(465)/false=STARTTLS(587). Auth optional (unauthenticated
relay ok); SMTP_USER without SMTP_PASSWORD is flagged. SMTP_FROM parses
"Name <addr>". New emailConfigStatus() for startup diagnostics.
- server.js: startup logs the transport and a LOUD error when the selected
transport is partially configured (some fields set, others missing) or when
EMAIL_TRANSPORT is invalid (falls back to graph). A fully-unset transport
stays a silent stdout fallback (unchanged dev behavior).
- nodemailer ^6.9.16 added as a production dep (bundled in the Docker image).
- .env.example + README: SMTP config section, Gmail example, transport table.
- test/email-transport.test.js: 15 tests — transport selection, config
validation (missing/partial/invalid), SMTP message building (from/prefix/
fromName override/text alt), sendEmail routing (mocked nodemailer), rawSubject,
dev-restrict on smtp, and the smtp_error never-throws path.
462/462 server tests pass. Boot verified for all four states (configured,
misconfigured, invalid, default).
Closes#173
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
express.static runs with index:false, so a bare /integrations/ fell through to the
SPA catch-all and rendered the dashboard login instead of the integrations hub — the
top-nav "Integrations" link, the canonical, and the sitemap entry all dead-ended at
login. Add an explicit route (like /agency, /sitemap.xml): /integrations/ -> the hub's
index.html, and /integrations -> 301 /integrations/. Spoke pages are real .html files
already served by static. Folded into a re-cut of v1.9.6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(diagnostics): device incident log — why a screen went offline/black, with device-attested cause
Field request (Bold/s_t_r_o_b_e): "screens go offline randomly — let us see the cause." Answers it
across the fleet with a unified incident log, and — the key insight — lets the DEVICE disambiguate
the cause the server can't: if the app process survived the gap it was a NETWORK problem (not a
reboot), and it can even tell a dropped Wi-Fi/Ethernet link from a link-up-but-server-unreachable
(router/upstream) failure.
Schema:
- device_status_log gains reason + detail (WHY each offline transition happened).
- NEW device_events table (unified incident feed): type (offline/online/display_off/display_on/
crash/reboot/network/app_error) + reason + detail, indexed, age-pruned + per-device capped.
Server:
- Capture the socket.io disconnect REASON (transport_close/ping_timeout/transport_error) instead of
discarding it — recorded in the offline-cause log. devices.offline_reason stays on the EXIT-SIGNAL
contract (crashed/clean_exit/silent) — a separate axis, preserved (violent death = 'silent').
- device:event handler (typed incidents) + device:connectivity-report handler (device-attested).
lib/incident-classify.js (pure, unit-tested) composes reason+detail: cold_start->reboot;
link_lost->network "Wi-Fi/Ethernet link lost"; else network "LAN up, server unreachable
(router/upstream)"; appends SSID / weak-signal (rssi<-75) / IP-changed. On a report it upgrades the
most-recent offline row from the server's guess to the device's ground truth.
- heartbeat timeout -> 'heartbeat_timeout'; retention/cap for device_events.
- Device-detail API returns statusLog.reason/detail + the last 50 device_events.
Device (Android WebSocketService): ConnectivityManager default-network callback (link-lost during a
gap) + Wi-Fi SSID/RSSI + IP snapshot -> device:connectivity-report on reconnect (app survived =>
network); ACTION_SCREEN_ON/OFF receiver -> device:event display_on/off ("screen went black"). All
guarded/feature-detected; no manifest change; compiles clean.
Web + Tizen players: reconnect connectivity-report (link_lost from navigator.onLine during the gap)
+ visibilitychange -> display_off/on. Best-effort (no wifi detail in a browser). Tizen exit-signal
marker slice untouched.
CMS (device-detail): the offline cause on the uptime-timeline hover + a new "Recent incidents" panel
(merged offline periods + typed events, friendly labels, detail, relative time + down-duration).
Built as a 4-way parallel agent fan-out over disjoint domains against a locked contract, then
integrated. Verified: full server suite 443/443 (incl. the seam fix keeping the exit-signal contract
intact), Android compileDebugKotlin clean, all players + CMS node -c clean.
Refs #170.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(diagnostics): internet-reachability probe — split "our server down" from "no internet" (#170)
Follow-up to the incident log: when a device's link is UP but it's offline, "router/upstream" was a
catch-all. The device now probes a public host (1.1.1.1 / 8.8.8.8 :443) DURING the gap, so the cause
pinpoints blame:
- link_lost=true -> Wi‑Fi/Ethernet link lost (device's own link)
- link up, internet_ok=true -> server_down: internet reachable, OUR server was unreachable
- link up, internet_ok=false -> no_internet: router/ISP down
- link up, no probe result -> generic router/upstream (unchanged fallback)
- Android WebSocketService: fire a short daemon-thread TCP probe (443, either host) at disconnect;
the result rides the connectivity-report as internet_ok (omitted if the gap ends before it finishes).
- lib/incident-classify.js: 3-way split on internet_ok; new reasons server_down / no_internet.
- Frontend i18n: device.event.server_down / .no_internet labels.
- Tests: +3 classify cases (server_down, no_internet, link_lost wins over internet_ok). 12/12.
Verified: classify 12/12, Android compileDebugKotlin clean, node -c clean. Refs #170.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(diagnostics): log an 'upgrade' incident (old → new app_version) — server-side (#170)
When a device reports an app_version different from the stored one, applyDeviceInfo logs an
'upgrade' device_events row (detail 'old → new'). Server-side, so it covers Android/Tizen/web with
no client change; a fresh pair (no prior version) isn't counted. Adds device.event.upgrade label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The subprocess-booting test suites hand-picked fixed ports in a cramped ~3955-4021 range, and
156-schedule-read-path deviated to a RANDOM port (3900 + rand%90) that overlapped those fixed
ports. Under CI load two servers could race on the same port, surfacing as flaky "no such table:
devices" / "FOREIGN KEY constraint failed" (a server answering a request against a half-migrated
or wrong DB). It's environmental — the suites pass locally and in isolation.
Fix: a shared test/helpers/free-port.js (bind :0 on loopback, read the OS-assigned port, release)
called in before() so every suite gets a guaranteed-unique ephemeral port — concurrent suites can
no longer collide, and no one has to hand-assign ports.
- Codemod converted 30 suites: const PORT = <fixed|random> -> let PORT (+ BASE) assigned via
`PORT = await freePort()` at the top of before().
- 3 hand-fixed (different structure): 148-eviction-storm (lowercase `base`), boot-health (no
before() — allocates PORT + a throwaway SEED_PORT inside the test, replacing the hardcoded
3894), totp-keyrotation (no before() — allocates at the test start before bootServer()).
No fixed 39xx/40xx ports remain. Full server suite 435/435; the 4 hand-touched suites pass in
isolation. Pure test-infra change — no app code touched.
* fix(content): rotation-aware media dimensions — portrait no longer stored landscape (#170)
Ingest recorded CODED width/height and ignored rotation, so a portrait phone video
(coded 1920x1080 + 90° Display-Matrix) or a portrait photo (EXIF orientation 6) was
stored LANDSCAPE. The player then rendered it wrong-aspect and letterboxed — the
"portrait content degraded + blue bar at the bottom" symptom in #170. The reporter's
workaround (pre-rotate + mark Landscape) is exactly what this bug forces.
- lib/media-orientation.js (new): pure, unit-tested display-dimension helpers = single
source of truth for ingest AND the backfill. videoDisplayDims() reads the modern
Display-Matrix side_data rotation (falls back to the legacy tags.rotate, sign-normalized);
imageDisplayDims() honors EXIF orientation 5..8. Odd quarter-turns swap W/H.
- lib/content-ingest.js: use the helpers for stored dims; add sharp .rotate() so image
THUMBNAILS are auto-oriented too (video thumbs were already auto-rotated by ffmpeg).
- scripts/backfill-rotation-dims.js (new): idempotent, dry-run-by-default maintenance to
correct already-uploaded portrait media (re-probe -> fix dims -> regenerate image thumbs).
- test/media-orientation.test.js: 5 bites (tag + Display-Matrix, sign/normalize, EXIF 5..8,
the blue-bar landscape->portrait case, null-safety).
Scopes #170 to its residual-on-1.9.4 issues; the 1.9.3 "never displays" slice was #162 +
the remote_url-null download fix, already shipped in 1.9.4. The slow low-res/orientation-
cycling first load is tracked separately in #170 pending repro data.
Refs #170.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(android): honor EXIF orientation in ImageLoader so portrait photos render upright (#170)
Completes the rotation-aware media fix on the PLAYER side. The server ingest fix (this
branch) corrects stored dimensions + auto-orients the thumbnail, but the panel draws the
full-res original via BitmapFactory, which ignores EXIF — so a portrait photo (landscape
pixels tagged "rotate 90") still rendered sideways on the screen. QA root-cause pass on
#170 caught this gap: the Android player reads no stored dims and applied no EXIF.
ImageLoader now reads the EXIF orientation (from the file for cached content, from the byte
stream for remote_url images — ExifInterface(stream) is API 24+, minSdk is 24) and rotates/
flips the decoded bitmap via a Matrix (all 8 orientations). NORMAL/UNDEFINED is a no-op (no
extra allocation); a transformed copy recycles the source; OOM falls back to the source
rather than crashing. Videos were already correct (ExoPlayer honors the rotation matrix).
Verified: :app:compileDebugKotlin clean.
Refs #170. Rides with the server rotation-dims fix on this branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Agency-portal uploads previously all landed at the workspace library root, unsorted.
Instead of the issue's whole-workspace folder dropdown (which would leak every folder
name to an external party), bind ONE folder per agency token — admin-controlled and
agency-invisible — and scope the portal picker strictly to that folder's own subtree
(Hybrid-C). Fully backwards-compatible: no bound folder -> root, exactly as before.
Model / multi-workspace: an agency token is bound to ONE workspace at issuance, so the
token key IS that workspace's private link and the bound folder lives in that workspace.
An admin with N workspaces mints one token per workspace (each with its own auto-folder).
No workspace-switcher in the portal — the token is the tenant boundary.
Backend:
- api_tokens.upload_folder_id (additive; ON DELETE SET NULL -> deleting the folder falls
back to root).
- lib/agency-targets.folderSubtree(): recursive-CTE helper = the SINGLE confinement source
shared by GET /api/agency/folders AND the POST /api/agency/content target check, so the
set the agency can SEE and the set it may WRITE to can never drift. Workspace-guarded at
the anchor row; descendants inherit the workspace (folders.js forbids cross-ws parents).
- routes/agency.js: GET /folders (bound subtree only); POST /content defaults to the bound
folder and 403s any folder_id outside the subtree.
- routes/tokens.js: create auto-creates "Agency — <name>" (or binds a picked folder,
validated same-workspace, respecting the 100-folder cap) inside the token tx; new
PUT /:id/upload-folder to rebind; listing surfaces the bound folder name.
- middleware/apiToken.js + lib/content-ingest.js: upload_folder_id onto req.apiToken; ingest
writes folder_id.
Frontend:
- Agency portal: folder <select> shown only when a real subfolder choice exists (identifies
the "Main folder" root client-side without learning the token's folder id).
- Settings: folder pick at token creation, bound-folder display, rebind modal.
- i18n: 7 new apitoken.* keys across all 5 locales.
Tests (429/429):
- test/agency-folder.test.js: 5 folderSubtree confinement bites (subtree in, siblings out,
workspace guard, null -> root).
- test/agency.test.js (+1 e2e): auto-create, default-to-bound, in-subtree pick lands there,
sibling -> 403, admin-pick, unknown-pick -> 400, rebind-to-root.
Closes#158.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(system-control): Tier 0/1 controls with no device-owner dependency [#160]
Track A of the system-control split (Track B = device owner, shipped in #168). Ships the
capabilities that need NO device owner, with graceful per-tier degradation.
Capability reporting (keystone):
- DeviceInfo now reports can_write_settings / accessibility_enabled / overlay_granted
alongside the existing tier/device_owner flags; server persists them (3 additive device
columns, older APKs default to 0); dashboard gates controls + shows what's grantable.
Android SystemControl (new, all best-effort / no-op when unsupported):
- Tier 0 (no permission): media volume (AudioManager STREAM_MUSIC), per-window brightness
(WindowManager.LayoutParams.screenBrightness — dims our window only).
- Tier 1 (WRITE_SETTINGS): system-wide brightness + screen-off timeout (Settings.System).
- Commands set_volume / set_brightness / set_system_brightness / set_screen_timeout wired
in MainActivity.onCommand; ALLOWED_COMMANDS extended for the group path.
- SetupActivity gains a one-time WRITE_SETTINGS grant row (mirrors the overlay/accessibility
grants); manifest declares WRITE_SETTINGS.
Dashboard:
- device-detail "System control" section (any Android panel): volume + this-app brightness
sliders always; system brightness + sleep-timeout only when the panel reports
can_write_settings, else a "grant on the panel" hint. Sends on release (not drag).
Validated live on a non-owner tier-0 panel: dashboard → set_volume 0.75/0.15 → the panel's
STREAM_MUSIC volume moved to 11/2 (of 15). 423 server tests green.
Closes#160.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(system-control): volume slider reflects real volume + device-owner brightness/timeout [#160]
Two fixes from live testing:
1. Volume "doesn't remember" — the slider hardcoded 50 because the panel never reported its
current volume. Now DeviceInfo reports media_volume (0..1); a new lightweight device:info
socket event lets the panel re-report right after a set_volume (no full re-register / playlist
re-push); server stores devices.media_volume; the dashboard inits the slider from it.
Validated: dashboard set_volume 0.60 -> panel STREAM_MUSIC 2->9 (of 15) -> stored 0.60.
2. System brightness/timeout on a DEVICE OWNER — was gated only on WRITE_SETTINGS, which an
owner doesn't have. A device owner can set those via DevicePolicyManager.setSystemSetting
with no grant, so SystemControl now takes that path when isDeviceOwner(), and the dashboard
enables the Tier-1 controls when can_write_settings OR tier===2. STPolicy.setSystemSetting added.
deviceSocket device_info UPDATE extracted into applyDeviceInfo(), shared by device:register and
device:info. Migration: devices.media_volume REAL (additive). 423 server tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(system-control): brightness/timeout remember + move controls into a tab [#160]
Same "remember what it's set to" treatment as volume, now for brightness + sleep timeout, and
the System control section moves off the top into its own "Controls" tab.
Reporting (DeviceInfo -> device:info re-report -> devices columns -> dashboard slider init):
- system_brightness (read from Settings.System, no permission) + screen_off_timeout_ms.
- window_brightness: persisted in ServerConfig (survives relaunch, re-applied on MainActivity
launch) so the per-window slider reflects it too.
- reportInfoNow() now also fires after set_brightness / set_screen_timeout.
Dashboard: new "Controls" tab (any Android panel) holding the volume/brightness/timeout controls;
every control inits from the reported value; sleep dropdown preselects the current timeout.
Server: +3 additive columns (system_brightness, window_brightness, screen_off_timeout_ms); the
device_info UPDATE stores them. Migrations all additive/re-runnable. 423 server tests green.
Validated live: set_brightness 0.40 -> stored window_brightness 0.40; volume 0.30 -> 0.33.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(group-sync): synchronized playback per group (server + Android) [stage 1]
Play a group's shared playlist in lockstep across its displays — start/end items
together — by reusing the video-wall sync primitive with the spatial transform
removed and keyed on group_id instead of wall_id.
Server:
- device_groups += sync_enabled + leader_device_id (optional pin).
- buildPlaylistPayload emits a group_sync:{group_id, is_leader} block ONLY for a
member whose playlist matches the group's shared playlist (playlist-match guard —
a mismatched member is ignored, never synced). Membership via device_group_members.
- Leader auto-election: pinned-if-online -> first online matching member -> stable
fallback; computed (never persisted, so the operator's pin is preserved).
- group:sync / group:sync-request relay among eligible members (mirrors wall:sync,
guarded on membership + playlist match).
- Self-heal: re-push payloads to group members on (re)connect so is_leader refreshes.
- PUT /api/device-groups/:id accepts sync_enabled + leader_device_id and re-pushes.
Android:
- WallController is now mode-aware. WALL = transform + object-fit:fill + forced
follower-mute (UNCHANGED — every wall branch is byte-equivalent when isGroup=false).
GROUP = same leader/follower timing incl. the full video drift controller, but
full-screen (no transform), normal fit, and per-item mute honored (no forced mute).
- WebSocketService: emitGroupSync/emitGroupSyncRequest + onGroupSync/onGroupSyncRequest.
- MainActivity: dispatch emit by mode, parseGroupConfig, onPlaylistUpdate group hook.
Wall path is provably unchanged (additive mode, defaults to WALL). Kotlin compiles;
server suite 407/407.
Stage 2 (follow-up): web + Tizen parity, dashboard sync toggle + leader picker,
offline-sweep leader promotion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(group-sync): web + Tizen player parity [stage 2]
Mirror the Android group-sync generalization in the two web-based players so a group
with a shared playlist syncs across mixed player types, not just Android.
Web player (server/player/index.html):
- group:sync / group:sync-request handlers (index jump + latency-compensated video
drift, same maths as wall:sync) — no audio policy here, per-item mute honored.
- emitGroupSync() + applyGroupSync() (4Hz leader broadcast; follower sync-request).
NO CSS transform, NO forced mute (unlike applyWallMode).
- isFollower now also true for a group follower (suppresses self-advance).
- handlePlaylistUpdate handles data.group_sync (mutually exclusive with wall).
Tizen (tizen/js/player.js WallController + app.js):
- WallController is mode-aware: WALL styles the slice + forced follower semantics
(UNCHANGED when mode !== 'group'); GROUP clears any wall styling (full-screen) and
drives leader/follower timing only. syncId()/clearStageStyle() helpers; emitSync/
onSync/onSyncRequest key the event + id off the mode.
- app.js: group:sync/request socket handlers; onPlaylist enters group sync on a
group_sync block, else exits — content renders through the normal single-zone path.
Realigned the v4-exit-signal-phase3 TIZEN slice (682->696) shifted by the app.js edits.
Wall path is provably unchanged in both (additive group mode). Server suite 407/407;
both players' JS parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(group-sync): dashboard UI — sync toggle + leader picker [stage 3]
On each group's header row:
- "Sync" checkbox -> PUT /api/groups/:id { sync_enabled } enables synchronized
playback; server re-pushes to members so they enter/exit sync mode. A hint notes
it needs a group playlist and that a display on a different playlist is ignored.
- When on, a "Leader: Auto / <display>" picker -> { leader_device_id } (null = auto-
elect, which self-heals; or pin a specific member to always lead when online).
Adds api.updateGroup(id, data) and the en i18n strings (en-only, mirroring the
existing per-group UI keys; the i18n parity test is apitoken-scoped).
Frontend parses (ESM); server suite 407/407.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(group-sync): rework to clock/schedule sync + double-buffer + polish
Replace the leader/follower relay model with clock/schedule sync. Every
same-playlist member derives the identical (index, position) locally from a
server-disciplined clock + the deterministic playlist schedule, so sync:
- needs no server at play-time (offline-native), and
- has no leader role to double-elect (kills the split-brain class the leaked
WallController tick produced).
Server
- heartbeat-ack now carries server_ms + echoes client_ms for NTP-style clock
discipline; the client caches the offset (survives an outage).
- POST /groups/:id/resync -> group:resync (manual "Resync now").
- (kept: group_sync payload; leader machinery is now vestigial/ignored.)
Clients (web / Tizen / Android)
- Clock offset disciplined over the heartbeat, cached (localStorage / prefs).
- Schedule engine: pos = (syncedNow mod Sum(duration_sec)) with a CANONICAL
slot formula identical across platforms so mixed-platform groups can't drift.
- Snap-on-load: a fresh clip hard-seeks ONCE to the exact position (was ~5s of
gentle nudge to eat a ~0.3s load offset); steady-state keeps the gentle nudge.
- Double buffer: warm the next clip a few s before the boundary -> instant
switch, no black hold. Android pre-decodes on a throwaway surface so the swap
doesn't flash one wrong-aspect (landscape-stretched) frame.
- In-place duration edits: duration_sec dropped from the change signature and
applied in place, so a duration edit re-anchors the schedule WITHOUT a restart.
- Live-log shows discrete corrections (jump/align/seek) immediately; only the
steady-state line is throttled.
Android
- Fix leaked WallController leader tick: onDestroy() now stops it (Handler on the
main looper outlived the Activity -> zombie broadcaster / split-brain).
Dashboard
- Group leader picker -> "Resync now" button.
Tests: heartbeat-ack clock fields + resync route; exit-signal .wgt slice realigned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(dashboard): version indicator + GHCR update check with admin panel
- Add server/lib/ghcr-check.js: GHCR tag poller (36h cache, semver filter)
- Extend /api/version with latest_version and update_available
- Add POST /api/admin/check-update (force GHCR poll)
- Add POST /api/admin/trigger-update (Docker compose or manual instructions)
- Sidebar footer: version label + amber badge when update available
- Admin > System: version comparison card with Check/Update buttons
- 14 new tests (10 unit + 4 integration), 68/68 passing
Closes#163
* fix(dashboard): gate trigger-update to platform-admin + add GHCR fetch timeout
Review follow-up on #165 (the two blockers):
- trigger-update runs `docker compose up -d` on the HOST via docker.sock
(root-equivalent) but was behind requireAdmin, i.e. reachable by any
workspace-level admin. On a multi-tenant host that's a customer, not the infra
operator. Gate it with requirePlatformAdmin (DOCKER_UPDATE_ENABLED still gates
it further). check-update stays requireAdmin — it's a read-only GHCR poll.
- ghcr-check.checkNow had no fetch timeout. Node's global fetch has no default
timeout, so a hung GHCR connection never settled — leaving `inFlight` set
forever (the finally never ran), which wedged the background poller AND hung
any awaited checkNow (/api/admin/check-update). Add a 10s AbortController
timeout on both requests so the try/catch/finally always fire.
All 405 server tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ScreenTinker <hello@screentinker.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets an operator (or an MDM) own updates instead of the app self-installing, which
on managed panels shows a self-install confirm dialog over customer content
(#155). Three layered controls:
- GLOBAL (server): config.otaEnabled from OTA_ENABLED (default on). When off,
/api/update/check returns update_available:false, reason:ota_disabled_global —
the whole instance stops offering updates.
- PER-DEVICE (server + dashboard): new devices.ota_enabled column (default 1). When
0, that device is never offered an update (reason:ota_disabled_device). A
"Self-update (OTA)" toggle in the device settings flips it via PUT /api/devices/:id.
- AUTO-DETECT (Android): UpdateChecker stands down entirely when a foreign device
owner (an MDM/DPC) manages the panel — detected via getActiveAdmins() + not being
device owner ourselves. Pure client-side, errs safe, needs no server change.
The two server gates are enforced server-side so they cover EVERY client version,
not just ones with the client-side stand-down. When OTA is off the device still
reports its version (dashboard sees state); the MDM/operator owns the actual update.
For an MDM-managed fleet (e.g. Pivot/MAXHUB), turn OTA off and let the MDM push the
APK — the install-dialog race disappears from every angle.
Tests: +2 (per-device gate + a real OTA_ENABLED=false server for the global gate);
full server suite 393 pass; Android compiles.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the web player socket reconnects before the device is paired,
register() omitted device_id and device_token (gated behind config.paired).
This caused the server's fingerprint reclaim guard to treat the reconnect
as a fresh anonymous registration with a colliding fingerprint, firing
device:auth-error.
Now device_id and device_token are sent whenever they exist, regardless
of paired status. The pairing code is also reused across reconnects
instead of generating a new random code each time.
Closes#163
Co-authored-by: BlazzzPlay <fabianma7@gmail.com>
The v4-exit-signal-phase3 tests eval a hardcoded LINE RANGE out of tizen/js/app.js
(harness(TIZEN, 663, 697)). The #162 stage-owner fix inserted ~19 lines above that
block, so the slice no longer captured the crash/pagehide handlers and the three B/wgt
tests failed. Re-point the range to 682-716 (block content unchanged, verified identical).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server provisions a unique settings-menu PIN per device, but nothing surfaced
it — leaving the on-device hidden settings menu effectively unopenable. Show the
PIN on the device Info tab (native players only), with i18n across 6 locales.
Also backfill a unique 6-digit PIN for already-paired devices that predate the
settings_pin column, so the existing fleet isn't locked out (delivered on their
next reconnect via the existing device:paired re-send). Idempotent UPDATE.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /:id built the items array but never called schedulesForItem, so the playlist
editor rendered "always plays" for items that have a live schedule. Because the
editor re-PUTs whatever it loaded and PUT .../schedules is a wholesale
DELETE+INSERT, an unchanged save on a mis-loaded item silently wiped the real
schedule. Mirror GET /:id/items:351 so the read path returns the blocks the
editor and player already agree on.
Adds render / round-trip / wipe-trap regression tests (subprocess HTTP harness).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove stray brace that broke compilation (MainActivity line 985)
- Server generates unique 6-digit PIN per device during pairing
- PIN stored in encrypted SharedPreferences (ServerConfig.settingsPin)
- Fallback: generate random PIN locally if server doesn't send one
- Include settings_pin in device:paired on pair + reconnect
- DB migration: settings_pin column on devices table
- Hint changed from hardcoded 0000 to generic 'PIN' string
* fix(preview): replace srcdoc with server-side preview sessions to bypass CSP
Widget previews (clock, weather, etc.) were rendered via iframe.srcdoc,
which inherits the dashboard CSP script-src 'self'. This blocked the inline
scripts widgets need (setInterval for clock, fetch for weather), causing
previews to show blank/static content.
Replace srcdoc with ephemeral server-side preview sessions:
- POST /api/widgets/preview-session — stores rendered HTML (Map, 5min TTL)
- GET /api/widgets/preview-session/:id — serves the HTML via iframe src,
bypassing CSP like the device render endpoint already does
The old /api/widgets/preview endpoint is unchanged for backward compat.
* fix(preview): add rate limiter for /preview-session route
---------
Co-authored-by: BlazzzPlay <fabianma7@gmail.com>
The -patchN scheme (e.g. 1.9.2-patch3) parses as a semver prerelease, so decide()'s
superseded-prerelease guard refused to offer a newer stable core (1.9.3) to the existing fleet —
stranding every 1.9.2-patchN device on OTA (Force Update didn't help; the re-check re-returned
superseded-prerelease). isReleased() now counts -patchN as a shipped release, so those devices get
offered 1.9.3 via normal OTA, while GENUINE prereleases (-beta/-rc/-alpha) keep prerelease semantics
and newer cores are never downgraded. 6 new tests + 14 existing OTA tests green (388/388 suite).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI flaked on the 300k-row prune non-blocking assert: a healthy chunked prune hit a 417ms max
event-loop gap on a shared runner, over the strict 250ms bar (the same test passed on the prior
commit; the release bump changed no logic). These probes exist to catch a MULTI-SECOND freeze (the
pre-fix whole-table sort froze 40-48s) — not to enforce a sub-300ms latency SLA — so a strict bar is
fragile under runner contention/GC. Widen both to <1500ms: still << "seconds" (catches any real
regression) but robust on CI. No production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard CSP (script-src 'self') blocked Cloudflare's Web Analytics beacon. Add the two exact
entries the beacon needs (both required — script-only loads but silently can't report):
- script-src: https://static.cloudflareinsights.com (beacon script loads)
- connect-src: https://cloudflareinsights.com (beacon POSTs analytics back)
Exact domains, no wildcards. connect-src already had 'wss:'/'ws:' (socket.io) + 'https:' — those stay,
so the dashboard socket is unaffected; the explicit CF domain documents intent and survives any future
tightening of the broad 'https:'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Server-side keystone: the server now honors the v4 liveness contract uniformly across the MIXED
fleet (v4 + old pre-v4 + disconnected), all three clients depending on it.
- UNIFORM heartbeat-ack: emitted from the single shared device:heartbeat handler (uniform by
construction; no per-client/per-path branch), BEFORE the auth guard so a known device's watchdog
stays armed. Harmless to old clients (they ignore it).
- RECONNECT-WINDOW ack-gap fix (ackableHeartbeat): ack a KNOWN device (authed socket OR a device_id
that resolves) even mid-reconnect; NOT anonymous/never-authenticated sockets (degrade-safe);
identity-agnostic. No state mutation before requireDeviceAuth (auth surface unchanged; device_ids
are uuidv4).
- DASHBOARD LIVENESS (deriveLiveness): server-derived, VERSION-AGNOSTIC Healthy/Degraded/Offline
from signals every client sends (socket presence, heartbeat age, reconnect frequency); no client
status-push.
- IDENTITY CAPTURE (capture-don't-act): client_type/client_version/platform/contract_version columns;
degrades to legacy/unknown for old clients; NEVER breaks register.
- A-BUCKET FIX (QA): recordReconnect + persistIdentity gated on !isPlaylistRefresh (a ~45-60s refresh
is not a reconnect/new identity — matches #134), and the identity write is change-detected — closing
the WAL write-amplification (A1) and the benign-refresh -> false-"Degraded" (A2) regressions.
New lib/liveness.js (pure helpers, unit-tested). 30 new tests (uniform ack, ack-gap, mixed fleet,
identity capture, cross-client conformance, refresh-gate reproduce-then-prove); 366/366 total.
OTA artifact-availability is a separate concern (out of scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/player v4 — client-only (served web player). Brings the browser player onto the locked v4
liveness contract, IDENTICAL on the wire to the now-v4 APK and .wgt:
- v4 liveness watchdog: consume device:heartbeat-ack, lastServerMessageAt refreshes on ANY inbound
(onAny + engine ping) while ARMING gates on the ack (degrade-safe); 45s±10s jittered threshold;
backoff 1s/30s/±0.2 on the socket.io Manager; NO status/health poll; teardown-before-reopen (#148).
- Browser-specific half-open triggers (the /player-unique part): Page Visibility, sleep/resume
(pageshow/bfcache), network change (online) — all drive the SAME #148 teardown-first reconnect.
- THROTTLE-AWARE: silence computed by timestamp (not timer-fire-count); watchdog does NOT act while
the tab is HIDDEN (throttled-timer gap is expected); on becoming visible, verifyLivenessSoon()
resets the grace and reconnects ONLY if genuinely dead — never spuriously tears down a live socket.
- v4 client identity block on register (client_type=player / client_version / platform / v4).
Verified: arm-after-ack both directions, v4 values MATCH .wgt exactly, browser-trigger no-duplicate-
socket, throttle-aware reproduce-then-prove (real extracted checkLiveness/verifyLivenessSoon),
foreground-recovery one-socket. Depends on the server device:heartbeat-ack (core pass) — degrade-safe
until then.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete+re-pair mints a new device row whose INSERT omits every setting, silently resetting
orientation/name/playlist/etc to defaults (Bold MDM churn). Add a fingerprint-keyed
device_settings table (no FK to devices -> survives the cascade): snapshot on DELETE, auto-
restore on fingerprint-match re-pair (relinking the fp to the new id), operator re-adopt API
(GET /devices/removed + POST /devices/:id/re-adopt) for the changed-fingerprint case. Purge on
workspace/user/org deletion (no cross-tenant bleed). Orientation enum-validated on PUT + restore.
blocked preserved (re-enforced by the register kill-switch). Wall membership deferred (TODO).
Backend only — frontend re-adopt UI NOT built (awaiting API review). Local only, no bump/tag.
Close the disk-fill trap: with wal_autocheckpoint=0 a dead worker means nothing checkpoints.
Controller now respawns an unexpectedly-dead worker (bounded: RespawnMax/RespawnWindowMs +
backoff); on exhaustion it re-arms a conservative inline autocheckpoint (FallbackPages) on the
main connection + reclaims the backlog, logging loudly. Clean stopWalCheckpointer() teardown is
distinguished via a 'stopping' flag so SIGTERM never triggers respawn. Env-gated worker
fault-injection (WAL_CKPT_FAIL_START) for tests. Local only — no bump/tag.
Disable wal_autocheckpoint on the main connection; run PASSIVE checkpoints from a
worker_threads worker with its OWN better-sqlite3 handle, escalating to TRUNCATE on a
size high-water or PASSIVE-starvation. Removes the synchronous fsync-heavy checkpoint
from the event loop. Config: walCheckpointIntervalMs/HighWaterMB/StarvationRuns.
Local only — no bump/tag.
Field-safe SERVER net. A device opening duplicate/rapid sockets (the APK duplicate-socket bug,
separate track) currently churns through evictions during the reconnect-throttle's 30s
post-restart WARM-UP (only the hard ceiling 20 applies then, so an 8-in-9s burst passes
undamped and each new socket evicts the prior). This makes the server absorb it: a thrashing
PAIRED device converges to ONE stable connection and stays online.
- lib/session-settle.js (decision only; bounded, swept): shouldHold(deviceId, incumbentAlive)
— true only when a socket was accepted for this device within SESSION_SETTLE_WINDOW_MS
(config, default 2500ms) AND the incumbent is alive. Warm-up-independent.
- deviceSocket register gate (just before evictPriorSocket): if a LIVE incumbent exists and
we're inside the window, SOFT-REFUSE the new socket (device:throttled reason=session_settle
+ disconnect) and keep the incumbent; else accept + evict + (re)arm the window.
- LIVENESS SAFEGUARD (load-bearing): only hold when the incumbent socket is actually in the
/device namespace — a dead/half-open incumbent is replaced, NEVER stranding the device (max
hold is the 2.5s window from the incumbent's accept, then any new socket is accepted).
- Soft refusal, NEVER a quarantine (reuses patch1's paired-safe philosophy); single-session
enforcement intact for a legitimate move; unpaired/abusive flapping still caught by the
existing limiters. O(1), no loop impact.
Tests (liveness first-class): live incumbent holds + DEAD incumbent replaced (not stranded);
storm of 6 sockets converges to ONE, stays online, not quarantined (during warm-up); single-
session move past the window replaces cleanly; unit decision + bounded sweep. The
evicted-socket-rearm test shrinks its settle window so it still exercises the eviction path.
Suite 336/336.
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
The flap-limiter could 30-min quarantine a PAIRED, legitimate device on reconnect churn.
Behind Bold's single SNAT IP a repeated edge flush -> every device reconnects -> trips flap
-> quarantined -> a recoverable blip becomes a SUSTAINED FLEET-WIDE LOCKOUT we caused.
check(key, now, {paired}) now skips (and clears) the quarantine escalation for a paired
device — it still gets the brief soft cooldown if it truly hammers, but never the long
lockout. The register gate computes paired = device_id && validateDeviceToken(...) (a
matching STORED token, false for missing/mismatch) so a spoofed device_id can't claim the
exemption; unpaired/anon flapping (attacker / unprovisioned hammering) still quarantines.
Tests: unpaired flapper still quarantined; paired never quarantined (soft cooldown only);
paired creds RELEASE an in-flight quarantine; N paired devices from one SNAT IP all admitted
on reconnect and never quarantined across repeated flush cycles.
Live error: "Uncaught (in promise) TypeError: PlayerMediaHealth.shouldShowIdle is not a
function" inside the device:paired socket handler.
WHAT WAS ACTUALLY WRONG: not a missing/misnamed definition — the module DOES export
shouldShowIdle (served + unit-tested). It's a VERSION SKEW: /player/* is served network-first
by the service worker, so a transient module-fetch failure falls back to the STALE cache (an
older player-media-health.js that predates shouldShowIdle) while index.html loads fresh with
the call. window.PlayerMediaHealth then exists but lacks the method, and the call site
guarded the OBJECT (`window.PlayerMediaHealth ? ...`) not the METHOD — so it threw, aborting
the rest of the device:paired handler (the showStatus after it was skipped).
FIX: guard the METHOD at both call sites (typeof X.method === 'function') so a stale/partial
module can never throw — it falls back to the safe inline default (!isPlaying for the idle
decision) and the handler runs to completion.
SIBLING (errors travel in pairs): the needsReattach call in the "Playlist unchanged" branch
had the SAME object-not-method guard. It was inside a try/catch so it couldn't throw uncaught,
but a stale module would silently skip the re-attach/hideStatus. Guarded it the same way.
No service-worker change needed: network-first already self-heals on the next good load; the
method-guard covers the transient/offline-fallback skew permanently.
Tests: player-media-health.test.js +1 module-surface test (both needsReattach and
shouldShowIdle are exported functions — catches the define-vs-call class). Inline player JS
syntax-checked; both guards verified present. Suite 319/319.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONFIRMED from a live console capture (restore cache -> video plays -> reconnect ->
"Playlist unchanged" -> screen falls to "Waiting for content..."). Audio survived because
only the "showing content" VIEW was covered, not the audio path.
ROOT CAUSE: the server re-emits device:paired on EVERY re-register of an already-paired
device (ws/deviceSocket.js:510) — i.e. on every reconnect, while content is already playing.
The player's device:paired handler called showStatus('Waiting for content...') UNCONDITIONALLY
(the "falls through to idle" sibling), putting the idle overlay OVER the live video. The
following device:playlist-update -> "Playlist unchanged" branch returned early and never
cleared it, so the idle screen stuck on top of playing content.
FIX (idle screen only when genuinely idle; unchanged is a strict no-op that keeps playback):
- lib/player-media-health.js: new shouldShowIdle(state) — idle ONLY when nothing is playing
AND there's genuinely no content. Already-playing (or content-present-about-to-render) is
never idle.
- device:paired handler: gate showStatus on shouldShowIdle({isPlaying, hasContent}) instead
of showing it unconditionally. On a reconnect while playing -> no-op.
- "Playlist unchanged" branch: when healthy playback is confirmed, hideStatus() to clear any
stale idle overlay a reconnect's device:paired may have put up — so the confirmation can
never leave "Waiting for content..." over live content. Still leaves the actual media
element exactly as-is (no teardown, no flicker).
- sw.js cache v10 -> v11.
SIBLING SCAN: device:paired was the only unconditional idle reset. The connect() idle
prompts (connecting / connecting_muted) were already guarded by !isPlaying; empty-playlist
and no-renderable idles are genuine.
Tests: player-media-health.test.js +2 (shouldShowIdle: playing never idle; idle only when
empty+not-playing). Inline player JS syntax-checked; module served + guard referenced on a
booted server. Suite 318/318.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT CAUSE (hypothesis A, pre-existing — NOT a beta7 regression; server/player/index.html
is untouched since v1.9.2-beta6): handlePlaylistUpdate's "Playlist unchanged" branch blindly
returned. The media re-attach (renderContent) lives ONLY in the content-changed branch, so
if the <video> surface was lost (element detached from the DOM while still decoding — video
gone, audio still playing) a no-new-content refresh never re-attached it. New-content
refreshes were fine because they re-render.
FIX (make the refresh idempotent for the media surface, no flicker on the healthy path):
- server/lib/player-media-health.js (new, UMD + unit-testable, mirrors schedule-eval.js):
needsReattach(state) — re-attach ONLY when playback should be happening but the current
item's surface is actually lost (video null / detached / ended / errored; non-video: no
mounted surface). A healthy attached+live video returns false, so a routine poll stays a
no-op (no re-render, no flicker). Served at /player/player-media-health.js from the single
source; loaded by the player.
- index.html no-change branch: extract the current item's DOM facts and, iff
PlayerMediaHealth.needsReattach, call playCurrentItem() to re-render the current item.
Wrapped so the health check can never break a refresh.
- teardownCurrentMedia: also release currentVideoEl even when it was DETACHED from the
container — a detached-but-playing <video> keeps emitting audio and the container-scoped
querySelectorAll can't find it. This kills the "ghost audio" on re-attach.
- sw.js cache bumped v9 -> v10 so players pick up the new index.html + module.
Tests: test/player-media-health.test.js (6) exercises the branch selection — healthy video
-> no re-attach; detached/null/ended/errored -> re-attach; idle -> never; non-video by
surface presence. Inline player JS syntax-checked; module served + referenced verified on a
booted server. Suite 316/316.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The billing:read scope + dual-path gate were built but there was no way to MINT a token
(and it must NOT go in the workspace-scoped, self-service API-Tokens UI). Adds a server-side,
owner-only CLI — no new UI, no network endpoint. Owner-only BY CONSTRUCTION: it's a
host-side script, so filesystem/shell access = the platform owner.
- server/lib/billing-token.js (testable): mintBillingToken/revokeBillingToken/
listBillingTokens. Reuses the EXACT existing token path — same secret (st_ + 32 bytes
base64url), same SHA-256 hashing (hashToken), same api_tokens columns — no second format.
Resolves the platform OWNER (oldest platform_admin/superadmin; #14 collapsed superadmin ->
platform_admin so that's the top tier) and binds to their workspace. api_tokens.user_id +
workspace_id are BOTH NOT NULL (no platform-level token exists); the workspace binding is
VESTIGIAL for billing (billing:read is off-ladder -> can't reach any workspace router;
billing is platform-global), documented in-file rather than loosening NOT NULL pre-release.
- scripts/mint-billing-token.js: thin CLI wrapper. --name mints and prints the secret ONCE
(+ id, + "run as owner on host" warning), --list, --revoke <id> (soft revoke, mirrors the
dashboard DELETE).
Tests (4, test/billing-token-mint.test.js): minted row is scope EXACTLY billing:read with a
matching SHA-256 hash and no read/write/full/agency scope; the token reads GET
/api/billing/usage (200) but is refused on /api/devices (403) and /api/admin (401) — scope
isolation; revocation -> 401; mint requires a name; revoke refuses a non-billing id. CLI
smoked live (mint/list/revoke). Suite 310/310.
SPEC-vs-REALITY (again): spec said bcrypt + JSON `scopes`; this codebase uses SHA-256 + a
single `scope` TEXT column. Built to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Least-privilege way to read GET /api/billing/usage without requiring platform admin.
Additive + isolated: reuses the existing api_tokens scope system (the off-ladder 'agency'
scope is the precedent) and does NOT touch the shared role/permission checks other
endpoints rely on.
- New off-ladder scope 'billing:read' (routes/tokens.js SCOPES). Like 'agency' it is NOT
on the read<write<full ladder, so tokenScopeGate rejects a billing token on every
PUBLIC_ROUTER and JWT-only routers reject any st_ token -> the scope grants billing-read
and NOTHING else.
- DUAL-PATH gate requireBillingRead (middleware/apiToken.js), written as an EXPLICIT OR:
authorize if (billing:read token) OR (platform-admin session). Admins keep read access
but are NOT required to; the token path doesn't lock out admins or vice versa. Billing
route now mounted with bearerAuth (token OR JWT front door) + requireBillingRead (was
requireAuth + requirePlatformAdmin).
- MINTING is platform-admin only (stricter than read/write/full/agency, which any
workspace member may mint) since a billing:read token grants GLOBAL billing-read. Note:
no finer "owner" tier exists here (#14 collapsed superadmin->platform_admin), so
PLATFORM_ROLES is the top level required.
Tests (5, test/billing-authz.test.js): dual-path positive (token AND admin session both
200) + negative (user 403 / anon 401); scope isolation (billing token 403 on /api/devices,
401 on /api/admin; read token 200 on devices but 403 on billing); minting owner-only
(user + ordinary-admin 403, platform-admin 201); revocation -> 401. Existing token
firewall/partition suite (api.test.js) + billing-endpoint tests unchanged & green. Reused
the exact SHA-256 token-verification path (no bcrypt/new mechanism). Suite 306/306.
NOTE: spec described bcrypt + JSON `scopes` + an analytics:read precedent; this codebase
actually uses SHA-256 + a single `scope` TEXT column + 'agency' as the off-ladder
precedent. Implemented faithfully to the real system.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The storm harness asserted a tick COUNT (environment-timing-sensitive) that could flake
in CI. Kept the real invariant (max event-loop gap < 300ms) and loosened the tick
assertion to >=2 (enough to have measured a real gap), which cannot flake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the new internal states so we can SEE the limiters biting during the alpha soak
instead of grepping logs. /api/status now carries debug: {
flap: {buckets, quarantined},
ota_download: {inFlight, servedThisWindow, shedThisWindow, windowCount},
maintenance: {deleted, ms, at, running}, // last status-log prune
log_coalescer_buffer,
}. Aggregate counts only (no device ids/secrets), cheap in-memory reads. stats()
added to flap-limiter + ota-download-guard (singleton prod state), getMaintenanceStats
from database. Asserted in the booted /api/status test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coalesced loop-lag summary carried an arbitrary sample's p99. Now record() tracks
the MAX (peak) over the window and the summary emits it — the peak is the number that
matters during an incident: '[loop-lag] band=critical (x47 in 30s, peak 1502ms)'. Band
CHANGES still log immediately, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Booting against a pre-bloated 300k-row device_status_log, /api/status answers in <3s
while the table is still large (chunked startup prune trickling in the background), and
the backlog drains to the cap with the server responsive throughout. The old whole-table
sort froze boot ~40s. Fallout doc gets the P2 findings section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigated the fleet-wide simultaneous reconnect concern. buildPlaylistPayload is
synchronous but cheap: measured avg 0.078ms / max 0.70ms per call with a 200-item
snapshot; 230 devices = ~18ms CPU spread across 230 separate handler invocations (the
loop yields between them), never one block. Per-call << 50ms invariant. No change
needed; documented with the measured number.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block is a real lever now, so cover the authz path. Booted server + JWT; device and a
viewer membership seeded via the DB file. Proves POST /api/devices/:id/{block,unblock}:
owner (workspace_admin) -> 200; unauthenticated -> 401; cross-workspace user -> 403;
workspace_viewer -> 403 (read-only), with the DB blocked column asserted at each step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fixed OTA_DOWNLOAD_MAX_CONCURRENT=10 throttled a legitimate coordinated rollout even
on a perfectly healthy server, and a shed 503 costs a client a full ~30-min re-check
cycle. Made the download guard's concurrency + rate caps band-aware:
- normal -> serve FREELY (no cap): a whole-fleet rollout isn't staggered when healthy
- elevated -> the configured caps engage (early backpressure)
- critical -> shed 503 (the real protection, unchanged)
Kill switch OTA_DOWNLOAD_GUARD_ENABLED=false disables it entirely.
Tests updated: normal serves 50/50 with 0 shed; caps + shed now asserted under elevated;
storm harness OTA flood runs under elevated (the loaded state). Suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolveIdentity runs on every register (block + flap gates). It already returned on
device_id before any DB access; memoized the device_fingerprints statement (prepared
once, lazily) and documented the invariant. Test asserts a device_id-present resolve
prepares/runs ZERO device_fingerprints queries; a device_id-absent resolve does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flap limiter's auto-quarantine used to run `UPDATE devices SET blocked = 1` — a
PERMANENT, human-cleared block on an automatic trigger. A stuck-then-recovered device
stayed dark until someone noticed.
- Removed the auto-write from ws/deviceSocket.js. devices.blocked is now written ONLY by
an operator (dashboard endpoint / direct SQLite).
- Quarantine moved into lib/flap-limiter.js as IN-MEMORY, TIME-LIMITED state: after
connectRateQuarantineTrips trips in a window the identity is quarantinedUntil = now +
connectRateQuarantineMs (new, default 30m); check() then refuses cheaply with
reason:'quarantined' and AUTO-CLEARS when the window passes. Safe in-memory now that
Item A ended the restart loop, and a self-healing auto-action must not survive as a DB row.
- Log quarantine START once; repeat refusals go through the coalescer. Stale
"-> blocked=1" comments updated.
- connectRateQuarantineTrips=0 still disables it.
Tests: quarantine engages after N trips, refuses cheaply during the window, auto-clears
after connectRateQuarantineMs; and an integration flapper is quarantined while
devices.blocked stays 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Combines all four spiral inputs at once: a pre-bloated device_status_log (300k rows), a
maintenance sweep running over it (the old whole-table sort froze 40-48s), a hard
flapper, and an OTA download flood from a single SNAT IP. Asserts:
- the event loop never enters a multi-second freeze (max 10ms-ticker gap < 300ms,
vs 40-48s pre-fix) and the ticker keeps firing throughout,
- the chunked sweep drains the 300k backlog to the per-device cap,
- every limiter still bites under load: the flapper is refused, the OTA flood is shed
past the global window cap (served capped, never per-IP).
Suite 267/267.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Don't let telemetry/logging cook the loop under a storm.
- lib/log-coalescer.js: dedup+count high-frequency lines, flush ONE summarized line per
key per window ("[loop-lag] band=critical (x47 in 30s)"). Bounded buffer (auto-flush
at MAX_KEYS). Applied to the loop-lag "still loaded" line (band CHANGES stay immediate),
the per-request OTA check line, and "Device reconnected".
- loop-lag: event_loop_lag rows are BUFFERED and batch-inserted on a flush interval
(was a synchronous INSERT per sample); the buffer is bounded (drop-oldest). Its
retention prune now rides the Item-A chunkedDelete so this table can never repeat the
status_log bloat-then-freeze. /api/status still reads in-memory current (real-time
band unaffected).
- Bounded the previously un-evicted per-device Maps: content-ack limiter gets an idle
sweep (started in server.js); status-log-writer.lastWritten is capped (drop-oldest;
it only suppresses a redundant consecutive row, so eviction is safe).
Tests: N identical lines -> one counted line; single line verbatim; coalescer buffer
bounded under a distinct-key flood; content-ack Map swept of idle buckets.
loop-lag-integration updated for the batched-insert cadence. Suite 266/266.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Enforcement (deviceSocket): resolve identity ONCE via the SNAT-safe chain and check
blocked against the RESOLVED device_id (device_id directly OR fingerprint->device_id),
so a blocked device that reconnects WITHOUT a device_id is still caught — the old
"if (device_id)" gate let a device_id-less reconnect slip past. Still the first gate,
before flap/throttle/DB/playlist. Nulling the token still does NOT block (it
re-provisions) — the blocked column is the lever.
- Dashboard toggle: POST /api/devices/:id/{block,unblock} (write-gated + workspace-scoped
via checkDeviceOwnership) writes devices.blocked; takes effect on the device's NEXT
register with no restart. api.js + a Block/Unblock button in device-detail.js.
- Outage procedure documented in-code: direct SQLite
"UPDATE devices SET blocked = 1 WHERE id = <id>" works with the dashboard down.
Tests: blocked refused at handshake with no playlist build; device_id-less reconnect
with a mapped fingerprint still refused; unblock effective on next register, no restart.
Suite 262/262.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fleet SNATs to one IP, so nothing on the OTA path may key on IP.
- /api/update/check: EARLY-RETURN before any filesystem call when the breaker won't
offer (rate-backoff / up-to-date / phantom / client-newer). A looping client that
gets rate-backoff now does ZERO fs — the flood can't become a statSync flood.
- lib/apk-cache.js: resolve APK path/size/mtime once at boot + refresh on an interval;
the check/download endpoints read cached metadata (get() does no fs, proven by test).
- lib/ota-download-guard.js + /download/apk: GLOBAL concurrency + rate caps + critical-
band shed (503 Retry-After), NEVER per-IP. Replaces the per-IP-per-10min log throttle
(which hid the flood under SNAT) with a per-window served/shed aggregate so a download
flood is VISIBLE. Bounded single rolling-state object; in-flight released on finish/close.
- Breaker unchanged; no IP limiting or device_id requirement added (legacy field clients
send no device_id on OTA checks — must keep working).
Tests: apk-cache get() = 0 statSync over 1000 reads; download guard sheds past global
concurrency + per-window rate + critical band; admit() has no IP parameter. Suite 259/259.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #142 burst throttle (5/10s) misses a device flapping every 3-5s (~2-3/10s) — yet
each cycle is an expensive register+build+acks and one status_log row (the spiral
trigger). Now that Item A ends the restart loop that used to wipe in-memory throttle
state every ~40s, an in-memory sustained limiter can finally bite.
- lib/device-identity.js: SNAT-safe identity resolution — device_id -> fingerprint
(map via device_fingerprints -> device_id, else raw fp) -> device_token -> ONE bounded
global anon bucket. NEVER IP (the fleet SNATs to 10.10.10.1). An unidentifiable client
is still bucketed (collectively) so an anon flood is capped, never unthrottled.
- lib/flap-limiter.js: per-identity connect-frequency over a long window
(CONNECT_RATE_WINDOW_MS=5min, CONNECT_RATE_MAX=20; anon bucket cap 60). Over the rate
-> refuse + disconnect (cheap). Bounded by an idle sweep (anon bucket never swept).
Optional auto-quarantine: a device_id-resolved hard flapper -> blocked=1.
- Wired at the device:register gate BEFORE fingerprint tracking/throttle/DB/build,
skipping same-socket playlist refreshes. Sweep started in server.js.
Tests: 4s-flapper refused after the window max; 60s-normal never; two device_ids
independent (never IP); device_id-less bucketed by fingerprint; neither id nor
fingerprint capped via global anon; idle sweep preserves the anon bucket. Suite 254/254.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found in the alpha load test: client-chosen pairing codes collide by birthday
paradox, the provisioning INSERT hit UNIQUE(devices.pairing_code), the SqliteError
threw out of the (synchronous) socket handler -> uncaughtException -> logFatalAndExit
-> the WHOLE server exited and every device dropped. The colliding flood crash-LOOPED
the container (2 restarts).
Two layers, same "one device can't take down the fleet" theme as #142/#143/#144:
1. Narrow (deviceSocket.js): wrap the device:register provisioning INSERT in
try/catch — a UNIQUE pairing_code collision (or ANY db error) rejects THAT
registration (device:auth-error -> client retries) instead of throwing.
currentDeviceId/authenticated now set only AFTER the row exists (no half-auth
socket on failure).
2. Broader (lib/safe-socket.js): protectSocket() overrides socket.on per connection
so any handler throw is caught, logged (event + id + stack), the socket told, and
DISCONNECTED — per-CONNECTION fail-fast, not whole-PROCESS. We don't keep serving a
connection from possibly-half-mutated state (honors the existing fail-fast intent),
we just contain it to "one device reconnects" (a non-event after beta5). Wired into
both the /device and /dashboard connection handlers; auto-covers future handlers.
Audited first: no handler throws as control flow, so blanket-wrapping is safe.
Tests (mutation-verified, fail without their fix):
- register-insert-crash.test.js: a pairing_code collision AND a general bind error
each reject-one-device with no uncaughtException; server keeps serving.
- socket-handler-isolation.test.js: a throwing handler disconnects only that socket;
the server + other sockets stay alive.
Full suite 243/243.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/api/update/check offered the update whenever client !== latest (raw string
inequality, not semver) with no backoff. A device that can't APPLY the update
(broken OTA client 1.7.12, signing/Fire OS) keeps reporting the same version and is
told update_available=true on every poll; a fast poll loop saturates the event loop
(prod loop-lag 49s). All requests share one NAT IP, so IP-keying is useless.
server-only breaker (lib/ota-breaker.js), two independent axes:
- RATE breaker (primary, immediate): a key checking >THRESHOLD (3) times within
WINDOW (60s) is looping -> throttle update_available with exponential backoff
(30s->2m->8m->cap 30m). Healthy devices poll ~12 min and never approach this, so
rollout/stragglers are inherently safe -- NO grace-for-flood timer; slow == safe.
- PHANTOM guard (immediate): unrecognized version, or a prerelease of an OLDER core
(superseded old-minor beta e.g. 1.9.1-beta4), gets no-offer on the first check. A
RECENT real older version (beta3 vs latest beta4; stable 1.7.12) stays offerable.
- Never offers a downgrade (client >= latest -> no offer).
KEYING (#144 option 3): keyed on device_id when present, else reported version.
- server.js:581 accepts + logs ?device_id=, passes it to the breaker.
- UpdateChecker.kt:122 appends &device_id=<config.deviceId> (existing registered id;
omitted until provisioned). One-line client change.
beta4+ clients get precise per-device throttling; stuck legacy clients sending only
?version= are caught by the version-keyed + rate + phantom logic. Response gains
additive `reason` + `retry_after_seconds` (old clients ignore).
BOUNDED STATE: a periodic sweep (startSweep, wired in server.js) evicts buckets idle
> IDLE_RESET_MS so the keyed Map can't grow unbounded (churned device_ids); not
reset-on-access only.
SCOPE (deliberate): this targets the FAST flood + phantoms. The slow #144 drip
(stable 1.7.12 polling ~every 12 min, ~20/hr) stays below >3/60s and is NOT
throttled -- catching it needs #144 option-3 "skip-this-version after N cycles",
which is intentionally NOT in this build.
NOTE: carries a CLIENT/APK change -> versionCode must increment at the beta4 bump and
the release keystore is required for the APK. The device_id path only helps devices
that can install beta4+; the stuck legacy fleet is covered by the version-keyed path.
Tests: unit (lib/ota-breaker, injected time) a-f + comparator + escalation + sweep +
slow-drip-scope; HTTP integration (real endpoint, device_id passthrough). Full suite
green serial AND parallel (234). OTA-only delta -- reconnect/reclaim/shed/content-ack/
block untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bold: screens sit on the Connect page showing the server URL = paired server-side
but never told, so the app never starts playing.
Flow / gap (Step A):
- CLIENT leaves the Connect page ONLY on the 'device:paired' event — web player
(player/index.html) hides the setup screen; Android ProvisioningActivity.onPaired
launches MainActivity + finish(). That event is the sole signal.
- SERVER pushes 'device:paired' to the device's room from POST /api/provision/pair
(server.js) at pair time — but ONLY reaches a LIVE socket then. The normal
device_id reconnect path emitted device:registered + device:playlist-update but
NOT device:paired. So a screen paired while disconnected, or that reconnects after
pairing (exactly the screens cycling on the Connect page), is paired server-side
(user_id set, receiving playlists) yet never gets device:paired -> stuck on Connect.
Fix (server-only, uses the EXISTING client listener — no client update needed, which
matters because we can't push a client update to stuck screens): on the device_id
reconnect, if the device is paired (user_id set), re-emit 'device:paired'
{device_id, name}. Push-on-pair (server.js) already covers the live-at-pair-time
case; this covers paired-then-reconnect. A paired screen now leaves Connect and
plays on its next reconnect with no client change and no manual re-pair.
Tests (port 3989, real flow): provision -> pair via /api/provision/pair (socket
closed) -> reconnect RECEIVES device:paired (+name +playlist) — the stuck-screen
repro; an unpaired device gets NO device:paired (stays on the pairing flow); the fix
reuses the existing device:paired event (no new protocol). Full suite green serial
AND parallel (220); dbac699 / 404c330 / e734281 intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bold beta1: three devices spam "Fingerprint reclaim rejected ... device active
(status=offline, ~2500s since heartbeat, liveConn=false)" twice/~2s indefinitely —
contradictory: gone by every signal yet treated as active.
Root cause (NOT a missing clear — corrected the hypothesis). The reject condition
was `liveConn || status==='online' || secondsSince < RECLAIM_GRACE_SECONDS(24h)`.
For the observed devices liveConn=false and status=offline, so the ONLY true term
is `secondsSince < 24h` — an effective 24h CALENDAR grace, not a stale flag. Audited
the clears: liveConn (deviceConnections) is removed on the debounced disconnect
(heartbeat.removeConnection) AND the offline_timeout sweep (deviceConnections.delete);
status is set 'offline' on both. liveConn=false + status=offline PROVE the clears
ran — there is nothing stale to clear. The 24h time gate (mislabeled "device active")
blocked a legitimately-gone device from reclaiming for up to 24h, so it retried
every ~2s forever-in-practice. The "twice per ~2s" is two reclaim ATTEMPTS per cycle
(client reconnect + re-pair-on-auth-error), each hitting the single console.warn —
not double-logging in one attempt.
Fix:
- Decide "still alive" from RUNTIME signals: `!!liveConn || secondsSince <
reclaimSettleSeconds`. A device with no live socket and a heartbeat older than the
settle window is gone -> reclaimable. A live (or just-seen) device is still
rejected, so reclaim-abuse protection holds. NOT just ignoring "active" — it fixes
WHY it was stuck (the 24h gate). RECLAIM_SETTLE_SECONDS default 300 (was 24h).
SECURITY TRADEOFF flagged in config: shortens the anti-fingerprint-theft window;
raise to re-tighten. Tuning guess to validate vs Bold.
- Log throttle: the deferral logs at most once per device per RECLAIM_REJECT_LOG_
WINDOW_MS (default 60s) — collapses the double-log + the per-2s flood (same
discipline as the content-ack shed log). Cleared when a reclaim proceeds.
Recovery of the 3 wedged devices (2febcaa9, 1984694c, 139159eb): they SELF-HEAL on
their next reclaim attempt (~2s) once this ships — their heartbeats are ~2500s stale
(>300s settle) and liveConn=false, so the reclaim now succeeds. No operator SQL needed.
Tests (port 3988): gone device reclaims; live device still rejected; clear-on-leave
(disconnect clears liveConn -> stale device reclaims); deferral log <=1 per window.
Full suite green serial+parallel (217). reconnect-throttle.js, the dbac699 content-ack
limiter, and the 404c330 block/auth code untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Highest-priority #143 item (operator finding from Bold): nulling a device's token
did NOT lock it out — device 75c2a08a immediately reconnected and saturated the
loop. Two distinct defects:
1. Auth short-circuit (the cause). device:register used
if (device.device_token && !validateDeviceToken(...)) { reject }
so a NULL/empty STORED token made the guard falsy -> validation SKIPPED, and the
next block even MINTED a fresh token and persisted it. Nulling a token thus
RE-PROVISIONED the device instead of locking it out. Fix: drop the
`device.device_token &&` guard -> `if (!validateDeviceToken(device_id, device_token))`
(validateDeviceToken already returns false for null-stored/missing/mismatch), and
remove the legacy "mint a token for a null-token device" path (the re-provision
vector). An already-provisioned device (every row, incl. 'provisioning', is created
WITH a token) presenting null/empty/invalid is now REJECTED + disconnected.
The first-pairing seam is unaffected: a brand-new device has NO device_id and goes
through the pairing_code branch (which mints id+token) — a different code path.
2. No server-side kill switch. Added a `blocked` column (devices.blocked INTEGER
NOT NULL DEFAULT 0; schema.sql + a database.js migration). The block is the FIRST
gate at the top of device:register — before the fingerprint block, the reconnect
throttle, any DB writes, or playlist build — so a blocked device's socket is
refused immediately (auth-error 'Device blocked' + disconnect, zero further work).
It does NOT rely on null-token (the thing that failed). The row is re-read every
register, so a DIRECT SQLite edit takes effect on the device's NEXT reconnect with
NO server restart. Operator statements (dashboard-down, hand-edit):
block: UPDATE devices SET blocked = 1 WHERE id = '<device_id>';
unblock: UPDATE devices SET blocked = 0 WHERE id = '<device_id>';
Tests (port 3987): nulled-token provisioned device is REJECTED (75c2a08a repro);
blocked=1 refused at the first gate (no register/playlist); unblock reconnects;
first-pairing still works; normal valid-token device unaffected. Full suite green
serial AND parallel (213); reconnect-throttle.js + the dbac699 content-ack limiter
untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#142's content-ack dedup is insufficient: a device cycling 2-4 content IDs makes
every ack look unique so dedup never fires, while aggregate volume from ~30 devices
saturates the event loop (the #142 reconnect throttle kept the server responsive,
which is how this was even observable).
Folded ONE control on the content-ack path (no competing limiters; reconnect-
throttle.js untouched) in lib/content-ack-limiter.js:
- Step 1 — per-device RATE budget: caps TOTAL non-duplicate acks per device per
window regardless of differing content_id (the case dedup misses). Over budget =
DROP silently (the per-ack log+emit is the cost); log ONCE per device per window
when shedding starts. Keeps the #142 dedup (dedup'd repeats don't consume budget).
Per-device, in-memory, resets on restart (modeled on lastPlayLogAt; does NOT reuse
reconnect-throttle's ban-semantics bucket).
Env (TUNING GUESSES, validate vs Bold's fleet): CONTENT_ACK_MAX_PER_WINDOW=20,
CONTENT_ACK_RATE_WINDOW_MS=10000 (=2/s, above legit ~<=1/s, below the flood).
- Step 2 — global pressure valve: reuses the #142 loop-lag band (+ its hysteresis,
no second control loop). Under CRITICAL band, shed content-acks even for an
in-budget device; reconnects + dashboard/HTTP are ALWAYS processed; a healthy
device in a non-critical band is never touched by the valve. Valve open/close
logged once at the band edge in services/loop-lag.js (not per shed message).
Tests (unique ports 3985/3986, not the 3982/3983/3984 set):
- unit: the #143 regression (cycling ids evading dedup IS rate-limited), under/over
budget, dedup still works + doesn't consume budget, valve sheds in-budget under
critical while normal is untouched, rate precedence, window reset, per-device
isolation.
- integration: socket flood is capped to budget with a single shed-start log;
under-budget passes every ack; valve OPEN sheds content-acks while a reconnect +
/api/status still succeed.
Full suite green serial AND parallel (208 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the full #142 stack onto main on top of the 1.9.1 stable cut:
- device_status_log index + de-dupe
- event-loop lag telemetry (bounded)
- load-aware per-device reconnect throttle (the outage fix)
- global device_status_log retention sweep (STATUS_LOG_RETENTION_DAYS)
- content-ack dedup
- provisioning-row cleanup window 365d -> 24h
services/heartbeat.js deleted unclaimed provisioning devices with
created_at < now - (365 * 86400) — a YEAR — while its own comment said "older
than 24 hours". So socket-register pairing junk lingered ~365x longer than
intended. Change the window to 24 * 3600 to match the comment.
Correctness fix only — does NOT touch the pre-auth register path or add a rate
limiter (that pre-auth hardening is a separate security issue, out of this cut).
Extracted the sweep into pruneProvisioningDevices() (still in heartbeat.js, called
from the same interval) so it is unit-testable. Test asserts a >24h unclaimed
provisioning row is swept while a <24h row, an imported row (user_id set), and a
non-provisioning row are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
device:content-ack logged + emitted every message, so a device repeatedly
reporting the same "content <id>: ready" (observed from an older app version)
added avoidable load per message.
- Suppress identical (device_id, content_id, status) reports within
config.contentAckDedupMs (default 10s), modeled on the lastPlayLogAt throttle.
A status change has a different key and passes immediately; a fresh report after
the window passes too. In-memory, resets on restart. The handler does no DB
writes, so this is purely shedding redundant log+emit work.
test: integration over a real authenticated device socket — a burst of identical
"ready" collapses to one log/emit, a "ready" after the window passes, and a status
change is never deduped. Unique PORT (3984).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-device insert-time prune (deviceSocket.js) only ever touches a device
that is actively inserting, so it misses two paths: removed/idle devices whose
rows linger forever, and heartbeat.js's offline_timeout insert that bypasses
logDeviceStatus entirely. The reporter's 1.2M-row bloat accumulated UNDER a 7-day
per-device prune for exactly this reason.
- pruneStatusLog() (db/database.js): a GLOBAL time-range sweep across ALL devices,
modeled on the play_logs prune. Run once on startup (recovers a bloated table
right after deploy) and on the heartbeat interval (services/heartbeat.js).
- STATUS_LOG_RETENTION_DAYS env, default 3 (lower than the old hardcoded 7d; the
dashboard only shows a 24h uptime window, so 2-3d is ample for diagnostics).
- Deliberately NO per-device row cap: Step 3's throttle already bounds how fast a
storming device can generate status rows, so a cap would add sweep complexity
for little gain (noted for later if needed).
- NO VACUUM / auto_vacuum here (kept off the hot path); space reclaim is left as a
separate decision (see report).
test: deterministic in-process unit test proves the sweep deletes over-retention
rows across all devices — including a device absent from the devices table and an
offline_timeout row — while keeping recent rows; idempotent on an empty table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gates genuine reconnects PER DEVICE before the heavy register work (DB writes +
playlist build) runs, so a single flapping device can no longer saturate the
event loop and take down the server.
- Actuator is per-device, keyed on device_id (modeled on lastPlayLogAt). A device
is flagged only when it exceeds reconnectBaseMax genuine reconnects per window.
Same-socket playlist refreshes (isPlaylistRefresh) are exempt.
- Load-awareness is BANDED (normal/elevated/critical from the step-2 lag signal),
not a continuous controller. The band only MULTIPLIES an already-flagged
device's backoff; global lag never gates a healthy device.
- Hysteresis: escalate immediately while storming (tighten fast); decay one level
per reconnectReleaseMs of calm (release slow).
- HARD CEILING per device, independent of band and warm-up — a slow-ramp attacker
can't train through it.
- COLD START: for reconnectWarmupMs after boot, force the normal band and apply
only the hard ceiling, so a full-fleet reconnect after a deploy doesn't throttle
healthy screens. State is in-memory, resets on restart.
- Observability: every throttle engagement logs device, band, observed vs allowed
rate, and backoff. Throttled device gets device:throttled + a deferred disconnect.
Tests (api.test.js style):
- unit: healthy-never-throttled, storm-throttled-with-growing-backoff, band
multiplies backoff, hard-ceiling-even-in-warmup, warm-up leniency, neighbor
isolation, slow release.
- integration GATE (the required one): full-fleet reconnect right after restart
throttles NO healthy device; a single device storming IS throttled; a neighbor
stays unaffected while another storms.
- also fixes pre-existing test PORT collisions (my new integration files clashed
with totp.test.js:3979 and totp-keyrotation.test.js:3980 -> moved to 3982/3983);
full suite now green serially AND in parallel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuously samples event-loop delay via perf_hooks.monitorEventLoopDelay()
(C++-backed histogram; cheap). Each window persists mean/p50/p99/max to a new
event_loop_lag table and recomputes a coarse load band (normal/elevated/critical)
from the window p99. Standalone value: current lag is exposed on /api/status and
band changes are logged, so site lag is diagnosable independent of throttling.
The band feeds the #142 reconnect throttle (next commit) but ships first as its
own subsystem.
- event_loop_lag is bounded from day one: indexed on sampled_at + scheduled prune
(LAG_TELEMETRY_RETENTION_DAYS, small default) modeled on the play_logs prune.
Deliberately NOT another unbounded-growth table.
- Band transitions are asymmetric: jump up immediately (tighten fast), release one
level at a time after N calm samples below a deadband (release slow, no flap).
Pure nextBand() function, unit-tested deterministically.
- config: LAG_SAMPLE_INTERVAL_MS, LAG_RESOLUTION_MS, LAG_TELEMETRY_RETENTION_DAYS,
LAG_PRUNE_INTERVAL_MS, LAG_ELEVATED_MS, LAG_CRITICAL_MS, LAG_RELEASE_SAMPLES.
- tests: band-transition unit tests; integration proves sampling persists, stays
bounded under the prune, and surfaces on /api/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard uptime query (WHERE device_id=? AND timestamp>?) and the
per-device retention prune (WHERE device_id=? AND timestamp<?) were both full
table scans. At 1M+ rows (the outage report) this was the dashboard-degradation
cause that persisted even after the reconnect storm stopped.
- schema.sql: add idx_device_status_log_device_ts(device_id, timestamp); both
queries now SEARCH ... USING INDEX instead of SCAN (verified via EXPLAIN).
- database.js: same index as a migration for existing DBs (idempotent).
- schema.sql defined device_status_log twice; drop the duplicate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A mute toggle wrote the draft playlist_items + emitted a live device:mute-changed but only markDraft()'d — it never updated playlists.published_snapshot, the copy the device actually plays. So the device's item.muted stayed 0 and every loop/reload re-applied full volume: dashboard icon red but audio kept playing (Android; web's native <video> loop masked it). emitMuteChanged now surgically patches the matching item's muted (0/1) inside the published_snapshot and re-pushes the playlist, so loops re-apply the correct flag. Surgical patch (not publishPlaylist) so a mute toggle can't prematurely publish other draft edits or flip publish state. Adds a regression test that fails without the patch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
