Follow-up to #233, which made the upload ceiling configurable — the right call,
500MB is genuinely too low for video.
An environment variable is a string, so the value reached multer's
limits.fileSize as text where a number is expected. That survives some
comparisons through coercion and misbehaves in others, which is the worst kind
of bug to find later; the line directly above it already used parseInt for the
same reason. It is parsed properly now, and a suffix is accepted — someone
raising a limit for video is choosing "about 2GB", and 2147483648 is easy to
mistype by a factor of ten.
An unparseable value falls back to the default rather than becoming NaN or
zero. Either would reject every upload on the instance, from a typo in an env
file, with nothing on screen to explain it.
The documentation matters as much as the code here. MAX_FILE_SIZE is the LAST
limit in the chain: nginx caps the request body with client_max_body_size and
returns 413 before the app is reached — our own deployment sets 500M — and
Cloudflare caps uploads per plan at the edge. Raising the variable alone often
changes nothing, so the README now says so, with the nginx directive and a note
that an upload failing with nothing in the server log never reached the server.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uaeo9MvzKoyXuN6ZsbhtkL
Thanks — the hard-coded 500MB cap was genuinely too low for video, and making it configurable is the right call.
Merging as-is for the credit; a follow-up commit fixes two things this needs to actually work:
1. `process.env.MAX_FILE_SIZE` is a string, so the value reached multer as text rather than a number — the line directly above uses `parseInt()` for the same reason.
2. Raising it alone is not enough behind a reverse proxy. nginx caps request bodies at `client_max_body_size` (500M on our own deployment) and returns 413 before the app sees the upload, and Cloudflare's own cap applies too. That is now documented in the README alongside the variable.
The follow-up also accepts a suffix (`MAX_FILE_SIZE=2GB`) since typing the byte count is easy to get wrong.
Adds a pluggable email transport so self-hosters without Azure/M365 can send
mail through any standard SMTP server (Postfix, Gmail, Mailgun, SendGrid, corp
relay). Graph stays the default; behavior is byte-for-byte unchanged when
EMAIL_TRANSPORT is unset or "graph".
- config: EMAIL_TRANSPORT ("graph"|"smtp", default graph) + SMTP_HOST/PORT/
SECURE/USER/PASSWORD/FROM.
- services/email.js: branch by transport behind the SAME public sendEmail()/
isConfigured() surface. SMTP via nodemailer (lazy-required, like MSAL).
Shared across both transports: the "[ScreenTinker] " subject prefix (unless
rawSubject), the GRAPH_DEV_RESTRICT_TO allow-list, html-from-text derivation,
and the never-throws contract (failures log + return sent:false). SMTP_SECURE
true=implicit TLS(465)/false=STARTTLS(587). Auth optional (unauthenticated
relay ok); SMTP_USER without SMTP_PASSWORD is flagged. SMTP_FROM parses
"Name <addr>". New emailConfigStatus() for startup diagnostics.
- server.js: startup logs the transport and a LOUD error when the selected
transport is partially configured (some fields set, others missing) or when
EMAIL_TRANSPORT is invalid (falls back to graph). A fully-unset transport
stays a silent stdout fallback (unchanged dev behavior).
- nodemailer ^6.9.16 added as a production dep (bundled in the Docker image).
- .env.example + README: SMTP config section, Gmail example, transport table.
- test/email-transport.test.js: 15 tests — transport selection, config
validation (missing/partial/invalid), SMTP message building (from/prefix/
fromName override/text alt), sendEmail routing (mocked nodemailer), rawSubject,
dev-restrict on smtp, and the smtp_error never-throws path.
462/462 server tests pass. Boot verified for all four states (configured,
misconfigured, invalid, default).
Closes#173
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(dashboard): version indicator + GHCR update check with admin panel
- Add server/lib/ghcr-check.js: GHCR tag poller (36h cache, semver filter)
- Extend /api/version with latest_version and update_available
- Add POST /api/admin/check-update (force GHCR poll)
- Add POST /api/admin/trigger-update (Docker compose or manual instructions)
- Sidebar footer: version label + amber badge when update available
- Admin > System: version comparison card with Check/Update buttons
- 14 new tests (10 unit + 4 integration), 68/68 passing
Closes#163
* fix(dashboard): gate trigger-update to platform-admin + add GHCR fetch timeout
Review follow-up on #165 (the two blockers):
- trigger-update runs `docker compose up -d` on the HOST via docker.sock
(root-equivalent) but was behind requireAdmin, i.e. reachable by any
workspace-level admin. On a multi-tenant host that's a customer, not the infra
operator. Gate it with requirePlatformAdmin (DOCKER_UPDATE_ENABLED still gates
it further). check-update stays requireAdmin — it's a read-only GHCR poll.
- ghcr-check.checkNow had no fetch timeout. Node's global fetch has no default
timeout, so a hung GHCR connection never settled — leaving `inFlight` set
forever (the finally never ran), which wedged the background poller AND hung
any awaited checkNow (/api/admin/check-update). Add a 10s AbortController
timeout on both requests so the try/catch/finally always fire.
All 405 server tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ScreenTinker <hello@screentinker.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets an operator (or an MDM) own updates instead of the app self-installing, which
on managed panels shows a self-install confirm dialog over customer content
(#155). Three layered controls:
- GLOBAL (server): config.otaEnabled from OTA_ENABLED (default on). When off,
/api/update/check returns update_available:false, reason:ota_disabled_global —
the whole instance stops offering updates.
- PER-DEVICE (server + dashboard): new devices.ota_enabled column (default 1). When
0, that device is never offered an update (reason:ota_disabled_device). A
"Self-update (OTA)" toggle in the device settings flips it via PUT /api/devices/:id.
- AUTO-DETECT (Android): UpdateChecker stands down entirely when a foreign device
owner (an MDM/DPC) manages the panel — detected via getActiveAdmins() + not being
device owner ourselves. Pure client-side, errs safe, needs no server change.
The two server gates are enforced server-side so they cover EVERY client version,
not just ones with the client-side stand-down. When OTA is off the device still
reports its version (dashboard sees state); the MDM/operator owns the actual update.
For an MDM-managed fleet (e.g. Pivot/MAXHUB), turn OTA off and let the MDM push the
APK — the install-dialog race disappears from every angle.
Tests: +2 (per-device gate + a real OTA_ENABLED=false server for the global gate);
full server suite 393 pass; Android compiles.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the disk-fill trap: with wal_autocheckpoint=0 a dead worker means nothing checkpoints.
Controller now respawns an unexpectedly-dead worker (bounded: RespawnMax/RespawnWindowMs +
backoff); on exhaustion it re-arms a conservative inline autocheckpoint (FallbackPages) on the
main connection + reclaims the backlog, logging loudly. Clean stopWalCheckpointer() teardown is
distinguished via a 'stopping' flag so SIGTERM never triggers respawn. Env-gated worker
fault-injection (WAL_CKPT_FAIL_START) for tests. Local only — no bump/tag.
Disable wal_autocheckpoint on the main connection; run PASSIVE checkpoints from a
worker_threads worker with its OWN better-sqlite3 handle, escalating to TRUNCATE on a
size high-water or PASSIVE-starvation. Removes the synchronous fsync-heavy checkpoint
from the event loop. Config: walCheckpointIntervalMs/HighWaterMB/StarvationRuns.
Local only — no bump/tag.
Field-safe SERVER net. A device opening duplicate/rapid sockets (the APK duplicate-socket bug,
separate track) currently churns through evictions during the reconnect-throttle's 30s
post-restart WARM-UP (only the hard ceiling 20 applies then, so an 8-in-9s burst passes
undamped and each new socket evicts the prior). This makes the server absorb it: a thrashing
PAIRED device converges to ONE stable connection and stays online.
- lib/session-settle.js (decision only; bounded, swept): shouldHold(deviceId, incumbentAlive)
— true only when a socket was accepted for this device within SESSION_SETTLE_WINDOW_MS
(config, default 2500ms) AND the incumbent is alive. Warm-up-independent.
- deviceSocket register gate (just before evictPriorSocket): if a LIVE incumbent exists and
we're inside the window, SOFT-REFUSE the new socket (device:throttled reason=session_settle
+ disconnect) and keep the incumbent; else accept + evict + (re)arm the window.
- LIVENESS SAFEGUARD (load-bearing): only hold when the incumbent socket is actually in the
/device namespace — a dead/half-open incumbent is replaced, NEVER stranding the device (max
hold is the 2.5s window from the incumbent's accept, then any new socket is accepted).
- Soft refusal, NEVER a quarantine (reuses patch1's paired-safe philosophy); single-session
enforcement intact for a legitimate move; unpaired/abusive flapping still caught by the
existing limiters. O(1), no loop impact.
Tests (liveness first-class): live incumbent holds + DEAD incumbent replaced (not stranded);
storm of 6 sockets converges to ONE, stays online, not quarantined (during warm-up); single-
session move past the window replaces cleanly; unit decision + bounded sweep. The
evicted-socket-rearm test shrinks its settle window so it still exercises the eviction path.
Suite 336/336.
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fixed OTA_DOWNLOAD_MAX_CONCURRENT=10 throttled a legitimate coordinated rollout even
on a perfectly healthy server, and a shed 503 costs a client a full ~30-min re-check
cycle. Made the download guard's concurrency + rate caps band-aware:
- normal -> serve FREELY (no cap): a whole-fleet rollout isn't staggered when healthy
- elevated -> the configured caps engage (early backpressure)
- critical -> shed 503 (the real protection, unchanged)
Kill switch OTA_DOWNLOAD_GUARD_ENABLED=false disables it entirely.
Tests updated: normal serves 50/50 with 0 shed; caps + shed now asserted under elevated;
storm harness OTA flood runs under elevated (the loaded state). Suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flap limiter's auto-quarantine used to run `UPDATE devices SET blocked = 1` — a
PERMANENT, human-cleared block on an automatic trigger. A stuck-then-recovered device
stayed dark until someone noticed.
- Removed the auto-write from ws/deviceSocket.js. devices.blocked is now written ONLY by
an operator (dashboard endpoint / direct SQLite).
- Quarantine moved into lib/flap-limiter.js as IN-MEMORY, TIME-LIMITED state: after
connectRateQuarantineTrips trips in a window the identity is quarantinedUntil = now +
connectRateQuarantineMs (new, default 30m); check() then refuses cheaply with
reason:'quarantined' and AUTO-CLEARS when the window passes. Safe in-memory now that
Item A ended the restart loop, and a self-healing auto-action must not survive as a DB row.
- Log quarantine START once; repeat refusals go through the coalescer. Stale
"-> blocked=1" comments updated.
- connectRateQuarantineTrips=0 still disables it.
Tests: quarantine engages after N trips, refuses cheaply during the window, auto-clears
after connectRateQuarantineMs; and an integration flapper is quarantined while
devices.blocked stays 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Don't let telemetry/logging cook the loop under a storm.
- lib/log-coalescer.js: dedup+count high-frequency lines, flush ONE summarized line per
key per window ("[loop-lag] band=critical (x47 in 30s)"). Bounded buffer (auto-flush
at MAX_KEYS). Applied to the loop-lag "still loaded" line (band CHANGES stay immediate),
the per-request OTA check line, and "Device reconnected".
- loop-lag: event_loop_lag rows are BUFFERED and batch-inserted on a flush interval
(was a synchronous INSERT per sample); the buffer is bounded (drop-oldest). Its
retention prune now rides the Item-A chunkedDelete so this table can never repeat the
status_log bloat-then-freeze. /api/status still reads in-memory current (real-time
band unaffected).
- Bounded the previously un-evicted per-device Maps: content-ack limiter gets an idle
sweep (started in server.js); status-log-writer.lastWritten is capped (drop-oldest;
it only suppresses a redundant consecutive row, so eviction is safe).
Tests: N identical lines -> one counted line; single line verbatim; coalescer buffer
bounded under a distinct-key flood; content-ack Map swept of idle buckets.
loop-lag-integration updated for the batched-insert cadence. Suite 266/266.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fleet SNATs to one IP, so nothing on the OTA path may key on IP.
- /api/update/check: EARLY-RETURN before any filesystem call when the breaker won't
offer (rate-backoff / up-to-date / phantom / client-newer). A looping client that
gets rate-backoff now does ZERO fs — the flood can't become a statSync flood.
- lib/apk-cache.js: resolve APK path/size/mtime once at boot + refresh on an interval;
the check/download endpoints read cached metadata (get() does no fs, proven by test).
- lib/ota-download-guard.js + /download/apk: GLOBAL concurrency + rate caps + critical-
band shed (503 Retry-After), NEVER per-IP. Replaces the per-IP-per-10min log throttle
(which hid the flood under SNAT) with a per-window served/shed aggregate so a download
flood is VISIBLE. Bounded single rolling-state object; in-flight released on finish/close.
- Breaker unchanged; no IP limiting or device_id requirement added (legacy field clients
send no device_id on OTA checks — must keep working).
Tests: apk-cache get() = 0 statSync over 1000 reads; download guard sheds past global
concurrency + per-window rate + critical band; admit() has no IP parameter. Suite 259/259.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #142 burst throttle (5/10s) misses a device flapping every 3-5s (~2-3/10s) — yet
each cycle is an expensive register+build+acks and one status_log row (the spiral
trigger). Now that Item A ends the restart loop that used to wipe in-memory throttle
state every ~40s, an in-memory sustained limiter can finally bite.
- lib/device-identity.js: SNAT-safe identity resolution — device_id -> fingerprint
(map via device_fingerprints -> device_id, else raw fp) -> device_token -> ONE bounded
global anon bucket. NEVER IP (the fleet SNATs to 10.10.10.1). An unidentifiable client
is still bucketed (collectively) so an anon flood is capped, never unthrottled.
- lib/flap-limiter.js: per-identity connect-frequency over a long window
(CONNECT_RATE_WINDOW_MS=5min, CONNECT_RATE_MAX=20; anon bucket cap 60). Over the rate
-> refuse + disconnect (cheap). Bounded by an idle sweep (anon bucket never swept).
Optional auto-quarantine: a device_id-resolved hard flapper -> blocked=1.
- Wired at the device:register gate BEFORE fingerprint tracking/throttle/DB/build,
skipping same-socket playlist refreshes. Sweep started in server.js.
Tests: 4s-flapper refused after the window max; 60s-normal never; two device_ids
independent (never IP); device_id-less bucketed by fingerprint; neither id nor
fingerprint capped via global anon; idle sweep preserves the anon bucket. Suite 254/254.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bold beta1: three devices spam "Fingerprint reclaim rejected ... device active
(status=offline, ~2500s since heartbeat, liveConn=false)" twice/~2s indefinitely —
contradictory: gone by every signal yet treated as active.
Root cause (NOT a missing clear — corrected the hypothesis). The reject condition
was `liveConn || status==='online' || secondsSince < RECLAIM_GRACE_SECONDS(24h)`.
For the observed devices liveConn=false and status=offline, so the ONLY true term
is `secondsSince < 24h` — an effective 24h CALENDAR grace, not a stale flag. Audited
the clears: liveConn (deviceConnections) is removed on the debounced disconnect
(heartbeat.removeConnection) AND the offline_timeout sweep (deviceConnections.delete);
status is set 'offline' on both. liveConn=false + status=offline PROVE the clears
ran — there is nothing stale to clear. The 24h time gate (mislabeled "device active")
blocked a legitimately-gone device from reclaiming for up to 24h, so it retried
every ~2s forever-in-practice. The "twice per ~2s" is two reclaim ATTEMPTS per cycle
(client reconnect + re-pair-on-auth-error), each hitting the single console.warn —
not double-logging in one attempt.
Fix:
- Decide "still alive" from RUNTIME signals: `!!liveConn || secondsSince <
reclaimSettleSeconds`. A device with no live socket and a heartbeat older than the
settle window is gone -> reclaimable. A live (or just-seen) device is still
rejected, so reclaim-abuse protection holds. NOT just ignoring "active" — it fixes
WHY it was stuck (the 24h gate). RECLAIM_SETTLE_SECONDS default 300 (was 24h).
SECURITY TRADEOFF flagged in config: shortens the anti-fingerprint-theft window;
raise to re-tighten. Tuning guess to validate vs Bold.
- Log throttle: the deferral logs at most once per device per RECLAIM_REJECT_LOG_
WINDOW_MS (default 60s) — collapses the double-log + the per-2s flood (same
discipline as the content-ack shed log). Cleared when a reclaim proceeds.
Recovery of the 3 wedged devices (2febcaa9, 1984694c, 139159eb): they SELF-HEAL on
their next reclaim attempt (~2s) once this ships — their heartbeats are ~2500s stale
(>300s settle) and liveConn=false, so the reclaim now succeeds. No operator SQL needed.
Tests (port 3988): gone device reclaims; live device still rejected; clear-on-leave
(disconnect clears liveConn -> stale device reclaims); deferral log <=1 per window.
Full suite green serial+parallel (217). reconnect-throttle.js, the dbac699 content-ack
limiter, and the 404c330 block/auth code untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#142's content-ack dedup is insufficient: a device cycling 2-4 content IDs makes
every ack look unique so dedup never fires, while aggregate volume from ~30 devices
saturates the event loop (the #142 reconnect throttle kept the server responsive,
which is how this was even observable).
Folded ONE control on the content-ack path (no competing limiters; reconnect-
throttle.js untouched) in lib/content-ack-limiter.js:
- Step 1 — per-device RATE budget: caps TOTAL non-duplicate acks per device per
window regardless of differing content_id (the case dedup misses). Over budget =
DROP silently (the per-ack log+emit is the cost); log ONCE per device per window
when shedding starts. Keeps the #142 dedup (dedup'd repeats don't consume budget).
Per-device, in-memory, resets on restart (modeled on lastPlayLogAt; does NOT reuse
reconnect-throttle's ban-semantics bucket).
Env (TUNING GUESSES, validate vs Bold's fleet): CONTENT_ACK_MAX_PER_WINDOW=20,
CONTENT_ACK_RATE_WINDOW_MS=10000 (=2/s, above legit ~<=1/s, below the flood).
- Step 2 — global pressure valve: reuses the #142 loop-lag band (+ its hysteresis,
no second control loop). Under CRITICAL band, shed content-acks even for an
in-budget device; reconnects + dashboard/HTTP are ALWAYS processed; a healthy
device in a non-critical band is never touched by the valve. Valve open/close
logged once at the band edge in services/loop-lag.js (not per shed message).
Tests (unique ports 3985/3986, not the 3982/3983/3984 set):
- unit: the #143 regression (cycling ids evading dedup IS rate-limited), under/over
budget, dedup still works + doesn't consume budget, valve sheds in-budget under
critical while normal is untouched, rate precedence, window reset, per-device
isolation.
- integration: socket flood is capped to budget with a single shed-start log;
under-budget passes every ack; valve OPEN sheds content-acks while a reconnect +
/api/status still succeed.
Full suite green serial AND parallel (208 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
device:content-ack logged + emitted every message, so a device repeatedly
reporting the same "content <id>: ready" (observed from an older app version)
added avoidable load per message.
- Suppress identical (device_id, content_id, status) reports within
config.contentAckDedupMs (default 10s), modeled on the lastPlayLogAt throttle.
A status change has a different key and passes immediately; a fresh report after
the window passes too. In-memory, resets on restart. The handler does no DB
writes, so this is purely shedding redundant log+emit work.
test: integration over a real authenticated device socket — a burst of identical
"ready" collapses to one log/emit, a "ready" after the window passes, and a status
change is never deduped. Unique PORT (3984).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-device insert-time prune (deviceSocket.js) only ever touches a device
that is actively inserting, so it misses two paths: removed/idle devices whose
rows linger forever, and heartbeat.js's offline_timeout insert that bypasses
logDeviceStatus entirely. The reporter's 1.2M-row bloat accumulated UNDER a 7-day
per-device prune for exactly this reason.
- pruneStatusLog() (db/database.js): a GLOBAL time-range sweep across ALL devices,
modeled on the play_logs prune. Run once on startup (recovers a bloated table
right after deploy) and on the heartbeat interval (services/heartbeat.js).
- STATUS_LOG_RETENTION_DAYS env, default 3 (lower than the old hardcoded 7d; the
dashboard only shows a 24h uptime window, so 2-3d is ample for diagnostics).
- Deliberately NO per-device row cap: Step 3's throttle already bounds how fast a
storming device can generate status rows, so a cap would add sweep complexity
for little gain (noted for later if needed).
- NO VACUUM / auto_vacuum here (kept off the hot path); space reclaim is left as a
separate decision (see report).
test: deterministic in-process unit test proves the sweep deletes over-retention
rows across all devices — including a device absent from the devices table and an
offline_timeout row — while keeping recent rows; idempotent on an empty table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gates genuine reconnects PER DEVICE before the heavy register work (DB writes +
playlist build) runs, so a single flapping device can no longer saturate the
event loop and take down the server.
- Actuator is per-device, keyed on device_id (modeled on lastPlayLogAt). A device
is flagged only when it exceeds reconnectBaseMax genuine reconnects per window.
Same-socket playlist refreshes (isPlaylistRefresh) are exempt.
- Load-awareness is BANDED (normal/elevated/critical from the step-2 lag signal),
not a continuous controller. The band only MULTIPLIES an already-flagged
device's backoff; global lag never gates a healthy device.
- Hysteresis: escalate immediately while storming (tighten fast); decay one level
per reconnectReleaseMs of calm (release slow).
- HARD CEILING per device, independent of band and warm-up — a slow-ramp attacker
can't train through it.
- COLD START: for reconnectWarmupMs after boot, force the normal band and apply
only the hard ceiling, so a full-fleet reconnect after a deploy doesn't throttle
healthy screens. State is in-memory, resets on restart.
- Observability: every throttle engagement logs device, band, observed vs allowed
rate, and backoff. Throttled device gets device:throttled + a deferred disconnect.
Tests (api.test.js style):
- unit: healthy-never-throttled, storm-throttled-with-growing-backoff, band
multiplies backoff, hard-ceiling-even-in-warmup, warm-up leniency, neighbor
isolation, slow release.
- integration GATE (the required one): full-fleet reconnect right after restart
throttles NO healthy device; a single device storming IS throttled; a neighbor
stays unaffected while another storms.
- also fixes pre-existing test PORT collisions (my new integration files clashed
with totp.test.js:3979 and totp-keyrotation.test.js:3980 -> moved to 3982/3983);
full suite now green serially AND in parallel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuously samples event-loop delay via perf_hooks.monitorEventLoopDelay()
(C++-backed histogram; cheap). Each window persists mean/p50/p99/max to a new
event_loop_lag table and recomputes a coarse load band (normal/elevated/critical)
from the window p99. Standalone value: current lag is exposed on /api/status and
band changes are logged, so site lag is diagnosable independent of throttling.
The band feeds the #142 reconnect throttle (next commit) but ships first as its
own subsystem.
- event_loop_lag is bounded from day one: indexed on sampled_at + scheduled prune
(LAG_TELEMETRY_RETENTION_DAYS, small default) modeled on the play_logs prune.
Deliberately NOT another unbounded-growth table.
- Band transitions are asymmetric: jump up immediately (tighten fast), release one
level at a time after N calm samples below a deadband (release slow, no flap).
Pure nextBand() function, unit-tested deterministically.
- config: LAG_SAMPLE_INTERVAL_MS, LAG_RESOLUTION_MS, LAG_TELEMETRY_RETENTION_DAYS,
LAG_PRUNE_INTERVAL_MS, LAG_ELEVATED_MS, LAG_CRITICAL_MS, LAG_RELEASE_SAMPLES.
- tests: band-transition unit tests; integration proves sampling persists, stays
bounded under the prune, and surfaces on /api/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opt-in, default-off UI gate (per strobe's spec; verified his file refs first).
When set, hides the Subscription sidebar item + billing view and bounces
#/billing to the dashboard. Billing shown by default -> existing deployments
unchanged. UI-only: /api/subscription/* untouched (internal usage reads stay).
- config.js: config.hideBilling from HIDE_BILLING (mirrors selfHosted).
- auth.js: surface hide_billing on GET /api/auth/me (client already fetches it
at boot, stored on the user object).
- index.html: id="billingNavItem" on the Subscription <li> (mirrors adminNavItem).
- app.js: toggle billingNavItem in updateSidebarUser (next to the admin toggle);
guard #/billing -> history.replaceState('#/') + render dashboard (replaceState
so the back button doesn't loop into the guard).
- .env.example + README documented.
Spec assumptions verified against code: adminNavItem toggle pattern exists;
/me is fetched at boot and updateSidebarUser runs both at boot (cached user)
and post-/me, so no-flash holds on warm loads (one-time flash possible on the
first load after the flag flips — same as the admin nav, minor); route dispatch
is an if/else chain. Nav label is static (no data-i18n) so no i18n change.
Validated (headless Chrome, both states):
- flag unset -> Subscription tab present, #/billing renders (backward-compat).
- HIDE_BILLING=true -> tab hidden, #/billing redirects to #/.
- config maps HIDE_BILLING both ways; live /me default hide_billing=false.
- 149 server tests green. Default-off = zero change for existing deployments.
Known cosmetic (harmless): after the redirect the billing nav LINK keeps its
'active' class, but the nav item is display:none so it's never visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- server/version.js: shared version helper that reads the root VERSION file once
(fallback 0.0.0). Replaces the stale hardcoded 1.2.0 / 1.5.1 / 1.0.0 fallbacks
in /api/version, /api/update/check, and /api/status.
- config.js: DATA_DIR / DB_PATH / UPLOADS_DIR / CERTS_DIR env overrides for the
db, uploads, and certs/jwt-secret locations. Unset resolves to exactly the
legacy in-repo paths, so existing installs (including production) are
byte-for-byte unchanged. Guarded by test/config-paths.test.js.
- package.json: rename remote-display-server -> screentinker (+ lockfile name).
- scripts/bump-version.sh: one-shot bump across VERSION, package.json (+lock),
android (versionName and versionCode + 1), and the tizen widget version; makes
one commit plus an annotated tag; prints the push command, never pushes.
- .gitignore: global *.db / *.db-wal / *.db-shm / *.db.* so no database file
(including .db.devbak backups, at any path) can be committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MSP-style deployments want self-service signups created WITHOUT a personal
org, so an admin/operator can assign them into an existing customer org
afterward.
- config.autoCreateOrgOnSignup (AUTO_CREATE_ORG_ON_SIGNUP env), default
true - single-tenant and the hosted self-service flow are unchanged.
- ensureDefaultOrgForUser gains { allowCreate }: an existing membership is
always returned (idempotent); the MINT path is gated. allowCreate=false +
no membership -> returns null (user created org-less).
- register accepts a per-request createOrg flag overriding the deployment
default; the first-ever user is always given an org (never headless).
login / Google / Microsoft pass allowCreate from the global config, so an
org-less user is not silently given an org on next sign-in.
Edge case: a non-platform user with zero workspaces now lands on a "no
workspaces yet" empty state (new no-workspace view) instead of being bounced
into onboarding (whose pairing step needs a workspace). route() redirects
them there, and refreshCurrentUser() redirects once /me reveals zero
accessible_workspaces (covers the first-load race). The workspace switcher
already rendered an empty placeholder and resource routes already return []
for a null workspace, so nothing crashes in between.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Short-lived per-device queue covers the TV-flap window (issue #3):
when a device is mid-reconnect, prior code emitted to an empty room
and the event vanished. Now playlist-updates and commands targeting
an offline device are queued and flushed in order on the next
device:register for that device_id.
server/lib/command-queue.js (new):
- pendingPlaylistUpdate: per-device marker (rebuild via builder on
flush -> always fresh DB state, no stale snapshots)
- pendingCommands: per-device Map<type, payload> with last-of-type
dedup (most recent screen_off wins)
- TTL via COMMAND_QUEUE_TTL_MS env (default 30000)
- Active sweep every 30s prunes expired entries
Memory bounds: ~6 entries per device worst case (1 playlist marker
+ 5 command types), unref'd sweep timer.
Wired emit sites (8 total; the four direct socket.emit calls in
deviceSocket register handlers are intentionally NOT queued because
the socket is alive by definition at those points):
- server/routes/video-walls.js (pushWallPayloadToDevice)
- server/routes/device-groups.js (pushPlaylistToDevice)
- server/routes/content.js (content-delete fan-out)
- server/routes/playlists.js (pushToDevices + assign)
- server/services/scheduler.js (scheduled rotations)
- server/ws/deviceSocket.js x2 (wall leader reclaim/reassign)
server/ws/deviceSocket.js register paths now call flushQueue after
heartbeat.registerConnection + socket.join. Existing
socket.emit('device:playlist-update', ...) lines kept - they send
the initial state on register; the flush replays any queued events.
Player's handlePlaylistUpdate fingerprint check dedupes the
overlap.
Refs #3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make HEARTBEAT_INTERVAL and HEARTBEAT_TIMEOUT env-tunable so
self-hosters with slow/jittery networks don't have to edit
config.js (issue #3 reporter did exactly this to confirm the
diagnosis). Defaults unchanged at 10000ms / 45000ms so existing
deployments keep current behavior.
Same parseInt(env) || default pattern as PORT/HTTPS_PORT/PING_*.
README env table extended.
Refs #3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Connection-stability layer for issue #3. LG webOS WebKit (and other
TV-grade clients) miss Engine.IO pongs under decode load with the
Socket.IO defaults of 25s ping / 20s timeout, causing spurious
transport drops and a connect/reconnect/evict/disconnect loop on
the device. Default polling-first transport adds another fragility
layer via the polling->WebSocket upgrade dance.
- pingInterval / pingTimeout default to 30000 / 30000 (worst-case
dead-socket detection 60s, up from ~45s). Both env-configurable
via PING_INTERVAL / PING_TIMEOUT.
- Player Socket.IO client: transports: ['websocket', 'polling'].
Tries WebSocket first; falls back to polling on the same connect
attempt if WebSocket fails. Polling fallback preserved for
firewall-restricted networks.
App-level heartbeat checker is unchanged and remains the safety net
for clients that miss the transport-level ping/pong window.
Tradeoffs documented in inline comments. README env table extended
with PING_INTERVAL and PING_TIMEOUT rows.
Refs #3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-hosters running internal-only deployments don't need the
marketing homepage. With DISABLE_HOMEPAGE=true, requests to /
302-redirect to /app instead of serving the landing page.
Unset/false preserves current behavior.
Requested via discord feedback.
Replaces the unused EMAIL_WEBHOOK_URL stub with a real Microsoft Graph
Mail.Send pipeline via @azure/msal-node client-credentials flow. Prior
state on prod: every alert email was logged to journalctl and never
sent (21 fallback log lines per hour for the chronic-offline devices).
Four coordinated changes shipped as one commit since they're all part
of making email delivery actually work responsibly:
1. services/email.js (NEW): Graph send via plain HTTPS (no SDK), in-memory
MSAL token cache (refresh 60s pre-expiry), graceful stdout fallback
when GRAPH_* env vars absent. Drop-in replacement for the old webhook.
2. services/alerts.js refactored: sequential await around sendEmail (was
parallel fire-and-forget; first run hit Graph's MailboxConcurrency 429
ApplicationThrottled on a 30-device backlog). Sequential at ~250ms per
send takes 5-8s for the full backlog, well within the 60s tick. Also:
24h long-offline cutoff to stop nagging about chronic-offline devices
(the 20,000+ minute ones); 2-hour dedup window (was 1h) via a generic
shouldSendAlert(type, id, windowMs) helper that future alert types
(payment_failed, plan_limit_hit, etc.) can reuse.
3. Preferences UI: single checkbox in settings.js Account section bound
to users.email_alerts. Saved via the existing Save Profile button. PUT
/api/auth/me extended to accept email_alerts. requireAuth middleware
SELECT now includes email_alerts so it propagates via req.user.
4. Dev safety net: GRAPH_DEV_RESTRICT_TO env var as an allow-list. When
set, only listed recipients reach Graph; everyone else is suppressed
with a log line. Prevents local dev (which often runs against fresh
prod DB copies) from accidentally emailing real prod users. UNSET on
prod systemd unit so production fans out normally.
Also: package.json scripts use --env-file-if-exists=.env so local dev
picks up .env automatically (Node 20.6+ built-in, no dotenv dep). Prod
runs via systemd ExecStart and is unaffected. server/.gitignore added
to keep .env out of git.
Smoke verified end-to-end:
- Sequential send pattern verified (a prior parallel-send tick had hit
Graph's MailboxConcurrency 429 on 30 simultaneous sends; sequential
at ~250ms each completes the same backlog without throttling)
- 24h cutoff silenced 20/21 prod devices on the next tick
- Dev restrict suppressed the 1 within-24h send
- User-preference toggle flipped via UI -> DB -> alert path silently
continued before reaching even the suppression log
When DISABLE_REGISTRATION=true (or 1), POST /api/auth/register returns
403 with a clear error. OAuth endpoints (/google, /microsoft) also
refuse to auto-create new accounts — existing OAuth users can still
sign in. First-user setup (empty users table) is always allowed so a
fresh install can still be initialized.
GET /api/auth/config now returns registration_enabled so the login
view can hide the "Create Account" button and the trial banner when
registration is off. Absence of the flag is treated as enabled for
back-compat with older servers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ScreenTinker - open source digital signage management software.
MIT License, all features included, no license gates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
