* feat(diagnostics): device incident log — why a screen went offline/black, with device-attested cause
Field request (Bold/s_t_r_o_b_e): "screens go offline randomly — let us see the cause." Answers it
across the fleet with a unified incident log, and — the key insight — lets the DEVICE disambiguate
the cause the server can't: if the app process survived the gap it was a NETWORK problem (not a
reboot), and it can even tell a dropped Wi-Fi/Ethernet link from a link-up-but-server-unreachable
(router/upstream) failure.
Schema:
- device_status_log gains reason + detail (WHY each offline transition happened).
- NEW device_events table (unified incident feed): type (offline/online/display_off/display_on/
crash/reboot/network/app_error) + reason + detail, indexed, age-pruned + per-device capped.
Server:
- Capture the socket.io disconnect REASON (transport_close/ping_timeout/transport_error) instead of
discarding it — recorded in the offline-cause log. devices.offline_reason stays on the EXIT-SIGNAL
contract (crashed/clean_exit/silent) — a separate axis, preserved (violent death = 'silent').
- device:event handler (typed incidents) + device:connectivity-report handler (device-attested).
lib/incident-classify.js (pure, unit-tested) composes reason+detail: cold_start->reboot;
link_lost->network "Wi-Fi/Ethernet link lost"; else network "LAN up, server unreachable
(router/upstream)"; appends SSID / weak-signal (rssi<-75) / IP-changed. On a report it upgrades the
most-recent offline row from the server's guess to the device's ground truth.
- heartbeat timeout -> 'heartbeat_timeout'; retention/cap for device_events.
- Device-detail API returns statusLog.reason/detail + the last 50 device_events.
Device (Android WebSocketService): ConnectivityManager default-network callback (link-lost during a
gap) + Wi-Fi SSID/RSSI + IP snapshot -> device:connectivity-report on reconnect (app survived =>
network); ACTION_SCREEN_ON/OFF receiver -> device:event display_on/off ("screen went black"). All
guarded/feature-detected; no manifest change; compiles clean.
Web + Tizen players: reconnect connectivity-report (link_lost from navigator.onLine during the gap)
+ visibilitychange -> display_off/on. Best-effort (no wifi detail in a browser). Tizen exit-signal
marker slice untouched.
CMS (device-detail): the offline cause on the uptime-timeline hover + a new "Recent incidents" panel
(merged offline periods + typed events, friendly labels, detail, relative time + down-duration).
Built as a 4-way parallel agent fan-out over disjoint domains against a locked contract, then
integrated. Verified: full server suite 443/443 (incl. the seam fix keeping the exit-signal contract
intact), Android compileDebugKotlin clean, all players + CMS node -c clean.
Refs #170.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(diagnostics): internet-reachability probe — split "our server down" from "no internet" (#170)
Follow-up to the incident log: when a device's link is UP but it's offline, "router/upstream" was a
catch-all. The device now probes a public host (1.1.1.1 / 8.8.8.8 :443) DURING the gap, so the cause
pinpoints blame:
- link_lost=true -> Wi‑Fi/Ethernet link lost (device's own link)
- link up, internet_ok=true -> server_down: internet reachable, OUR server was unreachable
- link up, internet_ok=false -> no_internet: router/ISP down
- link up, no probe result -> generic router/upstream (unchanged fallback)
- Android WebSocketService: fire a short daemon-thread TCP probe (443, either host) at disconnect;
the result rides the connectivity-report as internet_ok (omitted if the gap ends before it finishes).
- lib/incident-classify.js: 3-way split on internet_ok; new reasons server_down / no_internet.
- Frontend i18n: device.event.server_down / .no_internet labels.
- Tests: +3 classify cases (server_down, no_internet, link_lost wins over internet_ok). 12/12.
Verified: classify 12/12, Android compileDebugKotlin clean, node -c clean. Refs #170.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(diagnostics): log an 'upgrade' incident (old → new app_version) — server-side (#170)
When a device reports an app_version different from the stored one, applyDeviceInfo logs an
'upgrade' device_events row (detail 'old → new'). Server-side, so it covers Android/Tizen/web with
no client change; a fresh pair (no prior version) isn't counted. Adds device.event.upgrade label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Server-side keystone: the server now honors the v4 liveness contract uniformly across the MIXED
fleet (v4 + old pre-v4 + disconnected), all three clients depending on it.
- UNIFORM heartbeat-ack: emitted from the single shared device:heartbeat handler (uniform by
construction; no per-client/per-path branch), BEFORE the auth guard so a known device's watchdog
stays armed. Harmless to old clients (they ignore it).
- RECONNECT-WINDOW ack-gap fix (ackableHeartbeat): ack a KNOWN device (authed socket OR a device_id
that resolves) even mid-reconnect; NOT anonymous/never-authenticated sockets (degrade-safe);
identity-agnostic. No state mutation before requireDeviceAuth (auth surface unchanged; device_ids
are uuidv4).
- DASHBOARD LIVENESS (deriveLiveness): server-derived, VERSION-AGNOSTIC Healthy/Degraded/Offline
from signals every client sends (socket presence, heartbeat age, reconnect frequency); no client
status-push.
- IDENTITY CAPTURE (capture-don't-act): client_type/client_version/platform/contract_version columns;
degrades to legacy/unknown for old clients; NEVER breaks register.
- A-BUCKET FIX (QA): recordReconnect + persistIdentity gated on !isPlaylistRefresh (a ~45-60s refresh
is not a reconnect/new identity — matches #134), and the identity write is change-detected — closing
the WAL write-amplification (A1) and the benign-refresh -> false-"Degraded" (A2) regressions.
New lib/liveness.js (pure helpers, unit-tested). 30 new tests (uniform ack, ack-gap, mixed fleet,
identity capture, cross-client conformance, refresh-gate reproduce-then-prove); 366/366 total.
OTA artifact-availability is a separate concern (out of scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Item 2: when the heartbeat checker marks a device offline it now also disconnects any socket
it still holds for it, so DB-offline can't diverge from socket-state into a silent half-open
(defensive — the live-socket guard already defers genuinely-live sockets).
Item 3: tighten half-open detection WITHOUT reintroducing the TV-WebKit decode-load risk the
30s pong-timeout was chosen for — lower only pingInterval 30s->15s (probe more often), KEEP
pingTimeout at 30s. Detection = interval+timeout = 45s (was 60s), and the client inherits
these via the handshake so BOTH ends detect a dead peer ~25% sooner. (Deliberately did NOT
drop pingTimeout to ~20s: MAXHUB is a video-playing TV-class device and the code comment
warns tighter timeouts cause spurious drops under decode load.)
Item 4: SO_KEEPALIVE on every accepted connection (lib/tcp-keepalive.js) so a half-open TCP
can't persist indefinitely at the OS layer, independent of the app ping.
Tests: server closes a non-ponging peer within ~pingInterval+pingTimeout while a ponging peer
survives; a device whose transport dies ends offline with its connection torn down; keepalive
applied to each accepted connection (and never breaks setup on error). Suite 328/328.
Implements the ByteTinker-Bold distribution-agreement billing math and surfaces it on a
standalone admin-only route. No UI (the API figure is the deliverable). Server-side only.
Contract math (lib/billing.js, config-driven; defaults ARE the agreement):
- ASD (per device/day) = min(1.0, online_seconds / (hours*3600)) # 28800 default
- BillableScreens (per month) = round-half-up( Sum ASD / days_in_month )
- Flat tier (not marginal): 1-499 $1.50 / 500-999 $1.25 / 1000+ $1.00; cost = screens*rate.
Single global rate card for now (per-tenant is a future concern; noted in code).
Data foundation:
- New durable rollup device_usage_daily(device_id, day 'YYYY-MM-DD', online_seconds),
index on day. status_log (3d) / telemetry (24h) can't back a billing month.
- Accumulated INCREMENTALLY off the heartbeat tick from the live connection map (same
source as devices_connected) - never reconstructed from logs. Each tick credits every
connected device's today-row (min(86400, +elapsed)), chunked + transactional (non-blocking);
per-tick credit capped (accrualCapSeconds) as a stall/restart guard.
- Retention ~400d, pruned via chunked-prune (pruneUsageDaily in runMaintenance).
API: GET /api/billing/usage?month=YYYY-MM (default current), requirePlatformAdmin, mounted
SEPARATELY from /api/status (billing is revenue data + a heavier aggregate; must not touch
the hot status path). Reads the rollup only. MTD figure averages over COMPLETED days only
(today shown in `daily` but excluded until it completes); is_final + billable_screens_final
appear once the month completes.
Tests (12): ASD math; billable round-half-up; flat tier/cost boundaries; accumulator
(accrues by interval, caps at 86400/day, disconnected doesn't accrue); report MTD-excludes-
today + final-month is_final; retention prune; endpoint authz (admin 200 / non-admin 403 /
anon 401) + billing absent from /api/status. Suite 301/301. First-full-month caveat +
formula in docs/billing.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coalesced loop-lag summary carried an arbitrary sample's p99. Now record() tracks
the MAX (peak) over the window and the summary emits it — the peak is the number that
matters during an incident: '[loop-lag] band=critical (x47 in 30s, peak 1502ms)'. Band
CHANGES still log immediately, unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Don't let telemetry/logging cook the loop under a storm.
- lib/log-coalescer.js: dedup+count high-frequency lines, flush ONE summarized line per
key per window ("[loop-lag] band=critical (x47 in 30s)"). Bounded buffer (auto-flush
at MAX_KEYS). Applied to the loop-lag "still loaded" line (band CHANGES stay immediate),
the per-request OTA check line, and "Device reconnected".
- loop-lag: event_loop_lag rows are BUFFERED and batch-inserted on a flush interval
(was a synchronous INSERT per sample); the buffer is bounded (drop-oldest). Its
retention prune now rides the Item-A chunkedDelete so this table can never repeat the
status_log bloat-then-freeze. /api/status still reads in-memory current (real-time
band unaffected).
- Bounded the previously un-evicted per-device Maps: content-ack limiter gets an idle
sweep (started in server.js); status-log-writer.lastWritten is capped (drop-oldest;
it only suppresses a redundant consecutive row, so eviction is safe).
Tests: N identical lines -> one counted line; single line verbatim; coalescer buffer
bounded under a distinct-key flood; content-ack Map swept of idle buckets.
loop-lag-integration updated for the batched-insert cadence. Suite 266/266.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#142's content-ack dedup is insufficient: a device cycling 2-4 content IDs makes
every ack look unique so dedup never fires, while aggregate volume from ~30 devices
saturates the event loop (the #142 reconnect throttle kept the server responsive,
which is how this was even observable).
Folded ONE control on the content-ack path (no competing limiters; reconnect-
throttle.js untouched) in lib/content-ack-limiter.js:
- Step 1 — per-device RATE budget: caps TOTAL non-duplicate acks per device per
window regardless of differing content_id (the case dedup misses). Over budget =
DROP silently (the per-ack log+emit is the cost); log ONCE per device per window
when shedding starts. Keeps the #142 dedup (dedup'd repeats don't consume budget).
Per-device, in-memory, resets on restart (modeled on lastPlayLogAt; does NOT reuse
reconnect-throttle's ban-semantics bucket).
Env (TUNING GUESSES, validate vs Bold's fleet): CONTENT_ACK_MAX_PER_WINDOW=20,
CONTENT_ACK_RATE_WINDOW_MS=10000 (=2/s, above legit ~<=1/s, below the flood).
- Step 2 — global pressure valve: reuses the #142 loop-lag band (+ its hysteresis,
no second control loop). Under CRITICAL band, shed content-acks even for an
in-budget device; reconnects + dashboard/HTTP are ALWAYS processed; a healthy
device in a non-critical band is never touched by the valve. Valve open/close
logged once at the band edge in services/loop-lag.js (not per shed message).
Tests (unique ports 3985/3986, not the 3982/3983/3984 set):
- unit: the #143 regression (cycling ids evading dedup IS rate-limited), under/over
budget, dedup still works + doesn't consume budget, valve sheds in-budget under
critical while normal is untouched, rate precedence, window reset, per-device
isolation.
- integration: socket flood is capped to budget with a single shed-start log;
under-budget passes every ack; valve OPEN sheds content-acks while a reconnect +
/api/status still succeed.
Full suite green serial AND parallel (208 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
services/heartbeat.js deleted unclaimed provisioning devices with
created_at < now - (365 * 86400) — a YEAR — while its own comment said "older
than 24 hours". So socket-register pairing junk lingered ~365x longer than
intended. Change the window to 24 * 3600 to match the comment.
Correctness fix only — does NOT touch the pre-auth register path or add a rate
limiter (that pre-auth hardening is a separate security issue, out of this cut).
Extracted the sweep into pruneProvisioningDevices() (still in heartbeat.js, called
from the same interval) so it is unit-testable. Test asserts a >24h unclaimed
provisioning row is swept while a <24h row, an imported row (user_id set), and a
non-provisioning row are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-device insert-time prune (deviceSocket.js) only ever touches a device
that is actively inserting, so it misses two paths: removed/idle devices whose
rows linger forever, and heartbeat.js's offline_timeout insert that bypasses
logDeviceStatus entirely. The reporter's 1.2M-row bloat accumulated UNDER a 7-day
per-device prune for exactly this reason.
- pruneStatusLog() (db/database.js): a GLOBAL time-range sweep across ALL devices,
modeled on the play_logs prune. Run once on startup (recovers a bloated table
right after deploy) and on the heartbeat interval (services/heartbeat.js).
- STATUS_LOG_RETENTION_DAYS env, default 3 (lower than the old hardcoded 7d; the
dashboard only shows a 24h uptime window, so 2-3d is ample for diagnostics).
- Deliberately NO per-device row cap: Step 3's throttle already bounds how fast a
storming device can generate status rows, so a cap would add sweep complexity
for little gain (noted for later if needed).
- NO VACUUM / auto_vacuum here (kept off the hot path); space reclaim is left as a
separate decision (see report).
test: deterministic in-process unit test proves the sweep deletes over-retention
rows across all devices — including a device absent from the devices table and an
offline_timeout row — while keeping recent rows; idempotent on an empty table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuously samples event-loop delay via perf_hooks.monitorEventLoopDelay()
(C++-backed histogram; cheap). Each window persists mean/p50/p99/max to a new
event_loop_lag table and recomputes a coarse load band (normal/elevated/critical)
from the window p99. Standalone value: current lag is exposed on /api/status and
band changes are logged, so site lag is diagnosable independent of throttling.
The band feeds the #142 reconnect throttle (next commit) but ships first as its
own subsystem.
- event_loop_lag is bounded from day one: indexed on sampled_at + scheduled prune
(LAG_TELEMETRY_RETENTION_DAYS, small default) modeled on the play_logs prune.
Deliberately NOT another unbounded-growth table.
- Band transitions are asymmetric: jump up immediately (tighten fast), release one
level at a time after N calm samples below a deadband (release slow, no flap).
Pure nextBand() function, unit-tested deterministically.
- config: LAG_SAMPLE_INTERVAL_MS, LAG_RESOLUTION_MS, LAG_TELEMETRY_RETENTION_DAYS,
LAG_PRUNE_INTERVAL_MS, LAG_ELEVATED_MS, LAG_CRITICAL_MS, LAG_RELEASE_SAMPLES.
- tests: band-transition unit tests; integration proves sampling persists, stays
bounded under the prune, and surfaces on /api/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reuses the existing scheduler + sendEmail infra (no new scheduler). The agency endpoint
enqueues one agency_notifications row per item added; a 15-min flush groups unsent rows per
token+playlist+action and sends ONE digest per group to the workspace owner/admins + the
playlist owner (deduped via UNION). Draft -> "added N items, awaiting approval"; published ->
"updated <playlist>".
Two robustness rules, both tested:
- Queue never balloons when SMTP is off: the endpoint skips enqueue when !isConfigured(),
and the flush drains-and-discards unsent rows as a backstop.
- sent_at is stamped ONLY after a successful send, so a failed send retries next cycle
instead of silently dropping.
Wired into boot via startAgencyDigest(). 147 suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each playlist item can carry schedule blocks (active days, start/end
time-of-day, optional start/end dates). An item plays when the screen's
local "now" matches at least one block; an item with no blocks always
plays. #74 covers time-of-day/day-of-week windows including overnight
wrap; #75 covers inclusive date ranges (auto-expiry). Evaluation is
on-device, so dayparting and expiry work offline.
- Shared evaluator contract: shared/schedule-vectors.json (39 vectors —
DST US+AU, overnight-wrap anchoring, timezone correctness, date
boundaries). Canonical JS evaluator in server/lib/schedule-eval.js;
Kotlin and Tizen ports kept in lockstep by drift guards (Tizen byte-diff
test, Kotlin JUnit reads the shared JSON, new android-test CI job).
- All three players (web, Android, Tizen) filter by schedule against their
own clock, idle with a "Nothing scheduled" message + 30s re-check when
everything is filtered, and fail open on any evaluator error.
- Editor: per-item schedule modal + row badge in the playlist editor;
client validation mirrors the server; editing marks the playlist draft.
- Part B (behaviour change): device/group schedule overrides now evaluate
in each device's effective timezone instead of server-local time.
- Device detail shows the reported timezone + a clock-skew warning.
- i18n for en/es/fr/de/pt across all new strings (namespaced itemsched.*
to avoid colliding with the device-schedule calendar's schedule.*).
- CHANGELOG documents the feature, the Part B change, the fail-open
guarantee, and the scheduled-single-video re-render tradeoff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daily sweep (15:00 UTC) emails a warm, personal "checking in" message
to users who signed up 3-14 days ago and still have no paired screen,
nudging them toward activation. Once per user, reuses the Graph
transport (services/email.js) via the existing fromName/rawSubject
options.
- New service services/activationNudge.js, started from server.js.
Self-correcting daily scheduler (recompute next 15:00 UTC each run;
no node-cron dependency).
- Eligibility (Option B, workspace-aware): created 3-14 days ago,
activation_nudge_sent_at IS NULL, COALESCE(email_alerts,1)=1 (only
an explicit opt-out of 0 is excluded; NULL/unset still qualify), and
ZERO devices owned by the user OR present in any workspace they
belong to. The workspace check avoids nudging engaged team members.
- Idempotency: activation_nudge_sent_at, stamped after send; paired
sentinel-1 backfill so the first sweep can't blast the dormant
legacy base. Only genuinely-new signups become eligible.
- GATE: HOSTED_INSTANCE=true (positive hosted signal, NOT !selfHosted).
A daily bulk sweep would be far worse to leak than a single email, so
a self-hoster who configured Graph but missed SELF_HOSTED won't blast
their user base. Unset -> neither scheduled nor sent. Documented in
.env.example.
The admin signup-notify recipient was hardcoded to
support@screentinker.com and shipped in the open-source code. Combined
with the opt-out SELF_HOSTED gate, any self-hoster who configured their
own Graph credentials but forgot SELF_HOSTED=true would fire their
users' signup PII (email, IP, country) into our support inbox.
Source the recipient from ADMIN_NOTIFY_EMAIL instead, defaulting to
null. When unset, the admin notification is skipped entirely and logged
("[SIGNUP-EMAIL] admin notify skipped (ADMIN_NOTIFY_EMAIL unset)"); the
user's welcome email is unaffected. Hosted prod sets the env var so its
notifications continue; self-hosters send nothing to us by default, and
the .com address no longer ships in code.
Document ADMIN_NOTIFY_EMAIL (and the related mail/self-host vars) in a
new .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin signup notifications were going to dw5304@gmail.com. Route them
to the monitored support@screentinker.com queue instead, so signups
land in the shared inbox rather than a personal account.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every new user now gets a personal welcome email from
"Dan at ScreenTinker" <support@screentinker.com>, and Dan gets an
admin notification, immediately after signup. Fired from all three
signup paths (local /register, Google, Microsoft) via a shared
helper (services/signupEmails.js) at the new-user branch only, so
OAuth logins of existing users don't re-trigger.
- Reuses the single Microsoft Graph transport (services/email.js).
Adds two optional, backward-compatible params: fromName (custom
From display name; address stays support@ so replies route there)
and rawSubject (skip the "[ScreenTinker] " prefix for clean
subjects "Welcome to ScreenTinker" / "New signup: <email>").
- Idempotency: users.welcome_email_sent_at, stamped after the send
block; non-null short-circuits so a user is only emailed once.
Paired backfill stamps all pre-existing users with sentinel 1 so
a future "IS NULL" sweep can't mistake the legacy base for
un-welcomed and blast them.
- Production-only: gated on !config.selfHosted so self-host
operators never emit mail from our domain or CC Dan.
- No retry logic by design (no re-trigger path on existing users);
per-email {sent, reason} is logged so a Graph hiccup is visible.
Admin notification includes workspace org name, email, UTC + Central
timestamp, client IP (CF-aware), CF-IPCountry, and user agent.
Slice 1 + 3 of the user-management feature from the May 12 plan.
Backend-only - no UI yet (slice 2 ships separately). Backend +
accept-handler together so the email accept link is functional
from day one without a half-state.
Endpoints added:
- GET /api/workspaces/:id/members (any member; via_org=true
for org-level entries,
read-only from ws context)
- GET /api/workspaces/:id/invites (workspace_admin)
- POST /api/workspaces/:id/invites (workspace_admin)
- DELETE /api/workspaces/:id/invites/:inviteId (workspace_admin)
- PUT /api/workspaces/:id/members/:userId (workspace_admin)
- DELETE /api/workspaces/:id/members/:userId (workspace_admin)
- POST /api/auth/accept-invite/:inviteId (requireAuth +
case-insensitive
email match)
Permission gating:
- canAdminWorkspace (existing) for admin-gated endpoints
- canAccessWorkspace (new helper in lib/permissions.js) for the
members read endpoint - mirrors canAdminWorkspace shape but
admits any workspace_members role plus org/platform paths
Security additions vs the original plan:
- Transaction-bounded collision check on POST /invites closes the
TOCTOU race between simultaneous duplicate POSTs (no UNIQUE
constraint on workspace_invites(workspace_id, email))
- Per-(inviter, workspace), hour-window rate limit on POST /invites
to prevent abuse / cost runaway. Env-configurable via
INVITE_RATE_LIMIT_PER_HOUR with conservative 50/hour default.
429 response is generic - does not echo the configured value.
- Invite expiry env-configurable via INVITE_EXPIRY_DAYS (default 7)
- PUBLIC_URL env var (optional) pins the accept-URL origin in prod;
falls back to request-derived for local dev
Rollback rule on email send: only graph_error (real send attempt
failed at Graph) deletes the row and returns 502. not_configured
and dev_restricted are intentional non-sends - keep the row, count
against rate limit, allow local accept-invite testing to proceed.
Other safety blocks:
- Cannot demote/remove the last workspace_admin (409)
- Cannot remove the parent-org's org_owner via workspace path (403)
- Accept-invite is idempotent if user already a member
- Expired invites delete-on-read and return 410
- Wrong-account accept returns 403 without touching the invite
Expired-invite cleanup added to services/heartbeat.js mirroring
the team_invites sweep pattern.
Verification: 9-case curl-driven E2E against the dev DB fixture
(switcher-test + invitee-existing + invitee-new mid-flow register).
All 9 pass: create / collision-409 / second-create / rate-limit-429 /
existing-user-accept / register-then-accept / wrong-account-403 /
expired-410 / viewer-cannot-invite-403.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Short-lived per-device queue covers the TV-flap window (issue #3):
when a device is mid-reconnect, prior code emitted to an empty room
and the event vanished. Now playlist-updates and commands targeting
an offline device are queued and flushed in order on the next
device:register for that device_id.
server/lib/command-queue.js (new):
- pendingPlaylistUpdate: per-device marker (rebuild via builder on
flush -> always fresh DB state, no stale snapshots)
- pendingCommands: per-device Map<type, payload> with last-of-type
dedup (most recent screen_off wins)
- TTL via COMMAND_QUEUE_TTL_MS env (default 30000)
- Active sweep every 30s prunes expired entries
Memory bounds: ~6 entries per device worst case (1 playlist marker
+ 5 command types), unref'd sweep timer.
Wired emit sites (8 total; the four direct socket.emit calls in
deviceSocket register handlers are intentionally NOT queued because
the socket is alive by definition at those points):
- server/routes/video-walls.js (pushWallPayloadToDevice)
- server/routes/device-groups.js (pushPlaylistToDevice)
- server/routes/content.js (content-delete fan-out)
- server/routes/playlists.js (pushToDevices + assign)
- server/services/scheduler.js (scheduled rotations)
- server/ws/deviceSocket.js x2 (wall leader reclaim/reassign)
server/ws/deviceSocket.js register paths now call flushQueue after
heartbeat.registerConnection + socket.join. Existing
socket.emit('device:playlist-update', ...) lines kept - they send
the initial state on register; the flush replays any queued events.
Player's handlePlaylistUpdate fingerprint check dedupes the
overlap.
Refs #3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously sendEmail() only logged on error/suppression paths; success
was silent. After prod deploy of c71c401 it was unclear whether the
first alert tick had actually delivered email or not - the answer was
yes but had to be derived from 'no error log + recipient query showed
matching device'. Add a log line on success so future observability
doesn't require detective work.
Replaces the unused EMAIL_WEBHOOK_URL stub with a real Microsoft Graph
Mail.Send pipeline via @azure/msal-node client-credentials flow. Prior
state on prod: every alert email was logged to journalctl and never
sent (21 fallback log lines per hour for the chronic-offline devices).
Four coordinated changes shipped as one commit since they're all part
of making email delivery actually work responsibly:
1. services/email.js (NEW): Graph send via plain HTTPS (no SDK), in-memory
MSAL token cache (refresh 60s pre-expiry), graceful stdout fallback
when GRAPH_* env vars absent. Drop-in replacement for the old webhook.
2. services/alerts.js refactored: sequential await around sendEmail (was
parallel fire-and-forget; first run hit Graph's MailboxConcurrency 429
ApplicationThrottled on a 30-device backlog). Sequential at ~250ms per
send takes 5-8s for the full backlog, well within the 60s tick. Also:
24h long-offline cutoff to stop nagging about chronic-offline devices
(the 20,000+ minute ones); 2-hour dedup window (was 1h) via a generic
shouldSendAlert(type, id, windowMs) helper that future alert types
(payment_failed, plan_limit_hit, etc.) can reuse.
3. Preferences UI: single checkbox in settings.js Account section bound
to users.email_alerts. Saved via the existing Save Profile button. PUT
/api/auth/me extended to accept email_alerts. requireAuth middleware
SELECT now includes email_alerts so it propagates via req.user.
4. Dev safety net: GRAPH_DEV_RESTRICT_TO env var as an allow-list. When
set, only listed recipients reach Graph; everyone else is suppressed
with a log line. Prevents local dev (which often runs against fresh
prod DB copies) from accidentally emailing real prod users. UNSET on
prod systemd unit so production fans out normally.
Also: package.json scripts use --env-file-if-exists=.env so local dev
picks up .env automatically (Node 20.6+ built-in, no dotenv dep). Prod
runs via systemd ExecStart and is unaffected. server/.gitignore added
to keep .env out of git.
Smoke verified end-to-end:
- Sequential send pattern verified (a prior parallel-send tick had hit
Graph's MailboxConcurrency 429 on 30 simultaneous sends; sequential
at ~250ms each completes the same backlog without throttling)
- 24h cutoff silenced 20/21 prod devices on the next tick
- Dev restrict suppressed the 1 within-24h send
- User-preference toggle flipped via UI -> DB -> alert path silently
continued before reaching even the suppression log
Fix: at connect, enumerate the user's accessible workspace_ids (direct workspace_members + org_owner/admin paths + platform_admin 'all') via new accessibleWorkspaceIds() helper in lib/tenancy.js; socket.join one room per workspace. All 12 dashboardNs.emit sites across deviceSocket / heartbeat / server.js / devices route / video-walls route now route via dashboardNs.to(workspaceRoom(...)).emit() with the workspace looked up from the relevant device or wall. New lib/socket-rooms.js holds the helpers and breaks a circular dependency (dashboardSocket already requires heartbeat, so heartbeat can't require dashboardSocket).
Inbound 6 commands rewired to canActOnDevice(socket, deviceId, tier): request-screenshot is read tier (workspace_viewer+); remote-touch/key/start/stop and device-command are write tier (workspace_editor+). Platform_admin and org_owner/admin always pass via actingAs. Legacy admin/superadmin branch dropped.
Lifecycle note: workspace-switch already calls window.location.reload (Phase 3 switcher), which forces a fresh socket with updated memberships - no per-emit re-evaluation needed.
Smoke tested with 3 simultaneous socket.io-client connections (switcher-test, swninja, dw5304 platform_admin) + direct canActOnDevice invocation for 6 user/device/tier combinations. All 9 outbound isolation cells and all 6 permission gates pass. Fixture mutation: switcher-test's Field Crew membership flipped from workspace_editor to workspace_viewer to exercise the read/write tier split in one login.
KNOWN REGRESSION (Phase 3 fix): platform_admin / superadmin no longer has cross-workspace 'see everything' view. Every route migrated tonight (2.2a-2.2m) deliberately removed the role-based bypass per design doc - cross-workspace visibility will come via dedicated admin endpoints in Phase 3, not magic role bypasses. Until Phase 3 ships, platform admins must switch-workspace to see other workspaces' data.
Express's req.ip was resolving to a Cloudflare edge address (e.g.
172.70.x.x) for any request fronted by Cloudflare, because trust proxy
was set to '1' — that trusts the immediate hop, which IS Cloudflare.
All activity_log rows from API paths captured the proxy, not the
client. The WebSocket path was unaffected and recorded the real IP.
Two layers of defense:
1. trust proxy now lists Cloudflare's published v4 + v6 ranges plus
loopback / linklocal / uniquelocal (config/cloudflareIps.js). With
this list req.ip resolves to the original client when fronted by
CF, and X-Forwarded-For from any non-trusted source is ignored —
so the value can't be spoofed.
2. New getClientIp(req) helper in services/activity.js prefers the
CF-Connecting-IP header but only honors it when the immediate TCP
peer is itself a trusted address. Same gate as trust proxy, so a
visitor who hits the origin directly with a forged header is
logged at their real address.
Routed all five activity-log call sites (auth login success/failure,
admin password reset, generic activityLogger middleware, and the
in-memory rate-limiter key) through the helper.
Logging-only change. No schema changes. Existing rows are not
modified — fix applies to new entries going forward.
Verified locally:
- Bare loopback hit logs 127.0.0.1 (not a proxy address).
- Helper unit cases including an untrusted peer (203.0.113.7) sending
a forged CF-Connecting-IP correctly fall back to the real peer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 group scheduling: schema migration adds group_id to schedules with
CHECK constraint, scheduler evaluates group+device schedules with priority,
group deletion converts schedules to per-device copies. Dashboard gets
playlist assignment dropdown and current playlist label on group headers.
Player persists audio unlock state in localStorage so version reloads
don't lose audio on unattended displays.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Schedule CRUD now includes playlist_id field. List queries join playlist name.
Scheduler tracks active overrides per device and reverts to original
playlist/layout when no schedule is active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces the assignments-table query with a playlist_items query keyed on
device.playlist_id. Also eliminates the duplicate payload builder in
scheduler.js — it now calls the shared buildPlaylistPayload.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ScreenTinker - open source digital signage management software.
MIT License, all features included, no license gates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
