Bold reported loop lag that grew with uptime and reset on restart. The
signature they saw — mean = p50 = p99 = max, identical to two decimals —
is not a fixed cost paid on every cycle. It is what an IntervalHistogram
window reports when it recorded exactly ONE delay: the mean is the raw
value, and every percentile returns the bucket ceiling above it.
Reproduced against their exact numbers (1329.07 / 1329.59). So the loop
took one long turn that swallowed the sampling second, episodically —
which is what they later confirmed independently.
The turn is ours, and it is now measured rather than theorised. Probing
the real worker against a real WAL with one reader mid-transaction: a
single main-thread write blocked for 4,936ms behind the worker's
wal_checkpoint(TRUNCATE), which then reported WAL 8.8MB -> 8.8MB. TRUNCATE
is the blocking form and its locks are held ACROSS connections, so moving
it to a worker kept the fsync off the loop but not the lock; and it does
not throw when it cannot get those locks, it returns busy=1 having sat on
SQLite's 5s busy timeout and reclaimed nothing. Five seconds of stalled
loop for zero benefit, and silent.
It was reached far too easily. The rule was "escalate if the WAL grew
across three consecutive 15s runs" — which any sustained 45-second write
burst satisfies. A customer's fleet powering on in the morning does it
daily.
Two gates, because either alone leaves the hole open. A size FLOOR, so a
WAL in the lower half of its budget can't buy a blocking checkpoint it has
nothing to reclaim from. And a COOLDOWN, because the floor alone fixes
nothing for Bold — their WAL already sits at 6.2MB against a 16MB
high-water, above any sane floor, so every burst would still escalate.
However long the pressure lasts, our own maintenance may now stall the
loop at most once per window. The high-water rule bypasses both and is
untouched: a runaway WAL is the one case worth blocking for, so the
"WAL cannot grow forever" invariant is exactly as strong as before.
A busy TRUNCATE now says so in the log instead of reading like a success.
Also softened the adjacent path: when the worker is declared
unrecoverable, engageFallback() re-arms inline autocheckpoint on the main
connection — a state that is STICKY for the life of the process, i.e.
exactly the shape of "degrades with uptime, a restart fixes it". It used
to also run an unconditional main-thread TRUNCATE on the way in; that now
happens only when the WAL is genuinely over high-water, and the fallback
state is served on /api/status rather than being inferable only from a log
line that may have rolled.
Telemetry, so the next report is self-explanatory: loop_lag carries
`samples` (~50 when healthy, 1 when a single turn swallowed the second),
`tick_gap_ms` measured on the WALL CLOCK independently of the histogram,
and `worst_tick_gap_ms`/`worst_tick_at` — monotone, so five-minute polling
can no longer miss an episode.
Band semantics are deliberately unchanged. A one-sample window during a
real stall is the correct trigger for the shed valve; suppressing it would
blind the protection at exactly the moment it is needed.
Separately, device_telemetry gets the age sweep it never had. The
per-heartbeat row cap only ever trims the device whose heartbeat is being
handled, so a device that STOPS reporting leaves its rows behind forever.
The new sweep is per-device (rides idx_telemetry_device rather than
scanning), chunked and yielding like the device_status_log one, and
defaults to 30 days to match the uptime report's own default window — so
it cannot remove rows that report would have shown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014skWYXJUWhF73EvNPgB2AS
The on-device settings PIN (devices.settings_pin, minted at pairing) and the pairing code
assigned to imported devices both came from
`Math.floor(100000 + Math.random() * 900000)`.
Math.random is not a CSPRNG. V8 implements it as xorshift128+, whose internal state is
recoverable from a handful of consecutive outputs, and every call in a process draws from
that one shared stream. Both values are also observable by ordinary users — settings_pin
is returned in device API responses today — so a user who collects a few outputs could
predict the values minted around them, including for other tenants.
lib/numeric-code.sixDigitCode() uses crypto.randomInt, which is CSPRNG-backed and
rejection-samples so the distribution stays uniform. Range is 100000..999999 inclusive,
identical to the old expression, so codes are still exactly six digits with no leading
zero — the on-device keypad and pairing UI are unchanged.
Deliberately NOT converted, because neither gates access: the image-generation seed in
lib/image-gen.js, and the anti-burn-in pixel jitter inside generated widget HTML.
Also unchanged: the settings_pin backfill in db/database.js, which uses SQLite's random()
— that is ChaCha20 seeded from OS entropy, not a weak PRNG.
This is the generator half of the finding only. The separate half — that settings_pin is
returned to every workspace member, including read-only roles — is a response-shape change
and waits on the consumer enumeration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uploaded files are served from the SAME ORIGIN as the dashboard, so how a browser
interprets them is a security boundary. Two things decided that interpretation, and
both were caller-controlled: the stored extension came from
`path.extname(originalname)`, and the only type check read `file.mimetype` — a request
header. A caller could therefore choose to have their bytes served as an active
document from the app origin.
Two independent invariants now hold the boundary:
1. INGEST — lib/upload-sniff.js sniffs magic bytes after multer writes a neutral
`.part` file (diskStorage names the file before any bytes exist, so the sniff cannot
happen there), maps the result through a hardcoded mime->extension allowlist, renames
accordingly, and stores the sniffed mime. Unsupported bytes are refused with a 400.
2. SERVING — upload responses carry `Content-Security-Policy: sandbox`, so if a response
is ever treated as a document it lands in an opaque origin with scripts disabled.
Anything outside the inline-safe extension set is additionally forced to download.
This holds regardless of how a file reached disk, so a future gap in (1) is contained
rather than exploitable.
Applied at every instance of the pattern, not just the first: lib/content-ingest.js,
the /replace route, the four content-serving paths across server.js and routes/content.js
(the latter pair currently shadowed by mount order, which is not a guarantee), and the
ZIP-import path in routes/status.js, which took its extension from the archive entry.
SVG stays accepted and stays inline: white-label logos are SVG, and octet-stream +
nosniff makes <img> fail. Scripts in an SVG never run in an image context, and the
sandbox CSP covers the one case where they would — a direct navigation. SVG is also no
longer handed to sharp, which removes the librsvg path where the open libvips CVEs live.
Existing rows are untouched — no migration. The four upload fixtures in agency.test.js
uploaded `Buffer.from('x')` declared as image/png; that is the exact "declared type is a
lie" case this closes, so the fixtures now use real PNG bytes. No assertion changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six places verified a session JWT inline instead of going through requireAuth,
each repeating a slightly different subset of its checks. Introduce
resolveSessionUser() in middleware/auth.js as the single definition of "this
token is a usable session, and here is whose it is", and route all of them
through it: the three /api/status token routes, the screenshot route, the
content-reference gate, and the /dashboard socket handshake. requireAuth is now
a thin wrapper over the same helper, so the two cannot drift.
Also:
- Give the pre-TOTP token a distinct audience so it is redeemable only through
verifyMfaPendingToken (POST /api/auth/totp/verify). verifyToken refuses any
token carrying an audience, so a token minted for one purpose cannot be
redeemed on another path.
- The dashboard socket handshake now takes userId/userRole from the live users
row rather than from the token claim, so role changes take effect on the next
connection instead of riding the token's remaining lifetime.
- Add test/session-token-resolution.test.js covering all six surfaces,
including the socket handshake.
Every call site keeps the status code and error body it returned before.
Net query cost: the content-reference gate and the socket handshake each gain
one users-by-id lookup (the same one requireAuth already does per request); the
other four are unchanged or replace an equivalent lookup.
In-flight pre-TOTP tokens are invalidated by the audience change; they live 5
minutes, so the window is a re-login at worst.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1. devices_connected (always on, never gated): a top-level /api/status field next to
loop_lag = LIVE WS socket count from the heartbeat connection map (getConnectedCount),
NOT devices.status='online' (which lags by the offline-timeout). The single
most-glanced operational number, so it can't disappear when debug is off. Also dropped
4 dead per-poll COUNT(*) queries the route computed but never returned.
2. debug block behind an admin flag: new minimal app_settings KV table (none existed;
ai_settings is per-workspace, white_labels is branding) + lib/app-settings.js (cached,
refresh-on-write so status polls read a cached boolean, not a DB row).
routes/status.js includes `debug` ONLY when status_debug_enabled is on (persisted value
overrides the STATUS_DEBUG_ENABLED env default); when off the key is omitted entirely.
3. Admin toggle: GET/PUT /api/admin/status-debug (requirePlatformAdmin, mirrors the
branding endpoints) + a checkbox in the Admin tab "Status endpoint" section
(mirrors the branding checkbox). Takes effect on the next poll, no restart.
Tests: devices_connected always present+numeric and rises with a live socket (booted +
socket.io-client); debug present by default, admin flips OFF -> key omitted (loop_lag +
devices_connected remain) -> ON again, no restart; non-admin 403, anon 401; unit coverage
for getConnectedCount + app-settings default/override. Suite 289/289.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the new internal states so we can SEE the limiters biting during the alpha soak
instead of grepping logs. /api/status now carries debug: {
flap: {buckets, quarantined},
ota_download: {inFlight, servedThisWindow, shedThisWindow, windowCount},
maintenance: {deleted, ms, at, running}, // last status-log prune
log_coalescer_buffer,
}. Aggregate counts only (no device ids/secrets), cheap in-memory reads. stats()
added to flap-limiter + ota-download-guard (singleton prod state), getMaintenanceStats
from database. Asserted in the booted /api/status test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuously samples event-loop delay via perf_hooks.monitorEventLoopDelay()
(C++-backed histogram; cheap). Each window persists mean/p50/p99/max to a new
event_loop_lag table and recomputes a coarse load band (normal/elevated/critical)
from the window p99. Standalone value: current lag is exposed on /api/status and
band changes are logged, so site lag is diagnosable independent of throttling.
The band feeds the #142 reconnect throttle (next commit) but ships first as its
own subsystem.
- event_loop_lag is bounded from day one: indexed on sampled_at + scheduled prune
(LAG_TELEMETRY_RETENTION_DAYS, small default) modeled on the play_logs prune.
Deliberately NOT another unbounded-growth table.
- Band transitions are asymmetric: jump up immediately (tighten fast), release one
level at a time after N calm samples below a deadband (release slow, no flap).
Pure nextBand() function, unit-tested deterministically.
- config: LAG_SAMPLE_INTERVAL_MS, LAG_RESOLUTION_MS, LAG_TELEMETRY_RETENTION_DAYS,
LAG_PRUNE_INTERVAL_MS, LAG_ELEVATED_MS, LAG_CRITICAL_MS, LAG_RELEASE_SAMPLES.
- tests: band-transition unit tests; integration proves sampling persists, stays
bounded under the prune, and surfaces on /api/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- server/version.js: shared version helper that reads the root VERSION file once
(fallback 0.0.0). Replaces the stale hardcoded 1.2.0 / 1.5.1 / 1.0.0 fallbacks
in /api/version, /api/update/check, and /api/status.
- config.js: DATA_DIR / DB_PATH / UPLOADS_DIR / CERTS_DIR env overrides for the
db, uploads, and certs/jwt-secret locations. Unset resolves to exactly the
legacy in-repo paths, so existing installs (including production) are
byte-for-byte unchanged. Guarded by test/config-paths.test.js.
- package.json: rename remote-display-server -> screentinker (+ lockfile name).
- scripts/bump-version.sh: one-shot bump across VERSION, package.json (+lock),
android (versionName and versionCode + 1), and the tizen widget version; makes
one commit plus an annotated tag; prints the push command, never pushes.
- .gitignore: global *.db / *.db-wal / *.db-shm / *.db.* so no database file
(including .db.devbak backups, at any path) can be committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4 group scheduling: schema migration adds group_id to schedules with
CHECK constraint, scheduler evaluates group+device schedules with priority,
group deletion converts schedules to per-device copies. Dashboard gets
playlist assignment dropdown and current playlist label on group headers.
Player persists audio unlock state in localStorage so version reloads
don't lose audio on unattended displays.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration (database.js): switched from sync execFileSync to async execFile
with promise wrapper, matching the pattern in playlists.js. Probes each
video content item, backfills content.duration_sec, and uses the real
duration in playlist_items. Falls back to the assignment's original
duration_sec if the probe fails or content isn't a video.
V1 import (status.js): moved assignment-to-playlist conversion out of the
synchronous db.transaction() so async ffprobe can run. Content files are
already on disk from the transaction, so probing works. Same fallback logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Export now includes playlists, playlist_items, and device.playlist_id.
Format bumped to screentinker-export-v2. Schedules include playlist_id.
V1 import converts old assignments array into per-device playlists inline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ScreenTinker - open source digital signage management software.
MIT License, all features included, no license gates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
