Every new subsystem is disable-able via env (flip + restart, no redeploy/bisect):
- FLAP_LIMITER_ENABLED=false -> flap limiter always allows.
- OTA_DOWNLOAD_GUARD_ENABLED=false -> download guard always admits.
- MAINTENANCE_BAND_GATE_ENABLED=false -> interval maintenance ignores band.
- CONNECT_RATE_QUARANTINE_TRIPS=0 -> quarantine off (already; confirmed).
Startup prune is never band-gated regardless. Kill switches table added to the fallout
doc. Tests assert each OFF behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The death-spiral amplifier: pruneStatusLog ran a whole-table ROW_NUMBER() sort,
40-48s synchronous on the 1.1M-row incident table, freezing boot -> healthcheck
fail -> restart loop.
- lib/chunked-prune.js: shared chunkedDelete (rowid IN (SELECT ... LIMIT ?) since
better-sqlite3 has no DELETE...LIMIT) — bounded batch + setImmediate yield between
batches, optional band-gate. Core invariant: no sync op blocks >~50ms ever.
- pruneStatusLog: rewritten per-device via a loose index-scan seek
(WHERE device_id > ? ORDER BY device_id LIMIT 1 — O(log n) each), retention +
newest-cap trimmed in bounded batches, async, re-entrancy-guarded, band-gated on
the interval / un-gated + fire-and-forget at startup so a bloated table self-heals
on deploy WITHOUT freezing boot.
- heartbeat.js: maintenance moved off the interval body into async band-gated
re-entrant runMaintenance(); play_logs + provisioning prunes chunked; offline-marking
stays synchronous.
- pruneTelemetry: bounded single statement (OFFSET 6000 LIMIT batch), stays sync.
- idx_devices_provisioning so the provisioning prune batch subquery is an index range.
Tests: correctness (per-device cap + retention, independent devices), 300k-row backlog
trims in many batches with max event-loop gap <250ms, band-gate no-op while critical +
startup runs regardless, re-entrancy (concurrent -> once). Existing prune tests updated
to await. Suite 247/247.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
services/heartbeat.js deleted unclaimed provisioning devices with
created_at < now - (365 * 86400) — a YEAR — while its own comment said "older
than 24 hours". So socket-register pairing junk lingered ~365x longer than
intended. Change the window to 24 * 3600 to match the comment.
Correctness fix only — does NOT touch the pre-auth register path or add a rate
limiter (that pre-auth hardening is a separate security issue, out of this cut).
Extracted the sweep into pruneProvisioningDevices() (still in heartbeat.js, called
from the same interval) so it is unit-testable. Test asserts a >24h unclaimed
provisioning row is swept while a <24h row, an imported row (user_id set), and a
non-provisioning row are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-device insert-time prune (deviceSocket.js) only ever touches a device
that is actively inserting, so it misses two paths: removed/idle devices whose
rows linger forever, and heartbeat.js's offline_timeout insert that bypasses
logDeviceStatus entirely. The reporter's 1.2M-row bloat accumulated UNDER a 7-day
per-device prune for exactly this reason.
- pruneStatusLog() (db/database.js): a GLOBAL time-range sweep across ALL devices,
modeled on the play_logs prune. Run once on startup (recovers a bloated table
right after deploy) and on the heartbeat interval (services/heartbeat.js).
- STATUS_LOG_RETENTION_DAYS env, default 3 (lower than the old hardcoded 7d; the
dashboard only shows a 24h uptime window, so 2-3d is ample for diagnostics).
- Deliberately NO per-device row cap: Step 3's throttle already bounds how fast a
storming device can generate status rows, so a cap would add sweep complexity
for little gain (noted for later if needed).
- NO VACUUM / auto_vacuum here (kept off the hot path); space reclaim is left as a
separate decision (see report).
test: deterministic in-process unit test proves the sweep deletes over-retention
rows across all devices — including a device absent from the devices table and an
offline_timeout row — while keeping recent rows; idempotent on an empty table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Slice 1 + 3 of the user-management feature from the May 12 plan.
Backend-only - no UI yet (slice 2 ships separately). Backend +
accept-handler together so the email accept link is functional
from day one without a half-state.
Endpoints added:
- GET /api/workspaces/:id/members (any member; via_org=true
for org-level entries,
read-only from ws context)
- GET /api/workspaces/:id/invites (workspace_admin)
- POST /api/workspaces/:id/invites (workspace_admin)
- DELETE /api/workspaces/:id/invites/:inviteId (workspace_admin)
- PUT /api/workspaces/:id/members/:userId (workspace_admin)
- DELETE /api/workspaces/:id/members/:userId (workspace_admin)
- POST /api/auth/accept-invite/:inviteId (requireAuth +
case-insensitive
email match)
Permission gating:
- canAdminWorkspace (existing) for admin-gated endpoints
- canAccessWorkspace (new helper in lib/permissions.js) for the
members read endpoint - mirrors canAdminWorkspace shape but
admits any workspace_members role plus org/platform paths
Security additions vs the original plan:
- Transaction-bounded collision check on POST /invites closes the
TOCTOU race between simultaneous duplicate POSTs (no UNIQUE
constraint on workspace_invites(workspace_id, email))
- Per-(inviter, workspace), hour-window rate limit on POST /invites
to prevent abuse / cost runaway. Env-configurable via
INVITE_RATE_LIMIT_PER_HOUR with conservative 50/hour default.
429 response is generic - does not echo the configured value.
- Invite expiry env-configurable via INVITE_EXPIRY_DAYS (default 7)
- PUBLIC_URL env var (optional) pins the accept-URL origin in prod;
falls back to request-derived for local dev
Rollback rule on email send: only graph_error (real send attempt
failed at Graph) deletes the row and returns 502. not_configured
and dev_restricted are intentional non-sends - keep the row, count
against rate limit, allow local accept-invite testing to proceed.
Other safety blocks:
- Cannot demote/remove the last workspace_admin (409)
- Cannot remove the parent-org's org_owner via workspace path (403)
- Accept-invite is idempotent if user already a member
- Expired invites delete-on-read and return 410
- Wrong-account accept returns 403 without touching the invite
Expired-invite cleanup added to services/heartbeat.js mirroring
the team_invites sweep pattern.
Verification: 9-case curl-driven E2E against the dev DB fixture
(switcher-test + invitee-existing + invitee-new mid-flow register).
All 9 pass: create / collision-409 / second-create / rate-limit-429 /
existing-user-accept / register-then-accept / wrong-account-403 /
expired-410 / viewer-cannot-invite-403.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix: at connect, enumerate the user's accessible workspace_ids (direct workspace_members + org_owner/admin paths + platform_admin 'all') via new accessibleWorkspaceIds() helper in lib/tenancy.js; socket.join one room per workspace. All 12 dashboardNs.emit sites across deviceSocket / heartbeat / server.js / devices route / video-walls route now route via dashboardNs.to(workspaceRoom(...)).emit() with the workspace looked up from the relevant device or wall. New lib/socket-rooms.js holds the helpers and breaks a circular dependency (dashboardSocket already requires heartbeat, so heartbeat can't require dashboardSocket).
Inbound 6 commands rewired to canActOnDevice(socket, deviceId, tier): request-screenshot is read tier (workspace_viewer+); remote-touch/key/start/stop and device-command are write tier (workspace_editor+). Platform_admin and org_owner/admin always pass via actingAs. Legacy admin/superadmin branch dropped.
Lifecycle note: workspace-switch already calls window.location.reload (Phase 3 switcher), which forces a fresh socket with updated memberships - no per-emit re-evaluation needed.
Smoke tested with 3 simultaneous socket.io-client connections (switcher-test, swninja, dw5304 platform_admin) + direct canActOnDevice invocation for 6 user/device/tier combinations. All 9 outbound isolation cells and all 6 permission gates pass. Fixture mutation: switcher-test's Field Crew membership flipped from workspace_editor to workspace_viewer to exercise the read/write tier split in one login.
ScreenTinker - open source digital signage management software.
MIT License, all features included, no license gates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
