Three changes, all about the same failure: sharing appears to be on while nothing
actually arrives.
SEND ON OPT-IN. Turning sharing on now reports immediately instead of waiting for
the next daily tick. Two reasons: the operator is standing right there, and
"nothing has been sent" for the next 24h reads as broken at exactly the moment
someone is checking whether it works. It also means an egress-filtered network
fails HERE, where we can name the host to unblock, rather than silently tonight
where nobody is watching.
NAME THE FAILURE. Failed attempts are now recorded separately from successes, so
Settings can say which address did not answer and why, instead of showing an empty
"nothing sent yet". A blocked outbound connection is the normal failure on a
self-hosted box and is otherwise completely invisible — the operator cannot tell a
firewall from a broken feature. A later success clears the complaint, so a stale
warning never outlives the problem it describes. Docs gained a section on it, and
the UI states plainly that nothing needs opening inbound.
OPERATOR COLLECTOR, ADDITIVE. TELEMETRY_EXTRA_ENDPOINT lets an operator post the
same three fields to their own collector.
The naming is the point. It replaces TELEMETRY_ENDPOINT, which was a true override
— and an override is the wrong shape here, because a variable called "endpoint"
that silently redirected the report someone agreed to SHARE would make the opt-in
mean something other than what the UI says. Our address is hard-wired and not
overridable; theirs is explicitly additional and named so it cannot be mistaken for
a replacement. Settings lists every destination a report goes to.
The operator collector is independent of the sharing switch, because it is their
server posting to their host and our opt-in has no business gating it. So an
operator who wants internal fleet numbers with nothing leaving for us sets it and
leaves sharing off — supported on purpose, and tested.
Destinations are attempted separately: one unreachable collector must not cost the
other its report.
1662/1662 pass. Tests pin the properties that matter: an operator collector never
replaces the shared report, sharing-off still sends nothing to us whatever else is
configured, one dead destination does not stop the other, and a failure records the
address actually tried.
There is no way to answer "how many screens run ScreenTinker?". The product is
self-hostable by design, so most installs are invisible to us on purpose — and
should stay that way. This asks once, and reports only if the operator says yes.
The entire payload is three fields:
{ instance_id, version, screen_count }
instance_id is a random UUID minted on first use and kept in app_settings. It
carries nothing about the install; its only job is to let two reports from the
same server be recognised as one server, so a count is a count rather than a sum
of duplicates. That makes a report pseudonymous rather than anonymous, and the
wording shown to operators says so rather than claiming otherwise.
The payload is short on purpose. Every field added costs participation, and
participation is the only thing that makes the resulting number worth quoting.
Player-platform counts were considered and left out: release assets are already
published per platform, so GitHub's per-asset download counts answer "where should
effort go" at zero privacy cost and without asking anyone for anything.
Verifiability is the feature, not the copy. Settings shows the ACTUAL payload this
server would send, generated live from its own data, plus what it last really sent
and when. The payload is built in one function so a reviewer can check it at a
glance, and the test fails if a field is ever added.
Both answers persist. Declining is remembered as 'off' rather than falling back to
'unasked', so the prompt cannot return after an update — re-prompting is how
telemetry earns its reputation and gets patched out.
Collector side is inert unless TELEMETRY_COLLECTOR=1, so a normal install never
exposes the endpoint. Reports upsert on instance_id rather than appending, so an
install reporting daily occupies one row rather than 365 a year. The source IP is
never read or stored — receiving one is unavoidable, logging it would quietly turn
a pseudonymous report into an identifiable one.
Tests pin the negative promises, which are the ones that rot silently: sends
nothing before consent, sends nothing after a decline, payload is exactly three
keys, id survives a restart, a failed send never records a phantom report. Screen
count excludes unpaired provisioning rows, which would otherwise overstate the one
number this exists to state honestly.
docs/telemetry.md documents the payload, what is not sent, how to verify it, and
that any published total is a floor rather than a basis for extrapolation.
1657/1657 pass.
getExternalFilesDir() returns null whenever external storage is unavailable, and
on a signage panel that is not exotic: no emulated volume, a vendor ROM that never
mounts one, an ejected card, storage still unmounted early in boot.
Both APK download paths did:
File(context.getExternalFilesDir(DIRECTORY_DOWNLOADS), name)
Java's File(File, String) treats a null parent as "no parent" and silently yields
a RELATIVE path, so the download targeted `ScreenTinker-x.y.z.apk` in the process
working directory — `/` — which is not writable. The write threw, the generic
catch swallowed it, and the caller reported only "failed to download or failed
signature verification".
That message is why this was expensive to find. The HTTP request SUCCEEDS (the
server logs a served download at the exact moment of each failure), so it does not
look like a download problem; the signing key is fine, so it does not look like a
verification problem; and because nothing is ever written there is no partial file
to find. It also never recovers — every attempt fails the same way, forever.
installFromUrl (the dashboard "Push an APK" button) carried the identical line, so
the obvious workaround for a panel in this state was broken by the same bug.
Both now use apkDir(), which falls back to internal storage. filesDir cannot be
unmounted: if it is gone the app is not running.
file_paths.xml gains a <files-path> for the same directory. The silent
PackageInstaller path streams the file itself and needs nothing there, but the
intent-based install FALLBACK resolves it through FileProvider and would throw
"Failed to find configured root" — turning an already-degraded panel into one that
cannot install at all.
⚠️ This cannot reach an affected panel over the air: the broken download path IS
the delivery mechanism, and Push APK shares the bug. A panel already in this state
needs one manual install to escape it.
Root cause is inferred from converging evidence on an Android 9 panel (HTTP served
at each failure, no APK anywhere on the device, the app's own external files dir
denied to its own UID, both paths failing identically, signing verified at parity
against the release it already installed). Handling a null return is correct
regardless — it must never become a relative path.
Covers everything since alpha6: the stage-sizing fix (#262), the operations
runbook (#261), the sharp removal (#263) and the better-sqlite3 pin (#264/#265),
plus the dependency-reinstall requirement that applies in both directions.
* Document the Node.js upgrade procedure and this build's reinstall requirement
Upgrading the runtime does not go through scripts/upgrade.sh, so nothing
reinstalls dependencies — which is precisely when the one remaining native
module goes stale. The runbook now covers the version floor imposed by
--env-file-if-exists, why the better-sqlite3 pin is exact, and why a version
without a matching prebuild can turn Restart=always into a boot loop.
Also records that this build changes dependencies in both directions: rolling
back past it needs the reinstall too, because earlier builds import sharp at
runtime and this one drops it from production dependencies.
Kept deployment-neutral — no hostnames, addresses, or environment specifics.
* Fix the only test that fails on Node 22
Node 22 added a built-in `navigator` global, defined as a getter with no
setter. The test's shim assigned to it, which throws "only a getter" under
'use strict' on 22 while being a normal assignment on Node 20, where the global
does not exist at all. It is configurable, so define it instead of assigning.
Defining it unconditionally is also the better fixture: Node 22's own navigator
reports the HOST locale, so a test reading its language would otherwise depend
on the machine or CI runner it happens to run on.
This was the single failure in an otherwise clean Node 22 run (1639/1640 with
better-sqlite3 12.9.0), and it is confined to test code — no production server
or frontend file assigns to globalThis.navigator.
1649/1649 on Node 20.
Prepares for the Node 22 move by decoupling it from the database driver, so the
two upgrades land as independently reversible steps rather than one flag day.
9.6.0 cannot work on Node 22. It uses the raw V8 API (220 v8:: references, zero
napi_), so it is ABI-locked per Node major, and its GitHub release assets carry
prebuilds for ABI 108/115/120 only — nothing for Node 22's 127. Its install script
is `prebuild-install || node-gyp rebuild --release`, so on Node 22 it silently
falls through to compiling raw-V8 code against Node 22 headers. That is not just
slow: lib/preflight-deps.js rebuilds synchronously before the server listens, and
prod's systemd unit is TimeoutStartSec=90 with Restart=always, so a slow or failing
compile is an unbootable loop rather than the intended self-heal.
12.9.0 ships prebuilds for BOTH Node 20 (ABI 115) and Node 22 (127), so neither the
current runtime nor the target has to compile anything.
THE PIN IS EXACT ON PURPOSE — ^12.9.0 would defeat it. 12.10.0 dropped the Node 20
prebuild while still advertising "20.x" in engines, so a caret resolves to 12.11.x
and reintroduces the from-source compile on today's runtime. Verified against the
release assets per version:
12.0.0 / 12.2.0 / 12.4.5 / 12.6.2 / 12.9.0 ABIs 115,127,...
12.10.0 / 12.10.1 / 12.11.1 ABIs 127,137,141,147 — no 115
The reasoning is recorded in preflight-deps.js, which is where anyone hitting the
matching failure will already be reading.
13.x was considered and rejected FOR NOW: it is the first N-API release, which ends
the per-major ABI problem for good (8 prebuilds keyed by platform, not ABI) and is
where we should eventually land — but engines is ">=22", so it cannot be adopted
while prod, alpha and CI all run Node 20. It is also three weeks old with three
patch releases, which is young for the one component that owns all the data.
No API changes to absorb: every major from 10 to 13 bumped only for dropping EOL
Node/Electron versions, so the ~1486 .prepare(), 46 .transaction() and 59 .pragma()
call sites are untouched.
Verified on Node 20: installed from a PREBUILT binary (no obj.target, so no
compilation), opens a real database, and the WAL path the #149 checkpointer depends
on still works — journal_mode=WAL engages on a file DB, pragma(...,{simple:false})
returns the expected shape, and a second connection from another handle reads and
runs wal_checkpoint(TRUNCATE). 1649/1649 tests pass.
* spike: replace sharp with pure-JS image ops (jimp + jsquash WASM)
Removes the last native dependency from the ingest path, so the server no longer
needs a per-platform/per-ABI prebuilt to thumbnail an image. Motivated by getting
the server onto hardware with no toolchain, but the ABI tax is paid on every
install — it is the same failure class lib/preflight-deps.js exists to explain.
lib/image-ops.js is the whole surface: metadata() and writeThumbnail(), which are
the only two things ingest ever asked sharp for.
Format parity holds. jpeg/png/gif/tiff/bmp are native to Jimp; webp and avif go
through @jsquash WASM, whose bundled .wasm must be compiled by hand because the
packages locate it with fetch(file://) and Node has no file:// fetch — the only
symptom otherwise is a bare "fetch failed". heic is unsupported, as it already
was: sharp advertises heif but its prebuilt libvips refuses HEVC.
#170 is preserved by a different mechanism. Jimp applies EXIF orientation at
decode and rewrites the tag to 1, so metadata() reports display dimensions and
imageDisplayDims() runs as a no-op instead of swapping W/H a second time. The
helper stays in the path so the rule keeps living in one place.
Verified: 1643/1643 tests pass, and ingest was exercised in a child process with
node_modules/sharp moved aside — jpeg, EXIF-rotated jpeg, png, webp, avif, gif
all measured and thumbnailed correctly, corrupt input still yields nulls with no
phantom thumbnail_path.
KNOWN BLOCKER, do not ship as-is: Jimp is pure JS on the main thread, where sharp
handed work to a libvips threadpool. A 12MP photo goes 65ms -> 1079ms, and the
event loop stalls for 1003ms of it (sharp: zero stalls). thumbnail-backfill.js
walks a whole library at boot, so this reproduces #240 exactly — blocked loop,
missed heartbeats, panels marked offline, reconnect churn. Needs a worker_thread
offload before this is viable; image-ops.js is the seam for it.
* Run image decoding on a worker thread
Fixes the blocker the previous commit shipped with. Pure-JS decoding costs ~1s of
solid CPU for a 12MP photo, and in-process that is not a slow upload but a stalled
event loop — no heartbeats, no socket traffic. thumbnail-backfill.js walks a whole
library at boot, so it reproduced #240 (blocked loop -> missed heartbeats -> panels
offline -> reconnect churn) from our own maintenance. sharp never did this because
libvips works on a threadpool.
image-ops.js is now a dispatcher over image-ops-worker.js; the work moved unchanged
to image-ops-core.js, so callers and their failure contract are untouched.
Measured on a 12MP photo: 1079ms wall with the loop stalled 1003ms, to 1881ms wall
for two ops with ZERO stalls and 185 timer ticks serviced. Wall time is worse and
that is fine — it is off the main thread now.
Design notes, all load-bearing:
- ONE JOB AT A TIME. A decoded 12MP bitmap is ~48MB of RGBA; overlapping jobs
multiply peak memory by queue depth, which is the wrong failure on the small
targets this change exists to reach. Costs no throughput — the work is CPU-bound
and one busy worker already saturates its core.
- unref'd while idle, ref'd only in flight. Otherwise scripts/backfill-rotation-
dims.js never exits and `node --test` hangs forever. Verified: a CLI-style run
exits in 104ms, code 0.
- decode failures reply as messages, so one bad upload cannot tear down the worker
and take unrelated queued jobs with it.
- in-process fallback if a thread cannot be had, warned rather than silent.
test/image-ops.test.js pins the loop-liveness property, which no functional test
would catch. Its thresholds were mutation-tested against the inline path: the first
version passed there too (4MP stalls only ~355ms, under a non-flaky threshold), so
the fixture is 12MP and the thresholds sit in the gap between the two behaviours —
worker ~90 ticks/~0ms, inline ~3 ticks/~897ms. It now fails inline, as a guard must.
1647/1647 pass. Ingest re-verified with node_modules/sharp moved aside.
* Measure and thumbnail an image from a single decode
Ingest asked for metadata() then writeThumbnail(), which decoded the file twice.
That pairing was free under sharp, whose .metadata() only parses the header, but
every decode here is a full one — ~1s for a 12MP photo — so the naive translation
doubled the most expensive thing on the ingest path.
image-ops.measureAndThumbnail() returns both from one decode. Full ingest of a
12MP photo: 2 decodes/~1.9s -> 1150ms, still with zero event-loop stalls.
The subtlety is the failure contract. In the two-call version width and height
were assigned BEFORE the thumbnail was attempted, so a failed thumbnail still left
usable dimensions on the row — the player needs them to letterbox. Merging naively
would have turned any thumbnail failure into total metadata loss. So a WRITE
failure is reported ({thumbnailWritten:false, thumbnailError}) with the dimensions
intact, and the caller sets thumbnail_path only when the write succeeded, keeping
the phantom-path discipline. A DECODE failure still throws — there is nothing to
report about an unreadable image.
backfill-rotation-dims.js deliberately keeps the separate calls: it probes every
image row but regenerates a thumbnail only for the few whose dimensions changed,
so pairing them there would decode files it has no reason to thumbnail.
Tests count decodes rather than timing them — an exact property, and a wall-clock
comparison would be flaky under load. The count filters for reads of the file under
test: Node's ESM loader also goes through fs.promises.readFile, so a raw call count
picks up jimp's and the WASM codecs' lazy loading and reads 30 instead of 1.
1649/1649 pass. Ingest re-verified across all 7 formats with sharp moved aside.
* Dockerfile: sharp is no longer a production dependency
--omit=dev now leaves it out entirely; better-sqlite3 is the only native module
the builder stage still needs a toolchain for.
A display could come up with a border on one edge the exact size of a system
bar that was set to hide -- sometimes. Reboot and it might fill the screen
correctly. It has been "sometimes doesn't fill" for a long time, across
devices, which is what a race looks like from the outside.
applyOrientation() read `resources.displayMetrics`, which reports the app
WINDOW rather than the panel. Immersive mode is a request: the bars hide and
the window grows several frames later. A playlist arriving before that finished
measured a bar-sized window and wrote it straight into rootView's layoutParams
-- and it stayed there, because the guard compared only the orientation STRING,
which never changes on a display that has always been landscape. The window
then expanded and the stage did not, leaving dead space the exact size of a bar
that was no longer on screen. onWindowFocusChanged re-asserted the immersive
flags but never re-measured, so nothing repaired it.
Rendering the cached playlist immediately at boot made losing that race the
common case rather than a rare one.
Three changes, each of which alone would help and which together make it
unwinnable:
- re-measure on onWindowFocusChanged, so a stage sized during any transient
window state corrects itself instead of being permanent;
- the guard compares the measured SIZE as well as the orientation, so
"landscape -> landscape" can repair a bad measurement;
- ask for full-bleed through WindowCompat/WindowInsetsControllerCompat as
well as the deprecated systemUiVisibility flags, because some OEM builds
honour only the modern route.
Deliberately measures the WINDOW, not the display. On one RK356x box
`dumpsys window` reports `init=1920x1080 app=1920x1024`: the firmware reserves
56px for a hidden bar, and those pixels are not the app's to paint. Sizing the
stage to the display there would not fill the gap, it would push the bottom of
every asset outside the window and crop it silently -- worse than a border. If
`app=` still differs from `init=` after this, the reservation is firmware
behaviour and has to be turned off on the device.
Also shipped as a 1.9.24-based build for the reporting customer, so the change
could be tested in isolation against ten releases of drift.
The README covers installing, upgrading, backing up and admin recovery -- the
happy paths. It says nothing about the parts that actually go wrong: which
deployment shape you are on and why the other shape's commands silently do
nothing, how to tell a deploy really took when a version string cannot prove it,
and the handful of traps that have each cost real time at least once.
docs/operations.md is that runbook. The load-bearing entries:
- the served APK is a bind-mounted FILE, so it must be replaced in place. mv
or cp gives the host a new inode while the container keeps serving the old
bytes, with nothing in any log to say so;
- the advertised apk_size must equal the served bytes or displays download,
reject and retry forever -- and the OTA query parameter is `version`, where
the wrong name produces a result that looks identical to a broken OTA;
- a version string does not prove new code is running, and neither does the
build hash: it covers the frontend, so a server-only change deploys with an
unchanged hash and looks exactly like a stale image;
- ownership before checkout, because a partial checkout leaves VERSION updated
while the code is the old release and no migrations ran;
- a service user with no home directory makes npm install nothing while
appearing to succeed;
- a prerelease sorts below its own release, and the Android update check
offers one to any older client on the stable channel;
- native modules are built for one Node ABI, and the mismatch presents as
hundreds of unrelated test failures rather than one clear error.
Deliberately generic: no addresses, hostnames, customer names or credentials, so
it is useful to anyone self-hosting rather than a description of one estate.
Every claim was checked against the code or the workflows rather than recalled.
The README described what SSO is and which variables exist. It did not say where
to click, which of the several plausible values to use, or what any failure
means -- so configuring it meant reading source, and every wrong turn produced an
error code with no stated cause.
docs/sso-setup.md walks both audiences: the operator wiring up Google or
Microsoft for the instance, and an organization admin bringing their own
provider and proving a domain. Written from doing it end to end against real
Google and Entra tenants, so the traps in it are the ones actually hit rather
than the ones imagined:
- MICROSOFT_TENANT_ID is the directory that AUTHENTICATES the user, not the
one the app registration lives in. For personal accounts those differ, and
using the visible Directory (tenant) ID fails every login with an error that
points at the tenant rather than at the setting;
- Web platform, not SPA -- a SPA registration is refused at the token endpoint
because the exchange is server-side and sends no Origin;
- a Web registration is a confidential client, so the secret is not optional;
- Entra needs the `email` optional claim added, or the token arrives with no
address and fails as no_email;
- Google's redirect URI matches byte for byte, and Testing publishing status
silently limits sign-in to listed test users.
Every error code the server can emit is in a table with its usual cause. Each
one was checked against the source rather than remembered, as were the variable
names and the DNS record format.
Also covers what the account rules mean in practice: linking deletes the
password, unlinking sets a new one in the same step, SSO-only clears passwords
irreversibly, and linking the platform admin makes that provider the only way
in.
"Authentication required" on every click of Link. The Settings button did
`location.href = /api/auth/oidc/<slug>/link/start`, which is a top-level
navigation -- and this app's session lives in localStorage and travels as an
Authorization header, so the request arrived anonymous and requireAuth refused
it, correctly.
The login /start route works precisely because it needs no session. Copying its
shape for a route that does need one was the mistake.
The client now FETCHES link start with its token and navigates to the URL it
returns. The transaction cookie is still set by that response, because a
same-origin fetch stores Set-Cookie normally, so the callback is unchanged.
beginOidc grew an asJson flag rather than a second copy of the PKCE/state/nonce
setup, so login and link still cannot drift apart.
Both mutations fail the new test: navigating straight at the route, and having
the server redirect instead of answering with JSON.
Two halves of the same problem: an account created with a password could never
use single sign-on, and the login page offered a credential before it knew
which one applied.
LINKING. Signing in with a provider never adopts an account that already has a
password -- that is the takeover the login path exists to refuse. The README
promised the way out ("the owner signs in locally and links from Settings") but
nothing had ever been built, so the refusal was a dead end rather than a
redirection. Settings now has a Sign-in method block: an account with a password
can link an instance-wide provider, and one on a provider can unlink back to a
password.
The account being linked comes from the SIGNED TRANSACTION -- the session that
started it -- never from the email in the returned token. That distinction is
the whole feature: taking it from the token would be the same email-keyed
takeover under a friendlier name. The email must still match the account's own,
because login resolves accounts by the asserted address, and one provider
subject may not be linked to two accounts.
Linking DELETES the password rather than keeping it alongside. One credential at
a time, and the confirmation says so in those words, because a password left
behind is a second way in that the user believes they replaced. Unlink therefore
takes the new password up front and writes it in the SAME statement as the
unlink -- never unlink now and set a password after, which leaves an account
briefly, or on failure permanently, with no way in.
Instance-wide providers only. An organization's provider is chosen by a
customer; letting one attach itself to a platform account would hand that
customer whatever the account can do.
IDENTIFIER-FIRST. The password box now appears only after an address has been
submitted, which is what lets the organization lookup happen before a credential
is offered: someone whose company requires its own provider is shown that,
rather than a password box that will be refused. Editing the address returns to
the identifier step so a corrected domain gets a fresh answer.
The per-keystroke lookup is gone with it. It answered for half-typed domains,
changed the form under someone mid-address, and spent a 10/min per-IP budget on
people who had not finished typing -- an office behind one address could exhaust
it without a single sign-in attempt.
Instance-wide providers stay visible at all times now, by decision: the server
refuses them for an SSO-only organization anyway, and hiding them made the page
change shape while typing.
Verified in a real browser, not only by rendering: password hidden -> submit ->
visible and focused -> edit the address -> hidden again, with no page errors.
Four mutations of the linking rules fail the tests (account from the email
instead of the session, keeping the password, allowing org providers, dropping
requireAuth).
The previous fix let the instance-wide Microsoft button work and left the
customer-facing path broken, which is the worst way round. An organization that
brings its own Entra tenant would publish the TXT record, watch its domain go
green, and still be refused at login with `email_unverified` -- because Entra
sends no such claim and rowToProvider pinned the assumption off for every org
provider.
Requiring a claim Microsoft does not emit is not a security control, it is an
outage. What makes it safe to stop requiring it is the proof that already gates
these providers: the callback confines an org provider to its DNS-verified
domains, and an address only reaches the check after passing that. Whoever
controls a domain's DNS controls its mail, which is the same trust that makes a
verification link meaningful.
So the assumption is DERIVED from proof -- `verified.length > 0` -- rather than
pinned off. A provider that has verified nothing still assumes nothing, which is
belt and braces: emailAllowedForProvider already refuses it, since an empty
allow-list matches no domain, but deriving it here means a future reordering of
those checks cannot silently widen it.
It is never a column, and there is no column for it to be read from. An
organization must not be able to switch this on for itself; it is a consequence
of DNS proof, not a setting. A test asserts both -- that the value is derived
next to `source: 'org'`, and that no `assume_email_verified` exists in the
schema.
Domain confinement is untouched. An explicit `email_verified: false` is still
refused from anyone.
Mutations all fail the tests: assuming unconditionally, never assuming, and
reading it from the row.
The OIDC callback required `claims.email_verified === true`. Entra ID v2 does
not send that claim at all, so every Microsoft login authenticated correctly
against the tenant and was then refused with `email_unverified` on the way back.
Nothing caught it: the SSO tests assert how the Microsoft issuer string is built
but never put a Microsoft-shaped token through the policy.
The strict check was itself a fix -- `=== false` had been accepting an omitted
claim -- and it is right for a provider a CUSTOMER configured, since such a
provider is chosen by the party it vouches for and its bare assertion is worth
nothing. What was wrong is treating that as a question about the token when it
is a question about who we trusted. `users.email_verified` is our own state; the
claim is the IdP's. An instance-wide provider was chosen by the operator -- the
same trust that already exempts it from domain confinement -- and Microsoft is
additionally pinned to one tenant GUID, so only that directory can issue a token
whose `iss` matches.
So the policy now depends on the provider, in emailIsVerified(), next to the
flag it reads so the two cannot drift:
- explicit true -> believed, from anyone
- claim absent, operator -> believed (Microsoft; opt-in for other IdPs)
- claim absent, org -> refused
- explicit false -> refused, always
Org providers pin the flag false in rowToProvider and never read it from the
row, so the takeover path the strict check existed to close stays closed.
Google is left strict: it does send the claim.
Also documents MICROSOFT_CLIENT_SECRET (supported in code, missing from the
table), that the redirect URI must be registered under Web rather than SPA, and
the email optional claim -- the other two ways an Entra setup fails.
All three mutations of this policy fail the new tests: reinstating the strict
check (3 failures), letting an org provider assume (1), and accepting an
explicit false (2).
#254 lets an organization opt out of widget iframe isolation so that players
can embed origin-strict third-party sites. It applied that opt-out to the
widget editor's Preview as well.
Preview is framed by the dashboard, from the dashboard's own origin, and the
dashboard keeps its session JWT in localStorage. So with the setting on, anyone
who can author a widget -- workspace_editor and up; viewers are refused at the
create route -- could put script in a text widget and read the session of
whichever admin clicked Preview. That is an editor -> admin escalation, and it
is not the risk the confirmation modal asks the admin to accept: a player runs
on a kiosk with a device token, an admin's dashboard session is a different
thing entirely.
The org setting is what makes players able to embed those sites, so the
/render path keeps consulting it. Preview is pinned to allow-scripts in both
places that build it -- the dashboard iframe and the server-side render -- so
neither a frontend change nor a new server caller can re-grant it alone.
Also correct the modal copy, which claimed same-origin would expose the session
of anyone viewing "a display or preview". Preview is now excluded, and the
display case is really the device token, so say that instead.
widget-preview-stays-isolated.test.js fails if either half is reverted; both
mutations were checked to fail before committing.
A QA sweep found unescaped interpolations outside the SSO work. Auditing them properly
turned up 34 genuine HTML sinks; 23 carry data a user, a device or an identity provider
controls, and those are escaped here.
The ones that mattered:
- app.js renders `user.name` in the shell on EVERY page, and an identity provider's
`name` claim is stored verbatim, so an IdP could script the whole dashboard
- designer element `label`/`location` and widget `location`/`query` land inside
value="" attributes, where a single quote breaks out
- content `folder` lands in a data-folder="" attribute
- device `name` is set by the operator OR reported by the panel itself
- workspace-members renders a SERVER error string through t(), which interpolates raw
⚠️ My first attempt was a codemod over everything my scanner flagged, and it was wrong.
It wrapped `progressText.textContent`, `block.title` and `confirm(...)` — none of which
are HTML, so escaping there shows users literal `<`. Worse, it wrapped
`title: ev.title ? ... : null`, an API PAYLOAD, which would have written escaped markup
into the database. I reverted the whole thing and narrowed to interpolations that are
genuinely inside an HTML template, then read all 34 and chose 23.
Skipped deliberately: static app strings, i18n output, ternaries yielding `selected`,
`window.location.origin`, and sites already escaped.
Verified in Chrome, not by inspection: the payload was seeded into user.name,
device.name, content.filename/folder, widget.name/config and video_wall.name (the first
attempt's seeds silently failed on column names — the API responses are checked now),
then eleven views were loaded. Zero executions, zero live img tags — AND the payload is
visible as inert text in 6/6 views, which is what proves the views rendered it rather
than the test proving nothing.
1609 tests; every frontend module parses as an ES module.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
Replaces an OAuth implementation that verified nothing that mattered. The Google path
asked tokeninfo whether an ACCESS token was valid and trusted the email in the reply;
the Microsoft path handed a bearer token to Graph /me and trusted that. Neither checked
who the token was issued FOR, so any site a user signed into that asked for `email` or
`User.Read` could replay that token here and be issued a session as them. Identity now
comes from an ID token: signature against the published JWKS, iss, aud, azp, exp, and a
nonce this server generated for that specific login.
- one flow for every provider (Authorization Code + PKCE, server-side), so Google and
Microsoft are ordinary entries rather than special cases; any OIDC provider works
- per-organization providers configured by customers, with sign-in domains PROVED by
a DNS TXT record — a claim reserves nothing until DNS says so, lapses after 8 hours
if unproved, and releases rather than renewing
- optional per-organization SSO-only, where removing the requirement needs a platform
admin's approval; the operator queue lives under Admin
- a boot-time dependency preflight, because this branch removes a dependency and a
rollback would otherwise not start
Instance-wide configuration is the default and unchanged: with no SSO variables set,
the login page and every auth flow behave exactly as before.
Six review rounds, sixteen agent audits. Roughly half of all defects found were in
FIXES rather than in original code — including an account takeover, three separate
lockouts, a CSP block that meant per-organization SSO had never worked in a browser at
all, and a stored XSS where the first fix escaped one of two copies of the same table.
Each is documented at the code it touches, because the reasoning is the part worth
keeping.
1609 tests.
Four HIGH findings. Two were mine, and one was a composition of two of my own fixes.
ONE EXTRA SLASH DEFEATED EVERY /api/auth LIMITER
`/api/auth//login` still reaches the login handler — Express normalises the mount
boundary for the router — but `app.use('/api/auth/login', rateLimit(...))` does not
match it, so the limiter never runs. A review got a real session after 60 unthrottled
password attempts. Same for //totp/verify (unlimited 6-digit brute force),
//forgot-password (unlimited reset mail to any address) and //sso/discover (the
customer-enumeration cap, gone). Fixing the limiter KEY could never help, because the
middleware was never invoked: the path is now collapsed to one canonical form before
routing. Pre-existing, and it falsified this file's own warning about walking past the
login limiter.
STORED XSS: I ESCAPED ONE COPY OF THE TABLE
My earlier fix patched views/admin.js line 357 and missed line 372 in the same
function — and missed views/settings.js entirely, which renders a SECOND copy of the
platform users table from the same endpoint, including the email in a raw text node.
The write path was `POST /api/admin/users`, whose EMAIL_RE barred only whitespace, so
an org or workspace admin (not a platform admin) could choose an address that executed
in the operator's session. Both tables escaped, both regexes tightened to reject markup
characters, verified against 11 address shapes.
I KILLED THE BREAK-GLASS WHILE CLOSING AN ORACLE
Hoisting the domain check above the account lookup — my fix for the enumeration oracle
— made `user.role !== 'platform_admin'` unreachable for enforced domains. On a
self-host the operator IS the org owner, and my would_lock_out_actor guard GUARANTEES
their address is inside the enforced set, so the recovery loop closed on itself:
approving a removal request needs a signed-in platform admin. Both properties hold now
by letting the operator through on a CORRECT PASSWORD only — every wrong answer is the
identical 403 whether the address exists, does not exist, or is theirs. Verified: 200 /
403 / 403 / 403.
Also fixed: enabling SSO-only locked out every password-holding member including the
admin who pressed the button (password refused by policy, SSO refused by
account_exists_local). An org provider now adopts a password account at a domain it has
PROVED by DNS when the org requires SSO — which is what a verified domain means, and
what every hosted identity product does.
SSO USERS WERE LANDING IN A PERSONAL ORG
The membership write added organization_members but no workspace_members, and
ensureDefaultOrgForUser looks at workspaces — so it minted each SSO user a private
organization and made it their current one. The customer's Members page read
"Members (1)" while their staff signed in successfully and were invisible.
ALSO: bcrypt on a NULL password_hash 500'd with a stack (and was an oracle for accounts
a provider deletion had returned to local); stranded_members was returned by the server
and discarded by the UI; a provider with zero domains was the one useless state with no
warning; two limiter shapes were missing (removal-request shared the garbage bucket —
an unauthenticated flood could deny the SSO break-glass path); doubled mail subject
prefixes; a DELETE that toasted "Saved"; a decided request left in the DOM with live
listeners; and a confirm dialog promising "immediately" when sessions already open
survive.
1609 tests, three clean runs. Limiter, break-glass, oracle parity and null-password all
verified against a running server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
THE HEADLINE FEATURE COULD NOT RUN.
"Continue with single sign-on" was a <form method="POST"> that redirected on to the
customer's identity provider. Chrome applies `form-action` across the WHOLE redirect
chain, and the dashboard sets `form-action 'self'`, so the hop to the provider was
aborted — silently. The user clicked and nothing happened: no navigation, no toast, no
spinner, a byte-identical page. Combined with SSO-only it was a total lockout: password
login answers 403 "use the single sign-on button", pointing at a button that cannot
work.
Every test I ran on this feature checked the button RENDERED. None clicked it.
The provider origins cannot be allowlisted — customers supply them at runtime. So the
page now fetches the destination and navigates itself; a script-initiated navigation is
not governed by form-action. The redirect answer is kept for a caller without
JavaScript, where the chain stays same-origin until the provider takes over. The slug
in the JSON is not a disclosure: following the old redirect put it in the address bar
and history anyway.
Verified in Chrome: the provider start endpoint is reached, zero CSP violations, zero
aborted requests — where before it was ERR_ABORTED plus a console violation.
STORED XSS IN THE PLATFORM ADMIN'S SESSION
admin.js interpolated user name, email and auth_provider into innerHTML unescaped, and
/register accepted an address whose local part was an img tag with an onerror handler —
no spaces, so it slipped the asserted-email check too. A reviewer registered
anonymously and got script execution on #/admin: the page operators are now emailed to.
Escaped, and registration refuses addresses that are not addresses. (The render bug
predates this branch; the reachability and the significance of that screen do not.)
ALSO
- the org SSO button is secondary while a password still works; two identical blue
buttons stacked sent people to their IdP by muscle memory after typing a password.
1609 tests, three clean runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
From the regression sweep. The first is a genuine regression against main.
A RATE-LIMITED DISCOVERY PERMANENTLY DEAD-ENDED THE LOGIN PAGE
lookupOrgSso checked that a body PARSED, not that the request succeeded — and a 429
body is valid JSON. So `data.sso` came back undefined, the single sign-on button was
hidden, the password box restored, and the domain recorded as answered: permanently,
for the life of the page. On an SSO-only domain that is the worst outcome available —
the password box then returns 403 and the button the user is told to use is not on the
screen. Discover is 10/min per IP and one person filling in the form costs up to four
calls, so a few colleagues behind one office address is enough. The comment above that
code already claimed to prevent exactly this; it only ever covered the 5xx case.
THE SSO-ONLY REFUSAL WAS AN ACCOUNT-EXISTENCE ORACLE
403 for an address that exists, 401 for one that does not — from an endpoint whose own
lockout returns 401 specifically to avoid that. The DOMAIN check now runs BEFORE the
account lookup, so both answer identically; whether a domain uses single sign-on is
already public through /sso/discover, so it reveals nothing new. The membership-level
refusal is deliberately downgraded to the generic 401, because a distinct answer there
would put the oracle back for exactly the accounts worth enumerating.
Verified: existing and invented addresses at an SSO-only domain both 403; and on an
instance with NO SSO configured, register/login/wrong-password/unknown-address are
201/200/401/401 — the hoisted check does not touch them.
BOOT PREFLIGHT
- a cold install ran `npm ci --omit=dev` unconditionally, so a first start on a
developer machine left `npm test` broken: same class of surprise as the prune this
file already warns about, through the other branch of the same if. Now production-
only.
- two servers starting together: the loser died with ENOTEMPTY even though the tree
was complete by then. It re-checks before failing.
- the opt-out accepted only '1', unlike every other boolean the server takes.
THE LIMITER FOLD, DONE PROPERLY
Unmatched paths under /api/organizations still minted a bucket each. My first fix was a
catch-all regex — which put every unknown path in ONE bucket WITH the real endpoints,
so flooding nonsense URLs exhausted the limit for /sso-only. That trades a bypass for a
denial of service. Folding is now by explicit shape: known endpoints keep their own
keys, everything else shares a bucket kept apart from all of them.
Verified: 120 unmatched paths give 60/60 (bypass closed), and after that flood
/sso-only, /sso and /sso/:id/test all still answer 401 rather than 429 (no starvation),
while 70 hits on one real endpoint do trip its own limit. The login trailing-slash
bypass stays closed.
1609 tests, three clean runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
A second attack round defeated three of the previous fixes and found a regression I
introduced. Each is reproduced-then-refused against a live server.
MEMBERSHIP: organization_members IS NOT HOW PEOPLE JOIN
Only three places write that table and nothing deletes from it — every INVITED user,
every admin-created account and every workspace assignment lands in workspace_members
and nowhere else. So keying enforcement on organization_members covered org owners and
people who had already used SSO: exactly the set the domain check already caught. A
reviewer invited an outside address into an SSO-only tenant, kept password login, read
the member list and content, and used it to invite more. Enforcement now asks whether
the user is in ANY workspace belonging to an SSO-only organization.
THE INTERLOCK ASKED THE WRONG QUESTION, TWICE
It fired only when a domain list became EMPTY, and it counted PROVIDERS. So:
- replacing acme.test with decoy.test removed every proof and sailed through — two
PUTs, and the customer's domain enforced nothing, with sso_only still reading true;
- with two providers you could disable the one owning your staff's domain, because
the other one, covering a domain nobody signs in at, still "enforced".
The question that matters is per-DOMAIN: after this change, is every domain that
enforces today still enforcing? Losing one needs the operator, whichever route gets you
there. The refusal now names the domain that would stop being covered.
REGRESSION I CAUSED: THE HAPPY PATH LOCKED THE OWNER OUT
Sign up with a personal address, create the org, verify the company domain, turn this
on — and enforcement covers you (you are a member) while your own address is outside
the verified domains, so passwords are refused AND your org's provider will not assert
for you either. No route removes a membership; reset succeeds but login still refuses.
Recovery meant a platform admin turning SSO off for the whole tenant. Enabling now
refuses when the actor's own address is not covered, naming it, and REPORTS everyone
else who will be stranded instead of letting them be discovered by support ticket.
ALSO
- POST /api/admin/users gated only on the target workspace, so you could mint
cfo@theircompany.test into your OWN workspace: login refused, but the row now has a
password_hash and an SSO login will not adopt one — permanently locking a real
person out of their own address. Now gated on the address's domain too.
- `ceo@acme.test.` (trailing root dot) slipped the registration gate.
- two rate-limited sub-paths were still unfolded because the generic org-id fold ate
`sso-only` as an organization id; the specific shapes are matched first now.
1609 tests, three clean runs. Verified live: invited outsider 403, swap refused,
sibling-disable refused, squat 400, self-lockout refused with the address named, and an
on-domain admin gets `stranded_members` back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
The approval workflow had no front door. The notification email told the operator to
"review it in ScreenTinker under Admin" and that screen did not exist — the only way to
approve was curl, while the tenant sat locked out of their own product. Admin now leads
with a removal-request section: who asked, for which organization, the reason they
gave, what approving does, and Approve/Reject. It hides itself when the queue is empty.
Approving is confirmed; rejecting is not, because rejecting only leaves the safe state.
REGISTRATION BYPASSED SSO-ONLY AND SQUATTED ADDRESSES
/register had no domain awareness: it issued a working session at an SSO-only domain,
and the account then held that address forever, because an SSO login will not adopt a
row that has a password. Registering ceo@acme.test before the real CEO's first login
left the address dead in both directions with no self-service way out. Refused now, and
"Create Account" is hidden on the login page for those domains — it was the only action
left on the card, so the page was inviting the one thing that cannot work.
THE NEW RATE LIMIT WAS DECORATIVE
/api/organizations carries three caller-chosen segments, and only the OIDC slug was
folded — so every request minted its own bucket. Measured: 120 calls with unique org
ids produced ZERO 429s, unauthenticated, against the limit that exists to bound
outbound discovery and live DNS. Now 60/60. The general problem was named in the
previous commit's own comment and then not applied to the mount it added.
XSS IN THE TOAST
showToast built innerHTML from server strings, including ones that reflect input
verbatim — a reviewer typed `<img src=x onerror=alert(1)>` as an issuer and got script
execution in the admin's session. Escaped.
ALSO
- the org SSO button sat BETWEEN the "Password" label and its input, so the label
described the button and the field had none; moved below the input, with a for=
- the OR divider survived when the providers under it were hidden
- provider action buttons were clipped off-screen at 375px with no way to scroll to
them — "Remove" was unreachable; the row wraps now
1609 tests. Verified in real Chrome: 13/13 on the approval loop and the login states,
including approving a request and watching password login re-open for that org.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
Three HIGH findings from the QA round. Each was demonstrated end to end against a
running server, and each is now refused there.
ENFORCEMENT PROTECTED A DOMAIN, NOT AN ORGANIZATION
ssoOnlyForEmail answers about an address's domain, so any account in the tenant at an
outside address kept password login — a contractor, an MSP, the one address nobody
remembered. And it could be manufactured: POST /api/admin/users accepts workspace_admin
and creates a LOCAL password account at any address bound to that workspace. A review
created backdoor@notacme.test, logged in with the password, landed in the SSO-only org,
and used it to create another. Enforcement is now keyed on MEMBERSHIP as well as domain
(ssoOnlyForUser), and that route refuses to mint password accounts into an SSO-only
organization at all. platform_admin keeps both, as the operator break-glass.
THE APPROVAL WORKFLOW WAS DECORATIVE
`sso_only` is honoured only while a provider is enabled and a domain is verified, so
`PUT {enabled:false}`, `PUT {email_domains:""}` and `DELETE` each switched enforcement
off — with sso_only still reading true, no request filed and the operator never told.
The delete variant additionally rewrites every federated account to `local`, after
which a password reset takes over accounts the identity provider was supposed to own.
Anyone who could file a request could simply turn the provider off instead. All three
now refuse with sso_only_locked when nothing else would still enforce, and say to ask
for approval.
FRESH INSTALLS FAILED THE MIGRATION AND FAILED OPEN
The ALTER adding organizations.sso_only sat in the column-migration array, which runs
BEFORE the multi-tenancy migration that creates the table: `[migrate] FAILED … no such
table: organizations`, one line among ~85. The instance then ran its whole first boot
with the SSO settings screen 500ing and ssoOnlyForEmail catching `no such column` and
answering "not required" — password login proceeding for an organization that had
switched it off. It self-healed on the second boot, which is what made it easy to miss.
The column is now added after the table exists, and the catch distinguishes "this
instance has no per-org SSO" (null, so single-tenant installs keep working) from drift
on a table that DOES exist (throw). Login treats an undeterminable answer as "required"
rather than letting a 500 escape or letting the login through.
Verified live, all four refused with enforcement intact and the operator still able to
sign in. 1609 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
