The installer and launcher fixes shipped in #248 and are in the changelog, but the
operator-facing lessons were not written down anywhere, and two of them are general
rather than Pi-specific:
- a piped installer cannot ask you anything, because the pipe IS stdin — the
prompt gets EOF and takes the default while looking like a choice
- X11 tools no-op silently on Wayland, so blanking suppression and cursor hiding
can be entirely absent while every command in your notes appears to have worked
The third answers the question in the issue that the fix did not address: overlay
FS is fine for a player-only Pi (the cache re-downloads) and quietly destructive
for an all-in-one, where the database, WAL, uploads and thumbnails are written
continuously and a read-only root discards them at every reboot.
A panel could not update, forever, while downloading content perfectly well. Its
content cache writes to INTERNAL storage and works; the APK path insisted on
EXTERNAL storage, which on that device is present but unwritable — so the download
died at outputStream(), before a single byte, roughly one second after the command.
The server logged the request as served, the client reported "failed to download or
failed signature verification", and nothing pointed at a directory.
apkDir() previously asked canWrite(), believed the answer, and returned external.
The alpha12 preflight then PROVED the directory with a real write and refused —
correctly diagnosing the problem and still not updating, because it had no way to
choose somewhere else. A fallback that only reports is not a fallback.
apkStagingDir() now walks candidates and returns the first that genuinely accepts
bytes:
internal /data/data/<pkg>/files/Download — always mounted, always writable
external the old location — kept, it survives uninstall
cache /data/data/<pkg>/cache/Download
files the app's files dir, no subdirectory to create
Internal is first because it cannot fail: it is this app's own private directory,
and if it is unwritable the app is not running. External is a convenience — visible
for a manual install, survives uninstall — and it is the one that breaks, so it is
no longer the default. Every candidate is proven by writing a probe byte and
deleting it, never by asking canWrite(), which returns true on volumes that then
refuse the write. That is precisely how this hid.
If no candidate works, the failure names every one it tried and why, instead of the
first excuse.
The pushed-APK path uses the same cascade; it shared the fault and reported none of
it.
⚠️ This cannot reach a panel already stuck: the broken path is the delivery
mechanism, and Push APK shares it. Such a panel needs ONE manual install, after
which it is permanently self-healing.
A plain string compare on the prerelease tag put "alpha11" below "alpha8", because
'1' < '8'. The OTA check therefore answered client-newer and refused to offer the
update — while reporting the newer build as `latest` in the same response:
latest_version: 1.9.34-alpha11 current_version: 1.9.34-alpha8
update_available: false reason: "client-newer"
So a fleet on alpha8 or alpha9 could not be moved forward at all, silently, and
nothing about the symptom pointed at version ordering. alpha10 was never really on
offer either; the last update that genuinely worked was alpha6 -> alpha8, where the
lexical order happens to be right by luck.
This is what semver specifies for a single alphanumeric identifier, and it is
simply not what the naming means. lib/version-precedence.js compares digit runs
NUMERICALLY, so alpha8 < alpha9 < alpha10 < alpha11, while leaving everything else
alphabetical — beta still outranks alpha, rc still outranks beta, and a release
still outranks any prerelease of the same core.
Dot-separated identifiers are compared per semver and a shorter run loses, so
moving the naming to the semver-correct `-alpha.11` form later needs no further
change here.
TWO comparators carried the assumption, each with a comment asserting lexical was
fine "for our naming" — true only while the counter stayed below 10. Both now use
the shared helper rather than a third copy drifting into the same trap:
- lib/ota-breaker.js the Android OTA path
- lib/brightsign-update.js the BrightSign host package, where a wrong-way
comparison replaces the script that boots the player
lib/ghcr-check.js was checked and is unaffected: it rejects prerelease strings
outright rather than ordering them.
Tests pin the exact stranding case end to end — decide('1.9.34-alpha8',
'1.9.34-alpha11') must be an offer, not client-newer — plus the reverse direction,
so a future change cannot merely invert it.
1668/1668 pass.
The bump was tagged before this landed — a quoting error swallowed the entry and
bump-version.sh's CHANGELOG guard is a warning, not a stop. The v1.9.34-alpha11
tarball therefore ships without it; the repo has it from here.
* OTA: say which failure happened, and stop refusing readable APKs on API 28/29
Two changes, both aimed at the same dead end: a panel that will not update and a
message that cannot tell you why.
NAME THE FAILURE. "failed to download or failed signature verification" covers
SEVEN distinct branches — three of them download failures where verification never
runs at all. Every specific reason went to logcat, and an unprivileged app UID
cannot read logcat on Android 9, so in the field the message was unactionable: it
named a symptom shared by unrelated causes and pointed at the wrong half of the
code as often as the right one. Each branch now records what actually happened and
the operator sees it:
"not installed — server returned HTTP 416 for the APK"
"not installed — could not read signing certificates (archive=0, installed=1) on API 28"
"not installed — APK is signed by a different key than the installed app"
"not installed — download/install threw IOException: ENOSPC (downloaded 41232 bytes)"
The byte count rides along on verification failures so a truncated download is
distinguishable from a genuine key mismatch — the two look identical today.
FALLBACK ARCHIVE CERT READ. On API 28/29 the archive's signer comes from the
legacy GET_SIGNATURES path (#139: signingInfo is null for ARCHIVES below API 30).
When PackageManager returns nothing there, we refused a possibly-fine APK with no
way to tell that apart from a real mismatch. It now reads the v1 signature itself
via JarFile before giving up.
This does NOT weaken the check. JarFile with verify=true only populates
JarEntry.certificates after the covered bytes have been read and verified — the
read IS the verification — and the extracted cert is still compared against the
installed app's. An unsigned, tampered or differently-signed APK still fails, and
any error in the fallback returns empty, which still refuses the install.
Deliberately NOT done: disabling signature verification on Android 9. It would mean
those panels silently installing whatever the server hands them — on device-owner
hardware that is remote code execution, and the cheap Android 9 boxes are the ones
most likely on a customer's flat network. It also might not fix anything, since
three of the seven branches never reach verification.
Context: a panel on alpha failed four forced updates this evening while succeeding
at an unattended one, and four separate theories for it died on contact with
evidence. The reason this took an evening is that the device could not say what
went wrong. That is the actual bug being fixed here.
* OTA: prove the destination is writable before downloading
Every theory this evening turned on whether the app could actually put a file
somewhere, and nothing in the code ever checked. A returned directory path is not
the same as a usable one: it can be missing, unwritable, on a volume that has gone
away, or simply full — and all four surfaced as the same opaque "failed to
download" as a genuine network fault.
apkDir() now proves the external directory before choosing it (exists-or-created,
and canWrite) rather than trusting a non-null path, and falls back to internal
storage when it does not hold up.
apkDirProblem() runs BEFORE the network call on both download paths and names the
real condition:
"cannot stage the update — no write permission on /storage/…/Download"
"cannot stage the update — only 6MB free on /data/…/Download, need ~18MB"
"cannot stage the update — write test failed in /storage/…: IOException EROFS"
It does not infer from canWrite(), which returns true on volumes that then refuse
the write; it writes a probe byte and deletes it. Free space is checked against
DOUBLE the APK, because the installer stages its own copy — a volume with exactly
the download's worth free still fails later, at install time, where the message is
even further from the cause.
The pushed-APK path gets the same preflight; it shared every one of these failure
modes and reported none of them.
Three changes, all about the same failure: sharing appears to be on while nothing
actually arrives.
SEND ON OPT-IN. Turning sharing on now reports immediately instead of waiting for
the next daily tick. Two reasons: the operator is standing right there, and
"nothing has been sent" for the next 24h reads as broken at exactly the moment
someone is checking whether it works. It also means an egress-filtered network
fails HERE, where we can name the host to unblock, rather than silently tonight
where nobody is watching.
NAME THE FAILURE. Failed attempts are now recorded separately from successes, so
Settings can say which address did not answer and why, instead of showing an empty
"nothing sent yet". A blocked outbound connection is the normal failure on a
self-hosted box and is otherwise completely invisible — the operator cannot tell a
firewall from a broken feature. A later success clears the complaint, so a stale
warning never outlives the problem it describes. Docs gained a section on it, and
the UI states plainly that nothing needs opening inbound.
OPERATOR COLLECTOR, ADDITIVE. TELEMETRY_EXTRA_ENDPOINT lets an operator post the
same three fields to their own collector.
The naming is the point. It replaces TELEMETRY_ENDPOINT, which was a true override
— and an override is the wrong shape here, because a variable called "endpoint"
that silently redirected the report someone agreed to SHARE would make the opt-in
mean something other than what the UI says. Our address is hard-wired and not
overridable; theirs is explicitly additional and named so it cannot be mistaken for
a replacement. Settings lists every destination a report goes to.
The operator collector is independent of the sharing switch, because it is their
server posting to their host and our opt-in has no business gating it. So an
operator who wants internal fleet numbers with nothing leaving for us sets it and
leaves sharing off — supported on purpose, and tested.
Destinations are attempted separately: one unreachable collector must not cost the
other its report.
1662/1662 pass. Tests pin the properties that matter: an operator collector never
replaces the shared report, sharing-off still sends nothing to us whatever else is
configured, one dead destination does not stop the other, and a failure records the
address actually tried.
There is no way to answer "how many screens run ScreenTinker?". The product is
self-hostable by design, so most installs are invisible to us on purpose — and
should stay that way. This asks once, and reports only if the operator says yes.
The entire payload is three fields:
{ instance_id, version, screen_count }
instance_id is a random UUID minted on first use and kept in app_settings. It
carries nothing about the install; its only job is to let two reports from the
same server be recognised as one server, so a count is a count rather than a sum
of duplicates. That makes a report pseudonymous rather than anonymous, and the
wording shown to operators says so rather than claiming otherwise.
The payload is short on purpose. Every field added costs participation, and
participation is the only thing that makes the resulting number worth quoting.
Player-platform counts were considered and left out: release assets are already
published per platform, so GitHub's per-asset download counts answer "where should
effort go" at zero privacy cost and without asking anyone for anything.
Verifiability is the feature, not the copy. Settings shows the ACTUAL payload this
server would send, generated live from its own data, plus what it last really sent
and when. The payload is built in one function so a reviewer can check it at a
glance, and the test fails if a field is ever added.
Both answers persist. Declining is remembered as 'off' rather than falling back to
'unasked', so the prompt cannot return after an update — re-prompting is how
telemetry earns its reputation and gets patched out.
Collector side is inert unless TELEMETRY_COLLECTOR=1, so a normal install never
exposes the endpoint. Reports upsert on instance_id rather than appending, so an
install reporting daily occupies one row rather than 365 a year. The source IP is
never read or stored — receiving one is unavoidable, logging it would quietly turn
a pseudonymous report into an identifiable one.
Tests pin the negative promises, which are the ones that rot silently: sends
nothing before consent, sends nothing after a decline, payload is exactly three
keys, id survives a restart, a failed send never records a phantom report. Screen
count excludes unpaired provisioning rows, which would otherwise overstate the one
number this exists to state honestly.
docs/telemetry.md documents the payload, what is not sent, how to verify it, and
that any published total is a floor rather than a basis for extrapolation.
1657/1657 pass.
getExternalFilesDir() returns null whenever external storage is unavailable, and
on a signage panel that is not exotic: no emulated volume, a vendor ROM that never
mounts one, an ejected card, storage still unmounted early in boot.
Both APK download paths did:
File(context.getExternalFilesDir(DIRECTORY_DOWNLOADS), name)
Java's File(File, String) treats a null parent as "no parent" and silently yields
a RELATIVE path, so the download targeted `ScreenTinker-x.y.z.apk` in the process
working directory — `/` — which is not writable. The write threw, the generic
catch swallowed it, and the caller reported only "failed to download or failed
signature verification".
That message is why this was expensive to find. The HTTP request SUCCEEDS (the
server logs a served download at the exact moment of each failure), so it does not
look like a download problem; the signing key is fine, so it does not look like a
verification problem; and because nothing is ever written there is no partial file
to find. It also never recovers — every attempt fails the same way, forever.
installFromUrl (the dashboard "Push an APK" button) carried the identical line, so
the obvious workaround for a panel in this state was broken by the same bug.
Both now use apkDir(), which falls back to internal storage. filesDir cannot be
unmounted: if it is gone the app is not running.
file_paths.xml gains a <files-path> for the same directory. The silent
PackageInstaller path streams the file itself and needs nothing there, but the
intent-based install FALLBACK resolves it through FileProvider and would throw
"Failed to find configured root" — turning an already-degraded panel into one that
cannot install at all.
⚠️ This cannot reach an affected panel over the air: the broken download path IS
the delivery mechanism, and Push APK shares the bug. A panel already in this state
needs one manual install to escape it.
Root cause is inferred from converging evidence on an Android 9 panel (HTTP served
at each failure, no APK anywhere on the device, the app's own external files dir
denied to its own UID, both paths failing identically, signing verified at parity
against the release it already installed). Handling a null return is correct
regardless — it must never become a relative path.
Covers everything since alpha6: the stage-sizing fix (#262), the operations
runbook (#261), the sharp removal (#263) and the better-sqlite3 pin (#264/#265),
plus the dependency-reinstall requirement that applies in both directions.
* Document the Node.js upgrade procedure and this build's reinstall requirement
Upgrading the runtime does not go through scripts/upgrade.sh, so nothing
reinstalls dependencies — which is precisely when the one remaining native
module goes stale. The runbook now covers the version floor imposed by
--env-file-if-exists, why the better-sqlite3 pin is exact, and why a version
without a matching prebuild can turn Restart=always into a boot loop.
Also records that this build changes dependencies in both directions: rolling
back past it needs the reinstall too, because earlier builds import sharp at
runtime and this one drops it from production dependencies.
Kept deployment-neutral — no hostnames, addresses, or environment specifics.
* Fix the only test that fails on Node 22
Node 22 added a built-in `navigator` global, defined as a getter with no
setter. The test's shim assigned to it, which throws "only a getter" under
'use strict' on 22 while being a normal assignment on Node 20, where the global
does not exist at all. It is configurable, so define it instead of assigning.
Defining it unconditionally is also the better fixture: Node 22's own navigator
reports the HOST locale, so a test reading its language would otherwise depend
on the machine or CI runner it happens to run on.
This was the single failure in an otherwise clean Node 22 run (1639/1640 with
better-sqlite3 12.9.0), and it is confined to test code — no production server
or frontend file assigns to globalThis.navigator.
1649/1649 on Node 20.
Prepares for the Node 22 move by decoupling it from the database driver, so the
two upgrades land as independently reversible steps rather than one flag day.
9.6.0 cannot work on Node 22. It uses the raw V8 API (220 v8:: references, zero
napi_), so it is ABI-locked per Node major, and its GitHub release assets carry
prebuilds for ABI 108/115/120 only — nothing for Node 22's 127. Its install script
is `prebuild-install || node-gyp rebuild --release`, so on Node 22 it silently
falls through to compiling raw-V8 code against Node 22 headers. That is not just
slow: lib/preflight-deps.js rebuilds synchronously before the server listens, and
prod's systemd unit is TimeoutStartSec=90 with Restart=always, so a slow or failing
compile is an unbootable loop rather than the intended self-heal.
12.9.0 ships prebuilds for BOTH Node 20 (ABI 115) and Node 22 (127), so neither the
current runtime nor the target has to compile anything.
THE PIN IS EXACT ON PURPOSE — ^12.9.0 would defeat it. 12.10.0 dropped the Node 20
prebuild while still advertising "20.x" in engines, so a caret resolves to 12.11.x
and reintroduces the from-source compile on today's runtime. Verified against the
release assets per version:
12.0.0 / 12.2.0 / 12.4.5 / 12.6.2 / 12.9.0 ABIs 115,127,...
12.10.0 / 12.10.1 / 12.11.1 ABIs 127,137,141,147 — no 115
The reasoning is recorded in preflight-deps.js, which is where anyone hitting the
matching failure will already be reading.
13.x was considered and rejected FOR NOW: it is the first N-API release, which ends
the per-major ABI problem for good (8 prebuilds keyed by platform, not ABI) and is
where we should eventually land — but engines is ">=22", so it cannot be adopted
while prod, alpha and CI all run Node 20. It is also three weeks old with three
patch releases, which is young for the one component that owns all the data.
No API changes to absorb: every major from 10 to 13 bumped only for dropping EOL
Node/Electron versions, so the ~1486 .prepare(), 46 .transaction() and 59 .pragma()
call sites are untouched.
Verified on Node 20: installed from a PREBUILT binary (no obj.target, so no
compilation), opens a real database, and the WAL path the #149 checkpointer depends
on still works — journal_mode=WAL engages on a file DB, pragma(...,{simple:false})
returns the expected shape, and a second connection from another handle reads and
runs wal_checkpoint(TRUNCATE). 1649/1649 tests pass.
* spike: replace sharp with pure-JS image ops (jimp + jsquash WASM)
Removes the last native dependency from the ingest path, so the server no longer
needs a per-platform/per-ABI prebuilt to thumbnail an image. Motivated by getting
the server onto hardware with no toolchain, but the ABI tax is paid on every
install — it is the same failure class lib/preflight-deps.js exists to explain.
lib/image-ops.js is the whole surface: metadata() and writeThumbnail(), which are
the only two things ingest ever asked sharp for.
Format parity holds. jpeg/png/gif/tiff/bmp are native to Jimp; webp and avif go
through @jsquash WASM, whose bundled .wasm must be compiled by hand because the
packages locate it with fetch(file://) and Node has no file:// fetch — the only
symptom otherwise is a bare "fetch failed". heic is unsupported, as it already
was: sharp advertises heif but its prebuilt libvips refuses HEVC.
#170 is preserved by a different mechanism. Jimp applies EXIF orientation at
decode and rewrites the tag to 1, so metadata() reports display dimensions and
imageDisplayDims() runs as a no-op instead of swapping W/H a second time. The
helper stays in the path so the rule keeps living in one place.
Verified: 1643/1643 tests pass, and ingest was exercised in a child process with
node_modules/sharp moved aside — jpeg, EXIF-rotated jpeg, png, webp, avif, gif
all measured and thumbnailed correctly, corrupt input still yields nulls with no
phantom thumbnail_path.
KNOWN BLOCKER, do not ship as-is: Jimp is pure JS on the main thread, where sharp
handed work to a libvips threadpool. A 12MP photo goes 65ms -> 1079ms, and the
event loop stalls for 1003ms of it (sharp: zero stalls). thumbnail-backfill.js
walks a whole library at boot, so this reproduces #240 exactly — blocked loop,
missed heartbeats, panels marked offline, reconnect churn. Needs a worker_thread
offload before this is viable; image-ops.js is the seam for it.
* Run image decoding on a worker thread
Fixes the blocker the previous commit shipped with. Pure-JS decoding costs ~1s of
solid CPU for a 12MP photo, and in-process that is not a slow upload but a stalled
event loop — no heartbeats, no socket traffic. thumbnail-backfill.js walks a whole
library at boot, so it reproduced #240 (blocked loop -> missed heartbeats -> panels
offline -> reconnect churn) from our own maintenance. sharp never did this because
libvips works on a threadpool.
image-ops.js is now a dispatcher over image-ops-worker.js; the work moved unchanged
to image-ops-core.js, so callers and their failure contract are untouched.
Measured on a 12MP photo: 1079ms wall with the loop stalled 1003ms, to 1881ms wall
for two ops with ZERO stalls and 185 timer ticks serviced. Wall time is worse and
that is fine — it is off the main thread now.
Design notes, all load-bearing:
- ONE JOB AT A TIME. A decoded 12MP bitmap is ~48MB of RGBA; overlapping jobs
multiply peak memory by queue depth, which is the wrong failure on the small
targets this change exists to reach. Costs no throughput — the work is CPU-bound
and one busy worker already saturates its core.
- unref'd while idle, ref'd only in flight. Otherwise scripts/backfill-rotation-
dims.js never exits and `node --test` hangs forever. Verified: a CLI-style run
exits in 104ms, code 0.
- decode failures reply as messages, so one bad upload cannot tear down the worker
and take unrelated queued jobs with it.
- in-process fallback if a thread cannot be had, warned rather than silent.
test/image-ops.test.js pins the loop-liveness property, which no functional test
would catch. Its thresholds were mutation-tested against the inline path: the first
version passed there too (4MP stalls only ~355ms, under a non-flaky threshold), so
the fixture is 12MP and the thresholds sit in the gap between the two behaviours —
worker ~90 ticks/~0ms, inline ~3 ticks/~897ms. It now fails inline, as a guard must.
1647/1647 pass. Ingest re-verified with node_modules/sharp moved aside.
* Measure and thumbnail an image from a single decode
Ingest asked for metadata() then writeThumbnail(), which decoded the file twice.
That pairing was free under sharp, whose .metadata() only parses the header, but
every decode here is a full one — ~1s for a 12MP photo — so the naive translation
doubled the most expensive thing on the ingest path.
image-ops.measureAndThumbnail() returns both from one decode. Full ingest of a
12MP photo: 2 decodes/~1.9s -> 1150ms, still with zero event-loop stalls.
The subtlety is the failure contract. In the two-call version width and height
were assigned BEFORE the thumbnail was attempted, so a failed thumbnail still left
usable dimensions on the row — the player needs them to letterbox. Merging naively
would have turned any thumbnail failure into total metadata loss. So a WRITE
failure is reported ({thumbnailWritten:false, thumbnailError}) with the dimensions
intact, and the caller sets thumbnail_path only when the write succeeded, keeping
the phantom-path discipline. A DECODE failure still throws — there is nothing to
report about an unreadable image.
backfill-rotation-dims.js deliberately keeps the separate calls: it probes every
image row but regenerates a thumbnail only for the few whose dimensions changed,
so pairing them there would decode files it has no reason to thumbnail.
Tests count decodes rather than timing them — an exact property, and a wall-clock
comparison would be flaky under load. The count filters for reads of the file under
test: Node's ESM loader also goes through fs.promises.readFile, so a raw call count
picks up jimp's and the WASM codecs' lazy loading and reads 30 instead of 1.
1649/1649 pass. Ingest re-verified across all 7 formats with sharp moved aside.
* Dockerfile: sharp is no longer a production dependency
--omit=dev now leaves it out entirely; better-sqlite3 is the only native module
the builder stage still needs a toolchain for.
A display could come up with a border on one edge the exact size of a system
bar that was set to hide -- sometimes. Reboot and it might fill the screen
correctly. It has been "sometimes doesn't fill" for a long time, across
devices, which is what a race looks like from the outside.
applyOrientation() read `resources.displayMetrics`, which reports the app
WINDOW rather than the panel. Immersive mode is a request: the bars hide and
the window grows several frames later. A playlist arriving before that finished
measured a bar-sized window and wrote it straight into rootView's layoutParams
-- and it stayed there, because the guard compared only the orientation STRING,
which never changes on a display that has always been landscape. The window
then expanded and the stage did not, leaving dead space the exact size of a bar
that was no longer on screen. onWindowFocusChanged re-asserted the immersive
flags but never re-measured, so nothing repaired it.
Rendering the cached playlist immediately at boot made losing that race the
common case rather than a rare one.
Three changes, each of which alone would help and which together make it
unwinnable:
- re-measure on onWindowFocusChanged, so a stage sized during any transient
window state corrects itself instead of being permanent;
- the guard compares the measured SIZE as well as the orientation, so
"landscape -> landscape" can repair a bad measurement;
- ask for full-bleed through WindowCompat/WindowInsetsControllerCompat as
well as the deprecated systemUiVisibility flags, because some OEM builds
honour only the modern route.
Deliberately measures the WINDOW, not the display. On one RK356x box
`dumpsys window` reports `init=1920x1080 app=1920x1024`: the firmware reserves
56px for a hidden bar, and those pixels are not the app's to paint. Sizing the
stage to the display there would not fill the gap, it would push the bottom of
every asset outside the window and crop it silently -- worse than a border. If
`app=` still differs from `init=` after this, the reservation is firmware
behaviour and has to be turned off on the device.
Also shipped as a 1.9.24-based build for the reporting customer, so the change
could be tested in isolation against ten releases of drift.
The README covers installing, upgrading, backing up and admin recovery -- the
happy paths. It says nothing about the parts that actually go wrong: which
deployment shape you are on and why the other shape's commands silently do
nothing, how to tell a deploy really took when a version string cannot prove it,
and the handful of traps that have each cost real time at least once.
docs/operations.md is that runbook. The load-bearing entries:
- the served APK is a bind-mounted FILE, so it must be replaced in place. mv
or cp gives the host a new inode while the container keeps serving the old
bytes, with nothing in any log to say so;
- the advertised apk_size must equal the served bytes or displays download,
reject and retry forever -- and the OTA query parameter is `version`, where
the wrong name produces a result that looks identical to a broken OTA;
- a version string does not prove new code is running, and neither does the
build hash: it covers the frontend, so a server-only change deploys with an
unchanged hash and looks exactly like a stale image;
- ownership before checkout, because a partial checkout leaves VERSION updated
while the code is the old release and no migrations ran;
- a service user with no home directory makes npm install nothing while
appearing to succeed;
- a prerelease sorts below its own release, and the Android update check
offers one to any older client on the stable channel;
- native modules are built for one Node ABI, and the mismatch presents as
hundreds of unrelated test failures rather than one clear error.
Deliberately generic: no addresses, hostnames, customer names or credentials, so
it is useful to anyone self-hosting rather than a description of one estate.
Every claim was checked against the code or the workflows rather than recalled.
The README described what SSO is and which variables exist. It did not say where
to click, which of the several plausible values to use, or what any failure
means -- so configuring it meant reading source, and every wrong turn produced an
error code with no stated cause.
docs/sso-setup.md walks both audiences: the operator wiring up Google or
Microsoft for the instance, and an organization admin bringing their own
provider and proving a domain. Written from doing it end to end against real
Google and Entra tenants, so the traps in it are the ones actually hit rather
than the ones imagined:
- MICROSOFT_TENANT_ID is the directory that AUTHENTICATES the user, not the
one the app registration lives in. For personal accounts those differ, and
using the visible Directory (tenant) ID fails every login with an error that
points at the tenant rather than at the setting;
- Web platform, not SPA -- a SPA registration is refused at the token endpoint
because the exchange is server-side and sends no Origin;
- a Web registration is a confidential client, so the secret is not optional;
- Entra needs the `email` optional claim added, or the token arrives with no
address and fails as no_email;
- Google's redirect URI matches byte for byte, and Testing publishing status
silently limits sign-in to listed test users.
Every error code the server can emit is in a table with its usual cause. Each
one was checked against the source rather than remembered, as were the variable
names and the DNS record format.
Also covers what the account rules mean in practice: linking deletes the
password, unlinking sets a new one in the same step, SSO-only clears passwords
irreversibly, and linking the platform admin makes that provider the only way
in.
"Authentication required" on every click of Link. The Settings button did
`location.href = /api/auth/oidc/<slug>/link/start`, which is a top-level
navigation -- and this app's session lives in localStorage and travels as an
Authorization header, so the request arrived anonymous and requireAuth refused
it, correctly.
The login /start route works precisely because it needs no session. Copying its
shape for a route that does need one was the mistake.
The client now FETCHES link start with its token and navigates to the URL it
returns. The transaction cookie is still set by that response, because a
same-origin fetch stores Set-Cookie normally, so the callback is unchanged.
beginOidc grew an asJson flag rather than a second copy of the PKCE/state/nonce
setup, so login and link still cannot drift apart.
Both mutations fail the new test: navigating straight at the route, and having
the server redirect instead of answering with JSON.
Two halves of the same problem: an account created with a password could never
use single sign-on, and the login page offered a credential before it knew
which one applied.
LINKING. Signing in with a provider never adopts an account that already has a
password -- that is the takeover the login path exists to refuse. The README
promised the way out ("the owner signs in locally and links from Settings") but
nothing had ever been built, so the refusal was a dead end rather than a
redirection. Settings now has a Sign-in method block: an account with a password
can link an instance-wide provider, and one on a provider can unlink back to a
password.
The account being linked comes from the SIGNED TRANSACTION -- the session that
started it -- never from the email in the returned token. That distinction is
the whole feature: taking it from the token would be the same email-keyed
takeover under a friendlier name. The email must still match the account's own,
because login resolves accounts by the asserted address, and one provider
subject may not be linked to two accounts.
Linking DELETES the password rather than keeping it alongside. One credential at
a time, and the confirmation says so in those words, because a password left
behind is a second way in that the user believes they replaced. Unlink therefore
takes the new password up front and writes it in the SAME statement as the
unlink -- never unlink now and set a password after, which leaves an account
briefly, or on failure permanently, with no way in.
Instance-wide providers only. An organization's provider is chosen by a
customer; letting one attach itself to a platform account would hand that
customer whatever the account can do.
IDENTIFIER-FIRST. The password box now appears only after an address has been
submitted, which is what lets the organization lookup happen before a credential
is offered: someone whose company requires its own provider is shown that,
rather than a password box that will be refused. Editing the address returns to
the identifier step so a corrected domain gets a fresh answer.
The per-keystroke lookup is gone with it. It answered for half-typed domains,
changed the form under someone mid-address, and spent a 10/min per-IP budget on
people who had not finished typing -- an office behind one address could exhaust
it without a single sign-in attempt.
Instance-wide providers stay visible at all times now, by decision: the server
refuses them for an SSO-only organization anyway, and hiding them made the page
change shape while typing.
Verified in a real browser, not only by rendering: password hidden -> submit ->
visible and focused -> edit the address -> hidden again, with no page errors.
Four mutations of the linking rules fail the tests (account from the email
instead of the session, keeping the password, allowing org providers, dropping
requireAuth).
The previous fix let the instance-wide Microsoft button work and left the
customer-facing path broken, which is the worst way round. An organization that
brings its own Entra tenant would publish the TXT record, watch its domain go
green, and still be refused at login with `email_unverified` -- because Entra
sends no such claim and rowToProvider pinned the assumption off for every org
provider.
Requiring a claim Microsoft does not emit is not a security control, it is an
outage. What makes it safe to stop requiring it is the proof that already gates
these providers: the callback confines an org provider to its DNS-verified
domains, and an address only reaches the check after passing that. Whoever
controls a domain's DNS controls its mail, which is the same trust that makes a
verification link meaningful.
So the assumption is DERIVED from proof -- `verified.length > 0` -- rather than
pinned off. A provider that has verified nothing still assumes nothing, which is
belt and braces: emailAllowedForProvider already refuses it, since an empty
allow-list matches no domain, but deriving it here means a future reordering of
those checks cannot silently widen it.
It is never a column, and there is no column for it to be read from. An
organization must not be able to switch this on for itself; it is a consequence
of DNS proof, not a setting. A test asserts both -- that the value is derived
next to `source: 'org'`, and that no `assume_email_verified` exists in the
schema.
Domain confinement is untouched. An explicit `email_verified: false` is still
refused from anyone.
Mutations all fail the tests: assuming unconditionally, never assuming, and
reading it from the row.
The OIDC callback required `claims.email_verified === true`. Entra ID v2 does
not send that claim at all, so every Microsoft login authenticated correctly
against the tenant and was then refused with `email_unverified` on the way back.
Nothing caught it: the SSO tests assert how the Microsoft issuer string is built
but never put a Microsoft-shaped token through the policy.
The strict check was itself a fix -- `=== false` had been accepting an omitted
claim -- and it is right for a provider a CUSTOMER configured, since such a
provider is chosen by the party it vouches for and its bare assertion is worth
nothing. What was wrong is treating that as a question about the token when it
is a question about who we trusted. `users.email_verified` is our own state; the
claim is the IdP's. An instance-wide provider was chosen by the operator -- the
same trust that already exempts it from domain confinement -- and Microsoft is
additionally pinned to one tenant GUID, so only that directory can issue a token
whose `iss` matches.
So the policy now depends on the provider, in emailIsVerified(), next to the
flag it reads so the two cannot drift:
- explicit true -> believed, from anyone
- claim absent, operator -> believed (Microsoft; opt-in for other IdPs)
- claim absent, org -> refused
- explicit false -> refused, always
Org providers pin the flag false in rowToProvider and never read it from the
row, so the takeover path the strict check existed to close stays closed.
Google is left strict: it does send the claim.
Also documents MICROSOFT_CLIENT_SECRET (supported in code, missing from the
table), that the redirect URI must be registered under Web rather than SPA, and
the email optional claim -- the other two ways an Entra setup fails.
All three mutations of this policy fail the new tests: reinstating the strict
check (3 failures), letting an org provider assume (1), and accepting an
explicit false (2).
