* spike: replace sharp with pure-JS image ops (jimp + jsquash WASM)
Removes the last native dependency from the ingest path, so the server no longer
needs a per-platform/per-ABI prebuilt to thumbnail an image. Motivated by getting
the server onto hardware with no toolchain, but the ABI tax is paid on every
install — it is the same failure class lib/preflight-deps.js exists to explain.
lib/image-ops.js is the whole surface: metadata() and writeThumbnail(), which are
the only two things ingest ever asked sharp for.
Format parity holds. jpeg/png/gif/tiff/bmp are native to Jimp; webp and avif go
through @jsquash WASM, whose bundled .wasm must be compiled by hand because the
packages locate it with fetch(file://) and Node has no file:// fetch — the only
symptom otherwise is a bare "fetch failed". heic is unsupported, as it already
was: sharp advertises heif but its prebuilt libvips refuses HEVC.
#170 is preserved by a different mechanism. Jimp applies EXIF orientation at
decode and rewrites the tag to 1, so metadata() reports display dimensions and
imageDisplayDims() runs as a no-op instead of swapping W/H a second time. The
helper stays in the path so the rule keeps living in one place.
Verified: 1643/1643 tests pass, and ingest was exercised in a child process with
node_modules/sharp moved aside — jpeg, EXIF-rotated jpeg, png, webp, avif, gif
all measured and thumbnailed correctly, corrupt input still yields nulls with no
phantom thumbnail_path.
KNOWN BLOCKER, do not ship as-is: Jimp is pure JS on the main thread, where sharp
handed work to a libvips threadpool. A 12MP photo goes 65ms -> 1079ms, and the
event loop stalls for 1003ms of it (sharp: zero stalls). thumbnail-backfill.js
walks a whole library at boot, so this reproduces #240 exactly — blocked loop,
missed heartbeats, panels marked offline, reconnect churn. Needs a worker_thread
offload before this is viable; image-ops.js is the seam for it.
* Run image decoding on a worker thread
Fixes the blocker the previous commit shipped with. Pure-JS decoding costs ~1s of
solid CPU for a 12MP photo, and in-process that is not a slow upload but a stalled
event loop — no heartbeats, no socket traffic. thumbnail-backfill.js walks a whole
library at boot, so it reproduced #240 (blocked loop -> missed heartbeats -> panels
offline -> reconnect churn) from our own maintenance. sharp never did this because
libvips works on a threadpool.
image-ops.js is now a dispatcher over image-ops-worker.js; the work moved unchanged
to image-ops-core.js, so callers and their failure contract are untouched.
Measured on a 12MP photo: 1079ms wall with the loop stalled 1003ms, to 1881ms wall
for two ops with ZERO stalls and 185 timer ticks serviced. Wall time is worse and
that is fine — it is off the main thread now.
Design notes, all load-bearing:
- ONE JOB AT A TIME. A decoded 12MP bitmap is ~48MB of RGBA; overlapping jobs
multiply peak memory by queue depth, which is the wrong failure on the small
targets this change exists to reach. Costs no throughput — the work is CPU-bound
and one busy worker already saturates its core.
- unref'd while idle, ref'd only in flight. Otherwise scripts/backfill-rotation-
dims.js never exits and `node --test` hangs forever. Verified: a CLI-style run
exits in 104ms, code 0.
- decode failures reply as messages, so one bad upload cannot tear down the worker
and take unrelated queued jobs with it.
- in-process fallback if a thread cannot be had, warned rather than silent.
test/image-ops.test.js pins the loop-liveness property, which no functional test
would catch. Its thresholds were mutation-tested against the inline path: the first
version passed there too (4MP stalls only ~355ms, under a non-flaky threshold), so
the fixture is 12MP and the thresholds sit in the gap between the two behaviours —
worker ~90 ticks/~0ms, inline ~3 ticks/~897ms. It now fails inline, as a guard must.
1647/1647 pass. Ingest re-verified with node_modules/sharp moved aside.
* Measure and thumbnail an image from a single decode
Ingest asked for metadata() then writeThumbnail(), which decoded the file twice.
That pairing was free under sharp, whose .metadata() only parses the header, but
every decode here is a full one — ~1s for a 12MP photo — so the naive translation
doubled the most expensive thing on the ingest path.
image-ops.measureAndThumbnail() returns both from one decode. Full ingest of a
12MP photo: 2 decodes/~1.9s -> 1150ms, still with zero event-loop stalls.
The subtlety is the failure contract. In the two-call version width and height
were assigned BEFORE the thumbnail was attempted, so a failed thumbnail still left
usable dimensions on the row — the player needs them to letterbox. Merging naively
would have turned any thumbnail failure into total metadata loss. So a WRITE
failure is reported ({thumbnailWritten:false, thumbnailError}) with the dimensions
intact, and the caller sets thumbnail_path only when the write succeeded, keeping
the phantom-path discipline. A DECODE failure still throws — there is nothing to
report about an unreadable image.
backfill-rotation-dims.js deliberately keeps the separate calls: it probes every
image row but regenerates a thumbnail only for the few whose dimensions changed,
so pairing them there would decode files it has no reason to thumbnail.
Tests count decodes rather than timing them — an exact property, and a wall-clock
comparison would be flaky under load. The count filters for reads of the file under
test: Node's ESM loader also goes through fs.promises.readFile, so a raw call count
picks up jimp's and the WASM codecs' lazy loading and reads 30 instead of 1.
1649/1649 pass. Ingest re-verified across all 7 formats with sharp moved aside.
* Dockerfile: sharp is no longer a production dependency
--omit=dev now leaves it out entirely; better-sqlite3 is the only native module
the builder stage still needs a toolchain for.
A second review pass, run against the previous commit, found four blockers — two of
them introduced by the fixes in that commit. It also confirmed the original account
takeover is closed: a hostile IdP with real TLS, discovery, JWKS and RS256 driving the
real routers now stops at domain_not_allowed, and all 16 bypass variants are refused.
DOMAIN OWNERSHIP (the root cause, not the symptom)
A claimed domain used to mean "nobody else claimed it". It now means the organization
published a record in that domain's own DNS — TXT or CNAME, at a dedicated
_screentinker-verify name rather than the apex, where an edit would sit beside SPF.
- an unverified domain routes NOBODY and cannot be asserted; it reserves the name
- an unverified claim LAPSES after 8 hours, so a domain cannot be held against its
real owner, and lapsing rotates the token so a record left over from an abandoned
attempt cannot satisfy a later claim
- a verified domain never expires — re-proving on a timer would log a customer out
over a DNS edit made months later
- routing and confinement read the VERIFIED set only, never the typed column
- configuring SSO now requires a verified email address
- platform admins are emailed when a domain is claimed; nothing is ever sent to the
claimed domain, which would let any tenant make this product email third parties
Instance-wide providers are exempt from all of it: they are the operator's own
configuration and keep the trust they have always had.
BLOCKERS FROM THE REVIEW
- two unauthenticated remote crashes, both one request, both "async handler throws
before its try": `Cookie: st_oidc_tx=%` (unguarded decodeURIComponent) and the
fail-closed secret added last commit, which turned a JWT_SECRET rotation into a
permanent crash loop. Fixed the CLASS with asyncRoute() rather than the instances.
- the SSRF guard was bypassable via IPv4-mapped IPv6 ([::ffff:127.0.0.1]) and also
refused every host beginning "fc"/"fd" (fcm.googleapis.com). Addresses are now
parsed and compared by RANGE. 42 cases verified.
- the takeover fix had NO test — the test named after it asserted two struct fields
and passed with the guard deleted. The decision is now a pure function and four
mutations were confirmed to turn the suite red.
- the PUT path never received the TOCTOU fix, so two orgs could end up holding one
domain and forEmail handed routing to the attacker's older row.
ALSO
- linking compared slugs, so an org could never rotate its own IdP, and fell open on
an empty auth_provider. It now asks which ORGANIZATION owns the slug.
- an account stranded by a deleted provider can be reclaimed by password reset —
proof of the mailbox, which is stronger than the IdP assertion that created it.
- /sso/claim accepted a pre-TOTP mfa_pending token and returned the full user row;
it now takes a purpose-built 120s claim token with a pinned algorithm and typ.
- the rate limiter keyed on a caller-controlled path, so a trailing slash bought a
fresh bucket — a real login brute-force bypass.
- domain_not_allowed and account_exists_other_provider rendered as "please try
again", advice that can never work.
- malformed asserted addresses are refused rather than trimmed into shape.
- dead config (microsoftTenantId defaulted to 'common', which the provider code now
refuses) and the orphaned google-auth-library dependency removed.
1591 tests pass. Domain lifecycle verified end to end against a running server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A
Two mistakes in the previous commit, both of which broke CI.
The lockfile was not regenerated after adding puppeteer-core to
devDependencies, and `npm ci` requires the two to agree — so every job that
installs dependencies failed before running anything.
The smoke test was also placed in test/, which I described as keeping it out of
`npm test`. It does not: `node --test` globs that directory, so the runner
picked it up regardless of intent, tried to drive a browser as a unit test, and
failed. It now lives beside the server as smoke-ui.js, with a note saying why,
so the next person does not put it back.
Verified the way it should have been the first time: npm ci succeeds, native
modules still load, npm test is 807/807 with no browser involved, and
`npm run smoke` is 32/32 on its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uaeo9MvzKoyXuN6ZsbhtkL
A whole class of defect found today was invisible to the unit suite, to a
syntax check and to review, and appeared only in front of a browser: a context
menu whose only item read "schedule.ctx_new", pointer handlers stacking on every
calendar render so one drop fired five PUTs, and a week grid that scrolled
sideways on a phone. Nothing in the repo could have caught any of them.
This keeps the checks that earned their place and throws away the scratch
scripts around them. It boots a server, drives every view, and asserts each view
renders, none raises an uncaught error, no untranslated key reaches the screen,
the calendar binds its handlers once however many times it re-renders, and
nothing overflows horizontally at phone width.
Deliberately NOT part of `npm test`. It needs a real browser, which CI does not
have, so it is `npm run smoke` and exits 0 with an explanation when puppeteer or
Chrome is missing — a test that fails for want of tooling teaches people to
ignore failures. puppeteer-core rather than puppeteer, so installing it does not
pull down a private copy of Chrome; it drives whichever one is already there.
Verified both ways: 32/32 against current main, and it fails on the listener
stacking when that fix is reverted. The missing-key case is covered by the unit
guard instead, since a context menu only exists once it has been opened.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uaeo9MvzKoyXuN6ZsbhtkL
sharp decodes uploaded files directly (lib/content-ingest.js, routes/content.js
both call sharp(file.path) on whatever a user uploaded), so its bundled libvips is
part of the request path rather than a build-time detail. Moves 0.33.5 -> 0.35.3,
libvips 8.15 -> 8.18.
Validated against the calls this codebase actually makes, because it is a major
bump: metadata() still reports EXIF orientation (1/3/6/8 all round-trip, which is
what lib/media-orientation.js exifSwapsWH and the rotation-aware dimensions depend
on), a bare .rotate() still auto-orients, and resize().jpeg().toFile() is unchanged.
png/webp/jpeg/gif/avif all still encode and decode, and malformed input still throws
rather than crashing.
The new libpng is stricter, which surfaced a latent problem in the AUTH-01 test: its
1x1 PNG literal had a corrupt IDAT chunk whose stored CRC did not match its data. The
old decoder accepted it; the new one refuses with "vipspng: libpng read error", so no
thumbnail was written and the content-gate assertions failed with a 404 that reads
like an auth regression. Replaced with a PNG whose every chunk CRC verifies. The
stricter decode is the correct behaviour and is kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a pluggable email transport so self-hosters without Azure/M365 can send
mail through any standard SMTP server (Postfix, Gmail, Mailgun, SendGrid, corp
relay). Graph stays the default; behavior is byte-for-byte unchanged when
EMAIL_TRANSPORT is unset or "graph".
- config: EMAIL_TRANSPORT ("graph"|"smtp", default graph) + SMTP_HOST/PORT/
SECURE/USER/PASSWORD/FROM.
- services/email.js: branch by transport behind the SAME public sendEmail()/
isConfigured() surface. SMTP via nodemailer (lazy-required, like MSAL).
Shared across both transports: the "[ScreenTinker] " subject prefix (unless
rawSubject), the GRAPH_DEV_RESTRICT_TO allow-list, html-from-text derivation,
and the never-throws contract (failures log + return sent:false). SMTP_SECURE
true=implicit TLS(465)/false=STARTTLS(587). Auth optional (unauthenticated
relay ok); SMTP_USER without SMTP_PASSWORD is flagged. SMTP_FROM parses
"Name <addr>". New emailConfigStatus() for startup diagnostics.
- server.js: startup logs the transport and a LOUD error when the selected
transport is partially configured (some fields set, others missing) or when
EMAIL_TRANSPORT is invalid (falls back to graph). A fully-unset transport
stays a silent stdout fallback (unchanged dev behavior).
- nodemailer ^6.9.16 added as a production dep (bundled in the Docker image).
- .env.example + README: SMTP config section, Gmail example, transport table.
- test/email-transport.test.js: 15 tests — transport selection, config
validation (missing/partial/invalid), SMTP message building (from/prefix/
fromName override/text alt), sendEmail routing (mocked nodemailer), rawSubject,
dev-restrict on smtp, and the smtp_error never-throws path.
462/462 server tests pass. Boot verified for all four states (configured,
misconfigured, invalid, default).
Closes#173
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
