Boot: install missing dependencies and rebuild the native module before starting

scripts/upgrade.sh already runs `npm ci`, so this is not for the normal path. It is
for the ways a box ends up with the wrong node_modules, both of which present as
"server will not start" with an error naming a file rather than the action needed:

  ROLLBACK      checking out an older tag to back out a bad release restores that
                tag's package.json but not its packages. This branch removes
                google-auth-library, so a rollback to main would not boot — and you
                are rolling back because something else already broke.
  NODE UPGRADE  better-sqlite3 is compiled against one ABI. Upgrading Node makes every
                boot fail with NODE_MODULE_VERSION, which reads like database
                corruption and is not.

Runs as the FIRST statement in server.js, before any dependency is required, and uses
only Node builtins — anything it imported could be the thing that is missing. Repairs
with `npm install --omit=dev` (never `ci` on a partly-populated tree, which would
delete a working node_modules to fix one package) or `npm rebuild better-sqlite3`, and
exits with the command to run if it cannot. ST_SKIP_DEP_PREFLIGHT=1 opts out.

⚠️ The first version of the native check was WRONG and I caught it only by running it
under a real version mismatch: better-sqlite3's entry point is plain JavaScript that
loads the compiled binding lazily, so `require()` succeeds under a Node the binary was
never built for. It reported a genuinely broken install as healthy. It now opens an
in-memory database, which is what actually pulls the binding in. A test pins that,
because the failure is invisible — the check keeps passing on every machine where
nothing is wrong.

Verified: a deleted dependency is detected, installed and the server boots (200); the
ABI mismatch is detected under Node 18 against a module built for Node 20 and reported
clean under Node 20; a healthy tree costs 8ms and touches no network.

1609 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvjey4FNam49MN7ybjcq6A

README.md

@@ -518,6 +518,24 @@ own admins.
 locked out until an operator acts. That is the intended trade — deliberate friction on the dangerous
 direction — but it should be a decision, not a surprise.
 
+#### Dependency preflight on boot
+
+Before anything else is loaded, the server checks that the packages this build declares are actually
+installed and that the native database module loads under the running Node. If either is wrong it
+repairs it (`npm install --omit=dev`, or `npm rebuild better-sqlite3`) and continues; if it cannot,
+it exits saying what to run rather than dying on a `MODULE_NOT_FOUND` naming a file.
+
+`scripts/upgrade.sh` already installs dependencies, so this is not for the normal path. It is for
+the ways a box ends up with the wrong `node_modules`:
+
+- **rolling back** to an older tag restores that tag's `package.json` but not its packages — and you
+  are rolling back because something is already wrong;
+- **upgrading Node** leaves `better-sqlite3` compiled against the previous ABI, which fails in a way
+  that reads like database corruption and is not.
+
+Set `ST_SKIP_DEP_PREFLIGHT=1` on an air-gapped host, or anywhere you manage `node_modules` yourself
+and do not want a boot reaching for the registry.
+
 #### Email (Microsoft Graph or SMTP)
 
 Email powers offline alerts, welcome/signup mail, admin notifications, and password reset. Two interchangeable transports are supported, selected by `EMAIL_TRANSPORT`:

server/lib/preflight-deps.js

@@ -0,0 +1,143 @@
+'use strict';
+
+/*
+ * Make sure the dependencies this build needs are actually installed and loadable — BEFORE anything
+ * requires them.
+ *
+ * The normal upgrade path (scripts/upgrade.sh) runs `npm ci --omit=dev`, so this is not for the
+ * happy case. It is for the three ways a running box ends up with the wrong node_modules:
+ *
+ *   ROLLBACK      checking out an older tag to back out a bad release restores that tag's
+ *                 package.json but not its packages, so the server dies on a MODULE_NOT_FOUND for
+ *                 something the newer build had removed. That is a bad moment to be reading a
+ *                 stack trace: you are already rolling back because something else broke.
+ *   NODE UPGRADE  better-sqlite3 is a native module compiled against one ABI. Upgrading Node makes
+ *                 every boot fail with NODE_MODULE_VERSION mismatch, which reads like database
+ *                 corruption and is not.
+ *   HAND EDITS    a `git checkout`, a partly-copied tree, an interrupted install.
+ *
+ * All three present as a server that will not start, with an error that names a file rather than
+ * the action needed. Detecting and repairing is a few seconds; diagnosing is an outage.
+ *
+ * ⚠️ Deliberately dependency-free — only Node builtins. Anything it required could be the very
+ * thing that is missing.
+ *
+ * Set ST_SKIP_DEP_PREFLIGHT=1 to turn it off (air-gapped hosts, or an operator who manages
+ * node_modules themselves and does not want a boot reaching for the network).
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { execFileSync } = require('child_process');
+
+const SERVER_DIR = path.join(__dirname, '..');
+const NODE_MODULES = path.join(SERVER_DIR, 'node_modules');
+const INSTALL_TIMEOUT_MS = 10 * 60 * 1000;   // a cold install on a Pi is genuinely slow
+
+/** Which declared dependencies are not on disk. */
+function missingDeps() {
+  let pkg;
+  try {
+    pkg = JSON.parse(fs.readFileSync(path.join(SERVER_DIR, 'package.json'), 'utf8'));
+  } catch {
+    return [];   // no package.json is not our problem to diagnose
+  }
+  const declared = Object.keys(pkg.dependencies || {});
+  return declared.filter((name) => {
+    // A scoped or nested name is still one directory below node_modules.
+    try { return !fs.existsSync(path.join(NODE_MODULES, name, 'package.json')); } catch { return true; }
+  });
+}
+
+/**
+ * Is the native module loadable by THIS Node?
+ *
+ * Checked by actually loading it, because the failure is an ABI mismatch that no version string
+ * comparison catches reliably — a rebuild against the same major can still differ.
+ */
+function nativeModuleBroken() {
+  try {
+    /*
+     * ⚠️ CONSTRUCT one, do not merely require it.
+     *
+     * better-sqlite3's entry point is plain JavaScript and loads the compiled `.node` binding
+     * lazily, so `require()` alone SUCCEEDS under a Node whose ABI the binary was not built for —
+     * the first version of this check did exactly that and reported a broken install as healthy,
+     * verified against a real Node 18 / Node 20 mismatch. Opening an in-memory database is what
+     * actually pulls the binding in, and it touches no file.
+     */
+    const Database = require('better-sqlite3');
+    new Database(':memory:').close();
+    return null;
+  } catch (e) {
+    const msg = String((e && e.message) || '');
+    if (/NODE_MODULE_VERSION|ERR_DLOPEN_FAILED|was compiled against a different/i.test(msg)) return msg;
+    if (/Cannot find module/i.test(msg)) return msg;
+    // Anything else is a real error in the module, not an installation problem — let it surface
+    // later with its own stack rather than being masked by an npm run.
+    return null;
+  }
+}
+
+function run(args, label) {
+  console.log(`[preflight] ${label}: npm ${args.join(' ')}`);
+  execFileSync('npm', args, { cwd: SERVER_DIR, stdio: 'inherit', timeout: INSTALL_TIMEOUT_MS });
+}
+
+function fail(reason, hint) {
+  console.error(`[preflight] ${reason}`);
+  console.error(`[preflight] ${hint}`);
+  console.error('[preflight] Set ST_SKIP_DEP_PREFLIGHT=1 to boot without this check.');
+  process.exit(1);
+}
+
+function preflight() {
+  if (process.env.ST_SKIP_DEP_PREFLIGHT === '1') return;
+
+  const missing = missingDeps();
+  const nodeModulesAbsent = !fs.existsSync(NODE_MODULES);
+
+  if (missing.length || nodeModulesAbsent) {
+    const what = nodeModulesAbsent
+      ? 'node_modules is missing'
+      : `${missing.length} dependency/dependencies missing: ${missing.slice(0, 6).join(', ')}${missing.length > 6 ? '…' : ''}`;
+    console.warn(`[preflight] ${what} — installing before start.`);
+    try {
+      /*
+       * `npm ci` when there is a lockfile and nothing installed: it is reproducible and it is what
+       * upgrade.sh uses. Otherwise `npm install`, because `ci` DELETES node_modules first and would
+       * throw away a working tree to fix one missing package.
+       */
+      const hasLock = fs.existsSync(path.join(SERVER_DIR, 'package-lock.json'));
+      if (hasLock && nodeModulesAbsent) run(['ci', '--omit=dev', '--no-audit', '--no-fund'], 'installing');
+      else run(['install', '--omit=dev', '--no-audit', '--no-fund'], 'installing');
+    } catch (e) {
+      fail(`could not install dependencies: ${e && e.message}`,
+        'Run `npm ci --omit=dev` in the server directory, or check network access to the npm registry.');
+    }
+    const still = missingDeps();
+    if (still.length) {
+      fail(`still missing after install: ${still.join(', ')}`, 'Check the npm output above.');
+    }
+    console.log('[preflight] dependencies installed.');
+  }
+
+  const nativeProblem = nativeModuleBroken();
+  if (nativeProblem) {
+    console.warn(`[preflight] better-sqlite3 will not load under Node ${process.version} — rebuilding.`);
+    console.warn(`[preflight]   ${nativeProblem.split('\n')[0]}`);
+    try {
+      run(['rebuild', 'better-sqlite3'], 'rebuilding native module');
+    } catch (e) {
+      fail(`could not rebuild better-sqlite3: ${e && e.message}`,
+        `Run \`npm rebuild better-sqlite3\` in the server directory. This usually means Node changed version (now ${process.version}) and the module needs recompiling; a build toolchain (python3, make, g++) must be present.`);
+    }
+    if (nativeModuleBroken()) {
+      fail('better-sqlite3 still will not load after a rebuild.',
+        'Delete server/node_modules and run `npm ci --omit=dev`.');
+    }
+    console.log('[preflight] native module rebuilt.');
+  }
+}
+
+module.exports = { preflight, missingDeps, nativeModuleBroken };

server/server.js

@@ -1,3 +1,14 @@
+/*
+ * FIRST, before any dependency is required: make sure they are installed and loadable.
+ *
+ * A rollback restores an older package.json but not its packages, and a Node upgrade leaves the
+ * native database module compiled against the wrong ABI. Both present as a server that will not
+ * start, with an error naming a file rather than the action needed — and the rollback case happens
+ * precisely when something else has already gone wrong. Repairing takes seconds; diagnosing at 2am
+ * does not. ST_SKIP_DEP_PREFLIGHT=1 turns it off.
+ */
+require('./lib/preflight-deps').preflight();
+
 const express = require('express');
 const http = require('http');
 const https = require('https');

server/test/preflight-deps.test.js

@@ -0,0 +1,83 @@
+'use strict';
+
+/*
+ * The boot-time dependency check.
+ *
+ * It exists for the moments nobody is at their best: a rollback that restores an older
+ * package.json but not its packages, and a Node upgrade that leaves the native database module
+ * compiled against the wrong ABI. Both present as "server will not start", with an error naming a
+ * file rather than the action needed.
+ */
+
+const { test } = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+
+const preflight = require('../lib/preflight-deps');
+
+test('a healthy install reports nothing missing and nothing broken', () => {
+  assert.deepEqual(preflight.missingDeps(), [], 'this tree should be complete');
+  assert.equal(preflight.nativeModuleBroken(), null, 'and the native module should load');
+});
+
+test('THE NATIVE CHECK CONSTRUCTS A DATABASE, it does not merely require the module', () => {
+  /*
+   * better-sqlite3's entry point is plain JavaScript that loads the compiled binding lazily, so
+   * `require()` SUCCEEDS under a Node whose ABI the binary was never built for. The first version
+   * of this check stopped at require and therefore reported a genuinely broken install — verified
+   * against a real Node 18 / Node 20 mismatch — as healthy.
+   *
+   * Pinned as source because the failure is invisible: the check keeps passing, on every machine
+   * where nothing is wrong, right up until the one where something is.
+   */
+  const src = fs.readFileSync(path.join(__dirname, '..', 'lib', 'preflight-deps.js'), 'utf8');
+  const fn = src.slice(src.indexOf('function nativeModuleBroken'), src.indexOf('function run('));
+  assert.match(fn, /new Database\(':memory:'\)/,
+    'nativeModuleBroken must open a database, or an ABI mismatch goes undetected');
+  assert.ok(!/^\s*require\('better-sqlite3'\);\s*$/m.test(fn), 'a bare require is not a load test');
+});
+
+test('missingDeps agrees with what is actually on disk', () => {
+  const pkg = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'package.json'), 'utf8'));
+  const declared = Object.keys(pkg.dependencies || {});
+  assert.ok(declared.length > 0, 'the server declares dependencies');
+  const reported = preflight.missingDeps();
+  for (const name of declared) {
+    const present = fs.existsSync(path.join(__dirname, '..', 'node_modules', name, 'package.json'));
+    assert.equal(present, !reported.includes(name), `${name}: presence and report disagree`);
+  }
+});
+
+test('the preflight uses only Node builtins', () => {
+  /*
+   * It runs BEFORE dependencies are installed, so anything it imported could be the very thing
+   * that is missing — and the failure would be the one it exists to prevent, with an extra layer
+   * of confusion on top.
+   */
+  const src = fs.readFileSync(path.join(__dirname, '..', 'lib', 'preflight-deps.js'), 'utf8');
+  const requires = [...src.matchAll(/require\('([^']+)'\)/g)].map((m) => m[1]);
+  const builtins = new Set(['fs', 'path', 'child_process', 'os', 'crypto', 'util']);
+  for (const r of requires) {
+    // better-sqlite3 is the thing being TESTED for loadability, not a dependency of this file.
+    if (r === 'better-sqlite3') continue;
+    assert.ok(builtins.has(r) || r.startsWith('.'), `preflight must not depend on ${r}`);
+  }
+});
+
+test('it can be turned off for an air-gapped host', () => {
+  const src = fs.readFileSync(path.join(__dirname, '..', 'lib', 'preflight-deps.js'), 'utf8');
+  assert.match(src, /ST_SKIP_DEP_PREFLIGHT/, 'an operator who manages node_modules must be able to opt out');
+  const saved = process.env.ST_SKIP_DEP_PREFLIGHT;
+  process.env.ST_SKIP_DEP_PREFLIGHT = '1';
+  try { assert.doesNotThrow(() => preflight.preflight(), 'opting out must be a clean no-op'); }
+  finally { if (saved === undefined) delete process.env.ST_SKIP_DEP_PREFLIGHT; else process.env.ST_SKIP_DEP_PREFLIGHT = saved; }
+});
+
+test('server.js runs the preflight BEFORE requiring anything', () => {
+  const src = fs.readFileSync(path.join(__dirname, '..', 'server.js'), 'utf8');
+  const preflightAt = src.indexOf("require('./lib/preflight-deps')");
+  const firstDep = src.indexOf("require('express')");
+  assert.ok(preflightAt > -1, 'server.js must run the preflight');
+  assert.ok(preflightAt < firstDep, 'it must come before the first dependency, or it cannot help');
+});