1.9.26's opt-in was passive: it stopped a sideloaded build being reverted, but there was still one
APK slot and latest_version was the server's own VERSION, so a beta had to be installed by hand on
every display. This makes it a real channel.
- apk-cache tracks two slots. ScreenTinker-beta.apk is optional and reaches only displays with
ota_beta = 1.
- A beta must DECLARE its version in a sidecar ScreenTinker-beta.apk.version. The server cannot
infer it — stable's version is the server's own constant because the two ship together, and
reading it from the APK means parsing binary AndroidManifest.xml on the request path. If the
sidecar is missing or unparseable the channel does not activate at all and opted-in displays keep
getting stable. Failing closed matters: advertising a version that does not match the bytes served
is the OTA-loop condition this fleet has been bitten by before.
- The check and the download resolve the channel identically and fall back to stable identically, so
apk_size always describes the bytes actually delivered. No APK change was needed — the client
already fetches whatever download_url it is handed, so displays in the field can be moved between
channels from the dashboard today.
Switching back needed care. Stable is semver-OLDER than the beta it replaces, so the ordinary
"never offer a downgrade" rule stranded the display and unticking the box would have been another
silent no-op. The first attempt returned any non-opted-in display running a pre-release — which
broke a #144 test, correctly: that would have dragged every existing pre-release tester back to
stable the moment their server upgraded, the exact harm the opt-in exists to prevent. So the return
now requires evidence we actually served that display the beta channel (devices.ota_channel_served,
written once on change, not per check). A tester ahead of the server on their own build is left
alone exactly as before.
Documented in the README, including the constraint that makes the switch-back physically possible:
beta builds must carry a versionCode no higher than the stable they branch from, because Android
refuses to install a lower one. Equal numbers install in both directions.
Verified end to end against a live server with two real signed APKs: stable serves 1.9.26, beta
serves 1.9.27-rc1, an unknown channel falls back to stable, removing the version file deactivates
the channel, and the full opt-in -> serve -> switch-back lifecycle produces offer / up-to-date /
channel-return in order. 859 server tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uaeo9MvzKoyXuN6ZsbhtkL
Handing someone a test build was a trap. A prerelease sorts BELOW its own release — 1.9.25-fix234d
is semver-older than 1.9.25 — so a sideloaded display asked "anything newer?", was correctly told
yes, and updated itself straight back off the build we had asked someone to test. Same versionCode,
so Android installed it without complaint. Silent, and within minutes.
That is what happened on #234: the reporter installed the fix, tested for an evening, and reported
nothing had changed. They were right. Their tablet was running the old code again by then, and I had
told them it was fixed without ever checking what the device reported.
Adds a per-display opt-in (devices.ota_beta, default 0, checkbox next to the OTA toggle). When set,
the display keeps a prerelease of the CURRENT core instead of being pulled back to its release.
Deliberately narrow in one direction and deliberately wide in the other:
- Narrow: it only holds a prerelease of the core already installed. A plain release, a -patchN
build, an upgrade to a newer core, and a display ahead of the server all behave exactly as before,
and the flag defaults off so a fleet that never sets it is unaffected.
- Wide: an opted-in display is exempted from the superseded-prerelease guard. That guard would
otherwise pin a tester on an old test build permanently — an older-core prerelease is never
offered anything, so they would have to notice and sideload their way out. Writing the test is
what surfaced that; opting in must never mean never updating again.
9 tests covering both directions, including that shipping a newer release pulls a beta display back
onto the release line. 845 server tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uaeo9MvzKoyXuN6ZsbhtkL
The update check is deliberately unauthenticated — every client version has to be
able to ask, including old ones that never learned to send a token — and it keys the
rate breaker on the caller-supplied device_id. Keying on IP is not available either:
the fleet SNATs behind one address, so per-IP would collapse a whole site into a
single bucket.
The result was that the bucket belonged to whoever cited the id rather than to the
device that owns it. A handful of requests naming a panel's UUID left that panel in
rate-backoff, un-updatable for up to half an hour at a time and renewable
indefinitely, while every other device stayed healthy.
Rather than adding auth (which would strand old clients) the state is now
self-healing: when a device registers on the /device socket with a valid device_token
its bucket is cleared. Noise is still possible, but it now lasts until the panel's
next genuine reconnect instead of as long as someone keeps poking.
This is not an escape hatch from the breaker's real job. A device stuck in an update
loop is re-registering legitimately, and clearing its rate state on each genuine
reconnect is what a healthy device looks like; the loop protection that matters is
the download guard. The version-keyed bucket that covers old clients sending only
?version= is a separate namespace and is deliberately not reachable this way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The -patchN scheme (e.g. 1.9.2-patch3) parses as a semver prerelease, so decide()'s
superseded-prerelease guard refused to offer a newer stable core (1.9.3) to the existing fleet —
stranding every 1.9.2-patchN device on OTA (Force Update didn't help; the re-check re-returned
superseded-prerelease). isReleased() now counts -patchN as a shipped release, so those devices get
offered 1.9.3 via normal OTA, while GENUINE prereleases (-beta/-rc/-alpha) keep prerelease semantics
and newer cores are never downgraded. 6 new tests + 14 existing OTA tests green (388/388 suite).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The debug block exposed only gauges (buckets, quarantined, inFlight) — state, not work.
A real flapping Firestick reads as flap.buckets:36, quarantined:0, indistinguishable
from healthy. Add lightweight in-memory throughput counters (total + last-completed
rolling window) so the server tells the flapper/flood story itself.
- lib/rolling-counter.js: shared bounded scalar counter (total, curWindow, lastWindow,
windowStart); rolls lazily on bump AND read (no timer), idle decays to 0.
DEBUG_STATS_WINDOW_MS default 60000.
- flap-limiter: refused{Total,LastWindow} (every allow:false), quarantineStarts{Total,
LastWindow} (a quarantine event stays visible after the gauge decays).
- ota-breaker: stats() rateBackoff{Total,LastWindow}.
- ota-download-guard: servedTotal/shedTotal alongside the per-window values.
- database: maintenance sweepsTotal (confirm the prune is firing, not stalled).
- routes/status: debug block gains ota_breaker + the new fields (aggregate-only, cheap).
Tests: rolling-counter window-roll + idle decay; each counter increments on the right
event; booted /api/status asserts the new fields present + numeric. Suite 285/285.
Fallout doc: observability section lists the fields + what each tells a soak-watcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second head of the OTA-loop root cause (#144), on the connection/heartbeat
layer: unbounded device-driven work with no circuit-breaker. Symptoms in Bold
prod — devices shown OFFLINE in CMS while online+playing, loop-lag simmer
(p99 300-1145ms), device_status_log grown to 1.1M rows.
False-offline (two causes, both fixed):
- evicted-socket re-arm race: evictPriorSocket runs before registerConnection,
so the evicted old socket's disconnect armed a fresh offline timer for a
just-reconnected device. Tag evicted socket ids and bail in the disconnect
handler (ws/deviceSocket.js).
- heartbeat checker false-positive: a device with a live socket in /device is
UP even if its in-memory lastHeartbeat is stale under lag; skip it instead of
marking offline (services/heartbeat.js).
Storm containment:
- batched/coalescing device_status_log writer (lib/status-log-writer.js): net
state per device per flush, breaking the storm->bloat->slow-write->lag loop.
- newest-N-per-device row-count cap in the global sweep (db/database.js): hard
bound regardless of churn; trims the existing 1.1M backlog on the first sweep.
Per-device prune unified to statusLogRetentionDays (was hardcoded 7d).
- reconnect-throttle idle-bucket sweep (lib/reconnect-throttle.js): the #142
throttle already existed; added the memory-bound sweep it lacked (wired in
server.js). No second breaker.
- cosmetic: cap the OTA breaker level counter (lib/ota-breaker.js).
- best-effort status-log flush on the crash path (server.js).
Tests: load harness (test/reconnect-storm-load.test.js) proves breaker engage,
clean offline-clear, no-throttle-on-normal-reconnect, batched writes, bounded
loop-lag; cause-1 re-arm race proven with teeth (test/evicted-socket-rearm.test.js).
Both mutation-checked (fail without their fix). Full suite 240/240.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/api/update/check offered the update whenever client !== latest (raw string
inequality, not semver) with no backoff. A device that can't APPLY the update
(broken OTA client 1.7.12, signing/Fire OS) keeps reporting the same version and is
told update_available=true on every poll; a fast poll loop saturates the event loop
(prod loop-lag 49s). All requests share one NAT IP, so IP-keying is useless.
server-only breaker (lib/ota-breaker.js), two independent axes:
- RATE breaker (primary, immediate): a key checking >THRESHOLD (3) times within
WINDOW (60s) is looping -> throttle update_available with exponential backoff
(30s->2m->8m->cap 30m). Healthy devices poll ~12 min and never approach this, so
rollout/stragglers are inherently safe -- NO grace-for-flood timer; slow == safe.
- PHANTOM guard (immediate): unrecognized version, or a prerelease of an OLDER core
(superseded old-minor beta e.g. 1.9.1-beta4), gets no-offer on the first check. A
RECENT real older version (beta3 vs latest beta4; stable 1.7.12) stays offerable.
- Never offers a downgrade (client >= latest -> no offer).
KEYING (#144 option 3): keyed on device_id when present, else reported version.
- server.js:581 accepts + logs ?device_id=, passes it to the breaker.
- UpdateChecker.kt:122 appends &device_id=<config.deviceId> (existing registered id;
omitted until provisioned). One-line client change.
beta4+ clients get precise per-device throttling; stuck legacy clients sending only
?version= are caught by the version-keyed + rate + phantom logic. Response gains
additive `reason` + `retry_after_seconds` (old clients ignore).
BOUNDED STATE: a periodic sweep (startSweep, wired in server.js) evicts buckets idle
> IDLE_RESET_MS so the keyed Map can't grow unbounded (churned device_ids); not
reset-on-access only.
SCOPE (deliberate): this targets the FAST flood + phantoms. The slow #144 drip
(stable 1.7.12 polling ~every 12 min, ~20/hr) stays below >3/60s and is NOT
throttled -- catching it needs #144 option-3 "skip-this-version after N cycles",
which is intentionally NOT in this build.
NOTE: carries a CLIENT/APK change -> versionCode must increment at the beta4 bump and
the release keystore is required for the APK. The device_id path only helps devices
that can install beta4+; the stuck legacy fleet is covered by the version-keyed path.
Tests: unit (lib/ota-breaker, injected time) a-f + comparator + escalation + sweep +
slow-drip-scope; HTTP integration (real endpoint, device_id passthrough). Full suite
green serial AND parallel (234). OTA-only delta -- reconnect/reclaim/shed/content-ack/
block untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
