mirror of
https://github.com/screentinker/screentinker.git
synced 2026-08-13 13:53:12 -06:00
59489b3b20
Bold reported loop lag that grew with uptime and reset on restart. The signature they saw — mean = p50 = p99 = max, identical to two decimals — is not a fixed cost paid on every cycle. It is what an IntervalHistogram window reports when it recorded exactly ONE delay: the mean is the raw value, and every percentile returns the bucket ceiling above it. Reproduced against their exact numbers (1329.07 / 1329.59). So the loop took one long turn that swallowed the sampling second, episodically — which is what they later confirmed independently. The turn is ours, and it is now measured rather than theorised. Probing the real worker against a real WAL with one reader mid-transaction: a single main-thread write blocked for 4,936ms behind the worker's wal_checkpoint(TRUNCATE), which then reported WAL 8.8MB -> 8.8MB. TRUNCATE is the blocking form and its locks are held ACROSS connections, so moving it to a worker kept the fsync off the loop but not the lock; and it does not throw when it cannot get those locks, it returns busy=1 having sat on SQLite's 5s busy timeout and reclaimed nothing. Five seconds of stalled loop for zero benefit, and silent. It was reached far too easily. The rule was "escalate if the WAL grew across three consecutive 15s runs" — which any sustained 45-second write burst satisfies. A customer's fleet powering on in the morning does it daily. Two gates, because either alone leaves the hole open. A size FLOOR, so a WAL in the lower half of its budget can't buy a blocking checkpoint it has nothing to reclaim from. And a COOLDOWN, because the floor alone fixes nothing for Bold — their WAL already sits at 6.2MB against a 16MB high-water, above any sane floor, so every burst would still escalate. However long the pressure lasts, our own maintenance may now stall the loop at most once per window. The high-water rule bypasses both and is untouched: a runaway WAL is the one case worth blocking for, so the "WAL cannot grow forever" invariant is exactly as strong as before. A busy TRUNCATE now says so in the log instead of reading like a success. Also softened the adjacent path: when the worker is declared unrecoverable, engageFallback() re-arms inline autocheckpoint on the main connection — a state that is STICKY for the life of the process, i.e. exactly the shape of "degrades with uptime, a restart fixes it". It used to also run an unconditional main-thread TRUNCATE on the way in; that now happens only when the WAL is genuinely over high-water, and the fallback state is served on /api/status rather than being inferable only from a log line that may have rolled. Telemetry, so the next report is self-explanatory: loop_lag carries `samples` (~50 when healthy, 1 when a single turn swallowed the second), `tick_gap_ms` measured on the WALL CLOCK independently of the histogram, and `worst_tick_gap_ms`/`worst_tick_at` — monotone, so five-minute polling can no longer miss an episode. Band semantics are deliberately unchanged. A one-sample window during a real stall is the correct trigger for the shed valve; suppressing it would blind the protection at exactly the moment it is needed. Separately, device_telemetry gets the age sweep it never had. The per-heartbeat row cap only ever trims the device whose heartbeat is being handled, so a device that STOPS reporting leaves its rows behind forever. The new sweep is per-device (rides idx_telemetry_device rather than scanning), chunked and yielding like the device_status_log one, and defaults to 30 days to match the uptime report's own default window — so it cannot remove rows that report would have shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014skWYXJUWhF73EvNPgB2AS |
||
|---|---|---|
| .. | ||
| activity.js | ||
| admin.js | ||
| agency.js | ||
| ai.js | ||
| assignments.js | ||
| auth.js | ||
| billing.js | ||
| contact.js | ||
| content.js | ||
| device-groups.js | ||
| devices.js | ||
| folders.js | ||
| kiosk.js | ||
| layouts.js | ||
| media.js | ||
| pip.js | ||
| player-debug.js | ||
| playlists.js | ||
| provisioning.js | ||
| reports.js | ||
| schedules.js | ||
| status.js | ||
| stripe.js | ||
| subscription.js | ||
| teams.js | ||
| tokens.js | ||
| video-walls.js | ||
| white-label.js | ||
| widgets.js | ||
| workspaces.js | ||