docs(#148): mass-disconnect + connection-lifecycle + half-open analyses

docs/148-connection-lifecycle-analysis.md

@@ -0,0 +1,143 @@
+# #148 — WebSocket connection-lifecycle analysis (idle-reap / half-open)
+
+**Status: investigation + mitigation spec. No code changed, nothing deployed.**
+
+## Headline
+
+Idle-reap by Bold's edge is a plausible **trigger**, but it is **not the whole story**, and a
+keepalive alone will not fix it. Two facts constrain the diagnosis:
+
+1. **The client already sends traffic every 15s** (`device:heartbeat`) plus receives a
+   Socket.IO ping every 30s — so a *pure idle-timer* above ~15–30s should never reap this
+   socket. If Bold's firewall still reaps it, the cause is DPI/SSL-inspection or a sub-15s
+   timeout, **not** simple idleness — and more keepalive won't help.
+2. **A Fire TV Stick survives on the SAME network + SAME server + SAME APK** (#148). Whatever
+   Bold's edge does, a well-behaved client rides through it. So the decisive difference is the
+   **client's ability to detect and recover from a silent (half-open) drop** — and the MAXHUB
+   build has a real gap there.
+
+**The fix is client self-heal + server half-open detection; the firewall config is a
+secondary trigger-reducer.** None of this is fixed by v1.9.2 (the APK socket code is
+unchanged), so this needs a new player build.
+
+---
+
+## Phase 0 — the keepalive stack (exact values, current main / 1.9.2)
+
+**Server (`server/config.js`, `server/server.js`):**
+- Socket.IO `pingInterval = 30000`, `pingTimeout = 30000` → **server→client ping every 30s**;
+  worst-case dead-socket detection = **60s**. (Engine.IO v4: the *server* pings, the client
+  pongs; these are real WebSocket frames, so a firewall that counts frames sees traffic.)
+- `heartbeatTimeout = 45000` → server marks a device **offline** after 45s with no
+  `device:heartbeat`. Marking offline **only flips DB status; it does NOT close the socket.**
+- **No `setKeepAlive`/SO_KEEPALIVE** anywhere → the OS never independently detects a dead
+  peer; the server relies solely on the 60s Engine.IO ping-timeout.
+- Transports: Socket.IO default (polling→websocket upgrade allowed). No websocket-only lock.
+
+**Client (Android, `WebSocketService.kt`):**
+- `reconnection = true`, `reconnectionAttempts = MAX_VALUE` (infinite), delay 1s→60s, jitter
+  0.5, connect `timeout = 20000`. **Configured to retry forever.**
+- App-level **`device:heartbeat` every 15s** (every 4th → playlist pull). **Fire-and-forget,
+  no ACK**, and guarded by `if (socket.connected() != true) return`.
+- Runs as a **foreground service (mediaPlayback) + PARTIAL_WAKE_LOCK** → not subject to Doze.
+- **No app-level liveness watchdog** — no last-server-message tracking, no forced reconnect;
+  it trusts `socket.connected()` and the socket.io-client-java Engine.IO ping-timeout.
+
+---
+
+## Phase 1 — the half-open / silent-death crux
+
+A firewall idle-reap (or SSL-inspection session drop) typically kills the TCP **silently — no
+FIN** — leaving both ends half-open.
+
+**Server:** holds the socket "connected" until its next ping gets no pong → closes on
+`pingTimeout` (~30–60s), then marks offline at 45s. With **no SO_KEEPALIVE**, 60s is the best
+it can do. Acceptable, but slow, and it never proactively probes TCP health.
+
+**Client — the gap:** it has **no independent liveness check**. On a half-open socket,
+`socket.connected()` still returns `true` (the lib hasn't noticed), so the 15s heartbeat keeps
+firing into the void and **`EVENT_DISCONNECT` may never fire** → the infinite-reconnect logic
+**never triggers** → the device sits on a dead socket = **"no retry."** Recovery depends
+entirely on the socket.io-client-java Engine.IO ping-timeout firing; if it's slow or missed on
+the MAXHUB's OkHttp/network stack, the client is stuck until the app/service restarts.
+
+This exactly matches #148: *server sees a clean/eventual disconnect, then nothing; the client
+loops or stops retrying.* The two sub-cases (both consistent, distinguished only by MAXHUB
+`logcat`):
+- **(a) half-open undetected** → client never reconnects ("no retry").
+- **(b) client does reconnect** but a rapid reap→reconnect cycle trips the server flap-limiter
+  → 30-min quarantine → reconnects refused → "loops connecting/waiting, then offline."
+
+**Fire TV reconciliation:** on the *same network + server + APK*, the Fire TV survives — so
+its network stack detects the half-open (or keeps the socket warm) and reconnects cleanly,
+riding through the same edge behavior the MAXHUB can't. The "1.9.2 staging, days" row adds a
+second reason (a single idle device with no fleet, no flapping, and possibly no aggressive
+firewall in staging) — but the **clean same-network comparison isolates the difference to the
+client's half-open handling.**
+
+---
+
+## Phase 2 — server-side contributor ruling
+
+| Candidate | Verdict |
+|---|---|
+| **Per-IP / total connection cap** (SNAT'd fleet = one IP) | **CLEARED.** No `maxConnections`, no per-IP handshake limit. The flap-limiter keys on **identity** (device_id→fingerprint→token→anon), never IP — SNAT-safe. |
+| **ScreenTinker's own proxy** (nginx/short WS read timeout) | **CLEARED.** The image runs `CMD ["node","server.js"]` — no proxy layer. Any WS timeout is on **Bold's edge**. |
+| **Anon-bucket collapse under SNAT** | **CLEARED for reconnects.** A reconnect carries the saved `device_id` → per-device bucket, not `anon:global`. |
+| **Flap-limiter / reconnect-throttle** | **IMPLICATED as an AMPLIFIER.** A reconnect on a fresh socket has `currentDeviceId=null` → `isRefreshConnect=false` → it **counts** toward the 20-connects/5-min limit. A repeated reap→reconnect cycle trips it → **30-min in-memory quarantine** → a recoverable blip becomes a 30-min offline. Because every device behind the edge gets reaped together, each trips its *own* limit at ~the same time → a **fleet-wide, synchronized-looking quarantine** (not via a shared bucket — via synchronized independent tripping). This is 1.9.2-only (beta5 has no flap-limiter). |
+
+---
+
+## Phase 3 — mitigation spec (propose; do NOT implement yet)
+
+### Fixes from OUR side (make it self-heal regardless of Bold's edge)
+1. **Client half-open watchdog (the primary fix).** Add an app-level liveness probe: send
+   `device:heartbeat` **with an ACK callback + a timeout** (e.g. ack expected within 10s); on a
+   missed ack, `socket.disconnect(); socket.connect()` (force a fresh reconnect). Equivalently,
+   track the last inbound server message and force-reconnect if none for >45s (> the 30s
+   pingInterval). This detects half-open death in ~15–25s regardless of the lib, and doubles as
+   the keepalive. Requires the server to ACK the heartbeat (or emit `device:heartbeat-ack`).
+   **This is an Android-APK change — it is NOT in v1.9.2.**
+2. **Server SO_KEEPALIVE.** `socket.setKeepAlive(true, ~30000)` on accepted connections so the
+   OS surfaces a dead peer independently of the 60s Engine.IO ping-timeout — faster, more
+   reliable server-side half-open detection.
+3. **Don't let the flap-limiter amplify a reap.** For an already-**paired device_id**, don't
+   escalate to the 30-min quarantine on reconnect churn (keep the soft reconnect-throttle for
+   loop protection, but a known device with a flaky link must keep being let back in). Scope
+   the exemption to paired device_ids so genuine anon/unidentified flappers are still limited —
+   **do not defeat the flap-limiter globally.**
+
+### Keepalive cadence (workaround — reduces the trigger, doesn't fix recovery)
+- The client already emits every 15s; the server pings every 30s. If Bold's timeout is a plain
+  idle-timer, this **already** defeats it. If reaping persists, it's DPI/SSL-inspection — a
+  keepalive can't fix that. Optionally tighten server `pingInterval` to ~25s as cheap insurance
+  (still far below the flap threshold, no load concern). **The heartbeat-with-ack in (1) is the
+  real keepalive+detector; a bare keepalive is not sufficient.**
+
+### Fix vs. workaround
+- **Fixes it (our side):** client heartbeat-with-ack + force-reconnect (1), server SO_KEEPALIVE
+  (2), flap-limiter paired-device exemption (3). With these, any silent drop self-heals in
+  ~15–60s and is never amplified into a 30-min lockout — independent of the firewall.
+- **Bold should still set (their side):** raise/disable the Sophos WebSocket idle/session
+  timeout for the ScreenTinker host; **disable SSL/DPI inspection** for that host (DPI is the
+  most likely reason a 15s-active socket still gets reaped); confirm no reverse proxy with a
+  short WS read timeout. These reduce how often the reap fires, but are not a substitute for
+  client self-heal.
+
+---
+
+## Questions for Dan to send Bold
+
+1. **Disconnect interval during quiet periods** (from #148): is it a round number
+   (60/120/300s)? A fixed round interval ⇒ idle/session-timeout confirmed. Irregular ⇒
+   half-open/other.
+2. **Sophos model + config:** the WebSocket/idle/session-timeout value on the ScreenTinker
+   host; is **SSL/deep-packet inspection** enabled for that host (this is the prime suspect for
+   reaping a socket that already has 15s traffic)?
+3. **Reverse proxy?** Any nginx / HAProxy / Cloudflare Tunnel / load balancer in front of the
+   ScreenTinker Docker container, and its WebSocket read/idle timeout?
+4. **MAXHUB `logcat` at the moment of a drop** (the decisive missing evidence): does it log
+   `EVENT_DISCONNECT` (client detected the drop → recovery is the throttle/quarantine story) or
+   **nothing** (half-open undetected → the client-watchdog fix is required)?
+5. Does the same MAXHUB survive when pointed at a server **with no firewall in the path** (e.g.
+   direct/LAN)? Confirms edge involvement vs. a pure client timer bug.

docs/148-half-open-fix.md

@@ -0,0 +1,142 @@
+# #148 — half-open death: mechanism, synchronization, and fix ownership
+
+**Status: investigation + fix spec. No code changed, nothing deployed.**
+
+## Headline (corrects the leading assumption)
+
+The premise "the SERVER sits on a half-open socket and never detects it" is **empirically
+false.** The Socket.IO server **actively pings and closes dead peers** — I confirmed it closes
+a non-ponging socket in `pingInterval + pingTimeout` (**1005ms** in a 500/500 test → **~60s in
+prod at 30/30**). And the server **hands those ping values to the client** in the Engine.IO
+handshake, so a compliant client runs the *same* ~60s dead-server detector.
+
+Therefore:
+- **The per-device fix is NOT "make the server detect half-open" — it already does.**
+- The MAXHUB "no retry" is a **client-side recovery failure** (its Engine.IO ping-timeout
+  isn't recovering the socket) that a server change **cannot fully fix over a severed TCP** —
+  the server's close frame can't reach a client whose TCP is dead.
+- The **simultaneity is Bold-edge or a reporting artifact**, not a 1.9.2 server mechanism.
+
+**A server-only 1.9.2.x patch can HELP (tighten detection via lower ping values, add
+SO_KEEPALIVE, stop the flap-limiter amplifying, make offline↔socket consistent) but is NOT
+proven sufficient alone.** Full resolution likely needs the client watchdog + Bold's edge fix.
+This is stated against the Phase-B gate below.
+
+---
+
+## Phase A — server-side detection (confirmed by test)
+
+| Behavior | Finding |
+|---|---|
+| Socket.IO `pingInterval`/`pingTimeout` | **30000 / 30000** (set, not default-off). Server sends a ping every 30s and **closes on missed pong** → dead-peer detection in **≤60s**. **Empirically verified** (`/tmp/halfopen.cjs`: 500/500 → closed at 1005ms). |
+| Client parameterization | The handshake open-packet carries `pingInterval`/`pingTimeout`; `socket.io-client:2.1.0` (EIO4) uses them for its **own** ping-timeout → the client also has a ~60s dead-server detector, **driven by the server's values**. |
+| SO_KEEPALIVE | **NOT set** anywhere. The OS never independently probes; Engine.IO ping is the only detector. |
+| `mark-offline` vs socket close | The heartbeat checker flips DB status to `offline` at **45s** and **does NOT close the socket**; Engine.IO closes it separately at **≤60s**. So the "offline-but-open" divergence is **bounded to ~15s**, not indefinite — but the two are not explicitly consistent. |
+| Server "sits forever" theory | **REFUTED.** The socket is gone within ~60s. |
+
+**Consequence:** because the client's detector is *parameterized by the server*, lowering the
+server's ping values **tightens the client's half-open detection without an APK update** — the
+one genuine server-only lever, and only useful **if the client's timer fires at all**.
+
+---
+
+## Phase B — the synchronization GATE (explicitly addressed)
+
+Half-open explains the *per-device* hang. For fleet-wide simultaneity, the candidates:
+
+1. **Server tick / event-loop freeze** — a pause > pingTimeout would make **every** client's
+   ping-timeout fire at once (true simultaneous mass disconnect). **RULED OUT on 1.9.2:** the
+   chunked prune removed the freeze; alpha runs clean for days; #148 logs show no freeze/
+   restart. (This *was* the beta5 mechanism — the 40–48s `ROW_NUMBER` prune freeze paused the
+   loop past pingTimeout → the whole fleet ping-timed-out together. It is the link between the
+   old "death spiral" and a genuine mass disconnect, and 1.9.2 fixed it.)
+2. **Bold's edge — conntrack / session-table flush** behind the single SNAT IP: a periodic
+   firewall event (NAT table timeout, policy reload, session flush) severs **all flows behind
+   that IP simultaneously** → every device goes half-open at the same instant. True
+   simultaneity, edge-owned.
+3. **Reporting artifact** — staggered client half-open deaths, then the **10s heartbeat checker
+   marks a batch offline on one tick** (plus the 5s deferral), so the CMS shows a synchronized
+   *offline wave* that does **not** correspond to a synchronized *sever*.
+
+**Verdict on the gate:** the per-device mechanism is real, and simultaneity is explainable —
+but I **cannot yet discriminate (2) a true edge-sever from (3) a reporting artifact** without
+Bold's data. What I *can* state firmly: **on 1.9.2 there is no server-side synchronizer** (the
+freeze is gone), so a true simultaneous *sever* would have to be edge (2). The disconnect-
+interval from #148 and whether the Fire TV *also* momentarily blips are the discriminators
+(questions below). **This is the honest "not fully proven" the gate asks for — the fix must not
+pretend the simultaneity is ours to fix.**
+
+**Fire TV reconciliation:** same network + server + APK, survives for days → its stack's
+client-side ping-timeout fires and it reconnects cleanly (self-heals within ~60s; a brief blip
+goes unnoticed). The MAXHUB on the identical path does not recover — the difference is
+**client-side recovery**, not the server or the edge (both hit the Fire TV too).
+
+---
+
+## Phase C — the reconnect amplifier (ours, under SNAT)
+
+- A **one-shot mass reconnect** from the single SNAT IP is **NOT refused**: each device carries
+  its own `device_id` → per-device flap bucket → one connect each, well under 20/5min. The
+  flap-limiter is identity-keyed (SNAT-safe), and reconnects carry `device_id` (not `anon`).
+  ✅ recovery from a single flush is not blocked.
+- **BUT** a *repeated* flush→reconnect cycle (or a reconnect that immediately re-drops) makes a
+  device accumulate connects → trips its own **20/5min → 30-min quarantine** → reconnects
+  refused → **sustained offline**. Because all devices trip ~together, this reads as a
+  synchronized fleet-wide lockout. **This is 1.9.2-only** (beta5 has no flap-limiter) and turns
+  a recoverable blip into a long outage. It must be fixed (Phase D-4).
+
+---
+
+## Does it exist in 1.9.2 as cut? / is the fix server-only?
+
+- The **server-detection gap does not exist** in 1.9.2 (it closes dead peers in ≤60s).
+- What **does** exist in 1.9.2 and is worth patching: (a) no SO_KEEPALIVE; (b) 60s detection is
+  slower than it needs to be, and the *client* window is server-parameterized so we can tighten
+  it without an APK; (c) `mark-offline` and socket-close aren't explicitly consistent; (d) the
+  **flap-limiter can amplify** a reconnect storm into a fleet-wide quarantine.
+- **Server-only reach:** items (a)–(d) are a legitimate **1.9.2.x server patch, no fleet APK
+  update** — and (b) tightens the *client's* detector for free. **But** if the MAXHUB's
+  client-side timer genuinely doesn't fire (true "no retry"), no server change wakes it over a
+  dead TCP → a **client watchdog is still required**, and the **edge synchronizer is Bold's**.
+  So: **ship the server patch (it strictly helps and de-risks the rollout), but do not tell
+  Bold it is guaranteed to fix #148 until the MAXHUB logcat shows the client recovers.**
+
+---
+
+## Fix spec (propose; do NOT implement until green-lit)
+
+**Server (1.9.2.x, no APK):**
+1. **Tighten ping cadence** — e.g. `PING_INTERVAL=20000`, `PING_TIMEOUT=20000` (→ ~40s
+   detection on both server AND client, since the client reads these). Do not go so low that
+   transient loop-lag false-disconnects healthy clients; gate/adjust with the existing loop-lag
+   band if lag is elevated. Env-driven, so tunable per deployment.
+2. **SO_KEEPALIVE** — `httpServer.on('connection', s => s.setKeepAlive(true, 20000))` (or on
+   engine.io's transport) as OS-level defense-in-depth.
+3. **Make mark-offline consistent with socket state** — when the heartbeat checker marks a
+   device offline, also `disconnect(true)` any lingering socket for it, so DB-offline can never
+   diverge into a silent half-open (and, on a one-directional break, the client is signalled).
+4. **Flap-limiter amplifier fix** — do NOT escalate a **paired `device_id`** to the 30-min
+   quarantine on reconnect churn (keep the soft reconnect-throttle for loop protection; a known
+   device with a flaky link must keep being let back in). Scope strictly to paired devices so
+   genuine anon/unidentified flappers are still limited — do **not** weaken the flap-limiter
+   globally or reintroduce the load risk it exists to prevent.
+
+**Client (Android APK — the real per-device fix, separate release):**
+5. **App-level liveness watchdog:** `device:heartbeat` with an **ACK + timeout**; on a missed
+   ack force `socket.disconnect(); socket.connect()`. Detects half-open in ~15–25s regardless
+   of the lib/OEM timer behavior. (Requires a server `device:heartbeat` ack.)
+
+**Bold (edge):** raise/disable the Sophos WS idle/session timeout for the host; **disable
+SSL/DPI inspection** for it; confirm no reverse proxy with a short WS read timeout.
+
+### Tests
+- **Server closes a dead peer within pingTimeout** — a client that completes the handshake then
+  stops ponging is disconnected within `pingInterval+pingTimeout` (already demonstrated:
+  `/tmp/halfopen.cjs`, 500/500 → 1005ms). Add as a regression at a short env timeout.
+- **offline ⇒ closed** — after the heartbeat checker marks a device offline, assert no open
+  socket remains for it (Phase D-3).
+- **SNAT mass reconnect not refused** — N distinct `device_id`s reconnecting once each from the
+  same IP are all admitted (flap-limiter identity-keyed); and a *repeated* cycle for a **paired**
+  device is not quarantined (Phase D-4).
+- **Client (instrumented/emulator):** after a silent transport kill, the watchdog forces a
+  reconnect within the ack-timeout window.

docs/148-mass-disconnect-analysis.md

@@ -0,0 +1,142 @@
+# #148 / #147 — "mass simultaneous disconnect" analysis (investigate-first)
+
+**Status: analysis only. No code changed, nothing deployed.**
+
+## Headline (the FIX-or-CARRY verdict Dan needs)
+
+**v1.9.2 does NOT fix #148/#147, and telling Bold to upgrade to it as "the fix" is not
+supported by the evidence — it may make the visible symptom worse.**
+
+- The disconnect is **client-initiated and MAXHUB-specific**, per #148's own server logs and
+  the clean same-server comparison (a Fire TV Stick on the *same server, same APK, same
+  network* stays connected; the MAXHUB drops). The server does **not** initiate it.
+- The bug reproduces on **every APK in the fleet (1.9.0 → 1.9.2)**, and **1.9.2's Android
+  player socket code is byte-identical to beta5** — the only android change between them is
+  the `versionCode` bump in `build.gradle.kts`. So the 1.9.2 APK will behave **identically**.
+- Worse: **1.9.2 ADDS a flap-limiter with a 30-minute in-memory quarantine that beta5 does
+  not have.** A MAXHUB that flaps (which is exactly what #148 describes) will be throttled
+  *and then quarantined* more aggressively on 1.9.2 than on beta5 — i.e. it may "go offline
+  and stop retrying" **faster**.
+- The one genuine server-side fix in 1.9.2 (the event-loop-freeze death spiral) is real but
+  addresses a **different** failure mode that is **not** present in #148 (there is no freeze,
+  no restart, no server-initiated close in the #148 logs).
+
+**Recommendation: walk back the "upgrade to 1.9.2 fixes this" advice.** The fix must target
+the MAXHUB Android client's WebSocket lifecycle, not the server.
+
+---
+
+## Phase 0 — the evidence (and the contradiction in the framing)
+
+There are two competing narratives, and the primary evidence resolves them:
+
+**#148 body + server logs (empirical):**
+- "Fire TV Stick on the **same APK, same network, same server** stays connected indefinitely
+  — this is **MAXHUB-specific**."
+- "Every disconnect is **client-initiated** — server sees a clean disconnect with the 5000ms
+  offline-transition deferral, then nothing."
+- "No errors, no blocks, no quarantine, **no throttle except after rapid reconnect cycling**
+  (throttle correctly identified the flapping)."
+- Trigger point: **after the initial content load completes** ("connect, register/pair,
+  receive content, then drop"), then loops connecting/waiting, then stops retrying.
+
+**Bold's later comment (the inference the task framing is built on):**
+- "mass simultaneous disconnect across separate physical locations proving the trigger is
+  server-side."
+
+**These contradict, and the primary evidence wins:** a server-side mass event would be
+*server-initiated* (the logs say client-initiated) and would drop the *Fire TV too* (it
+doesn't). So "server-side" is not supported.
+
+### Disconnect timing / periodicity
+There is **no fixed interval or wall-clock trigger** in #148. The trigger is a **per-device
+lifecycle event — completion of the initial content load.** The *appearance* of periodicity /
+simultaneity is a **reporting artifact** (see below), not a server clock.
+
+### Server-initiated vs missed-heartbeat — **missed-heartbeat, client-initiated**
+The server never closes these sockets. Per the logs it observes the **client** close (clean
+transport disconnect), starts the 5s offline-transition deferral (the #146 reconnect
+containment), then marks the device offline because no heartbeat/registration follows.
+
+---
+
+## Phase 1 — proof there is NO server-side mass-disconnect (1.9.2 / current main)
+
+Audited every path that can drop a device socket:
+
+| Mechanism | Can it close sockets? | Can it fire fleet-wide at once? |
+|---|---|---|
+| **Heartbeat checker** (`services/heartbeat.js`) | **No** — it `UPDATE devices SET status='offline'` in the DB (line 29) and deletes the in-memory conn; it **never** calls `disconnect()`. Line 45 is a *safety* `continue`-if-the-socket-is-still-live. | Marks *offline in the DB* in a batch each 10s tick, but does not disconnect anything. |
+| **Flap limiter / reconnect throttle** | Yes — refuses a *register* and `disconnect(true)`s **that one socket** | Only per-identity: a device is refused only if **it** exceeds 20 connects/5min. Not global. |
+| **Operator block** | Yes — one socket, only a `blocked=1` device | No |
+| **Evict-prior-socket** | Yes — the device's **own** previous socket on reconnect | No |
+| **`protectSocket`** (`safe-socket.js`) | Yes — the one socket whose handler threw | No |
+| **Maintenance / prune** | No — chunked, yields; touches tables, not sockets | No |
+| Global teardown (`io.disconnectSockets()`, namespace close, timer) | **Does not exist** | — |
+
+There is **no code path that disconnects many device sockets simultaneously.** The only
+"batch, periodic" behavior is the heartbeat checker **marking devices offline in the DB** —
+which is the server *reporting* that clients are gone, not *making* them gone.
+
+**Reconciling "mass simultaneous" without a server disconnect:** the MAXHUBs drop their own
+sockets (staggered, but correlated in time because they finish the initial content load
+around the same moment — e.g. after a content push or a shared power event). The heartbeat
+checker then flips a *batch* of them to `offline` on a single 10s tick (plus the 5s
+deferral), so the **CMS dashboard shows a synchronized offline wave** that never corresponds
+to a synchronized server *disconnect*. That reporting artifact is the "mass simultaneous"
+signal — and it behaves the same on beta5 and 1.9.2.
+
+---
+
+## Phase 2 — FIX or CARRY (1.9.2 vs 1.9.2-beta5)
+
+| Path | beta5 | 1.9.2 (main) | Effect on #148 |
+|---|---|---|---|
+| **Android player socket/reconnect code** | — | **byte-identical** (only `versionCode` bumped) | **Neither fixes nor changes it.** The client bug is untouched. |
+| **Heartbeat checker** | live-socket guard + marks offline, no close | same | No change to the offline-reporting artifact. |
+| **`pruneStatusLog`** | whole-table `ROW_NUMBER` sort (40–48s freeze) | `chunkedDelete` (non-blocking) | Fixes the **freeze/death-spiral** — a *different* failure mode, **not seen in #148** (no freeze/restart in the logs). |
+| **Flap limiter + 30-min quarantine** | **absent** | **present** (`lib/flap-limiter.js`, 20/5min → quarantine) | **CARRY / WORSEN:** a flapping MAXHUB is now throttled *and quarantined* — it will "stop retrying / go offline" **sooner** than on beta5. |
+
+**Verdict:** For the actual #148 mechanism (client-side drop), 1.9.2 is **inert** where it
+matters (identical client code, no server disconnect) and **counterproductive** on the
+secondary dynamic (the new flap-limiter quarantines the flapping client harder).
+
+### Reconciling the one success row (Fire TV, 1.9.2 staging, days)
+The clean isolation is in #148 itself: **same server, same APK, same network → Fire TV
+survives, MAXHUB drops.** The variable is the **device**, not the server. The task's other
+row (beta7 APK, beta5-prod vs 1.9.2-staging) confounds device *and* server and therefore
+can't isolate anything; the staging row also happens to be a **single idle device** (no
+fleet, tiny `device_status_log`, no flapping to throttle) — which removes every *secondary*
+aggravator too. Net: the survival tracks the **MAXHUB-vs-FireTV device difference**, with
+"staging = one idle device" masking the flap-limiter aggravation on top.
+
+**Most likely client root cause** (to be confirmed with MAXHUB `logcat`, the "not yet
+captured" item in #147): MAXHUB firmware/WebView power- or memory-management throttling or
+killing the app's socket/JS timers **after the heavy initial content load**, so the Engine.IO
+ping/pong lapses and the client closes — then its rapid reconnects trip the server throttle
+(and, on 1.9.2, the flap quarantine), ending in "offline, stops retrying." Fire TV's
+resource/power handling doesn't do this.
+
+### What would falsify this conclusion
+- MAXHUB `logcat` showing the **server** sent a close/`device:auth-error`/throttle *before*
+  the client closed (would move blame server-ward).
+- The Fire TV Stick *also* dropping under real fleet load on the same server (would reopen a
+  load/server-side theory).
+Capturing the MAXHUB `logcat` at the moment of disconnect is the single highest-value missing
+evidence.
+
+---
+
+## Proposed fix (do NOT implement in this task)
+
+1. **Client (the real fix):** in the MAXHUB/Android player, harden the socket lifecycle after
+   initial sync — a foreground service + partial wakelock to keep the socket alive, an
+   app-level heartbeat/keepalive independent of the WebView timer, and a reconnect policy that
+   backs off but **never permanently stops retrying**. Confirm against MAXHUB `logcat` first.
+2. **Server mitigation (reduces the *symptom*, not the cause):** since the flap limiter now
+   punishes a flapping MAXHUB harder, consider making it **more forgiving for
+   identified/paired devices** (raise `CONNECT_RATE_MAX`, or exempt a known device_id from
+   quarantine) so a client with a flaky socket keeps getting let back in instead of being
+   quarantined for 30 min. This makes 1.9.2 no worse than beta5 for this case while the client
+   fix is developed.
+3. **Do not** tell Bold that upgrading the server to 1.9.2 resolves #148/#147.