From d737b4f2b0b4ded00e6249122f060fd3a4393215 Mon Sep 17 00:00:00 2001 From: ScreenTinker Date: Thu, 2 Jul 2026 14:59:25 -0500 Subject: [PATCH] docs(#148): mass-disconnect + connection-lifecycle + half-open analyses --- docs/148-connection-lifecycle-analysis.md | 143 ++++++++++++++++++++++ docs/148-half-open-fix.md | 142 +++++++++++++++++++++ docs/148-mass-disconnect-analysis.md | 142 +++++++++++++++++++++ 3 files changed, 427 insertions(+) create mode 100644 docs/148-connection-lifecycle-analysis.md create mode 100644 docs/148-half-open-fix.md create mode 100644 docs/148-mass-disconnect-analysis.md diff --git a/docs/148-connection-lifecycle-analysis.md b/docs/148-connection-lifecycle-analysis.md new file mode 100644 index 0000000..111f0b6 --- /dev/null +++ b/docs/148-connection-lifecycle-analysis.md @@ -0,0 +1,143 @@ +# #148 — WebSocket connection-lifecycle analysis (idle-reap / half-open) + +**Status: investigation + mitigation spec. No code changed, nothing deployed.** + +## Headline + +Idle-reap by Bold's edge is a plausible **trigger**, but it is **not the whole story**, and a +keepalive alone will not fix it. Two facts constrain the diagnosis: + +1. **The client already sends traffic every 15s** (`device:heartbeat`) plus receives a + Socket.IO ping every 30s — so a *pure idle-timer* above ~15–30s should never reap this + socket. If Bold's firewall still reaps it, the cause is DPI/SSL-inspection or a sub-15s + timeout, **not** simple idleness — and more keepalive won't help. +2. **A Fire TV Stick survives on the SAME network + SAME server + SAME APK** (#148). Whatever + Bold's edge does, a well-behaved client rides through it. So the decisive difference is the + **client's ability to detect and recover from a silent (half-open) drop** — and the MAXHUB + build has a real gap there. + +**The fix is client self-heal + server half-open detection; the firewall config is a +secondary trigger-reducer.** None of this is fixed by v1.9.2 (the APK socket code is +unchanged), so this needs a new player build. + +--- + +## Phase 0 — the keepalive stack (exact values, current main / 1.9.2) + +**Server (`server/config.js`, `server/server.js`):** +- Socket.IO `pingInterval = 30000`, `pingTimeout = 30000` → **server→client ping every 30s**; + worst-case dead-socket detection = **60s**. (Engine.IO v4: the *server* pings, the client + pongs; these are real WebSocket frames, so a firewall that counts frames sees traffic.) +- `heartbeatTimeout = 45000` → server marks a device **offline** after 45s with no + `device:heartbeat`. Marking offline **only flips DB status; it does NOT close the socket.** +- **No `setKeepAlive`/SO_KEEPALIVE** anywhere → the OS never independently detects a dead + peer; the server relies solely on the 60s Engine.IO ping-timeout. +- Transports: Socket.IO default (polling→websocket upgrade allowed). No websocket-only lock. + +**Client (Android, `WebSocketService.kt`):** +- `reconnection = true`, `reconnectionAttempts = MAX_VALUE` (infinite), delay 1s→60s, jitter + 0.5, connect `timeout = 20000`. **Configured to retry forever.** +- App-level **`device:heartbeat` every 15s** (every 4th → playlist pull). **Fire-and-forget, + no ACK**, and guarded by `if (socket.connected() != true) return`. +- Runs as a **foreground service (mediaPlayback) + PARTIAL_WAKE_LOCK** → not subject to Doze. +- **No app-level liveness watchdog** — no last-server-message tracking, no forced reconnect; + it trusts `socket.connected()` and the socket.io-client-java Engine.IO ping-timeout. + +--- + +## Phase 1 — the half-open / silent-death crux + +A firewall idle-reap (or SSL-inspection session drop) typically kills the TCP **silently — no +FIN** — leaving both ends half-open. + +**Server:** holds the socket "connected" until its next ping gets no pong → closes on +`pingTimeout` (~30–60s), then marks offline at 45s. With **no SO_KEEPALIVE**, 60s is the best +it can do. Acceptable, but slow, and it never proactively probes TCP health. + +**Client — the gap:** it has **no independent liveness check**. On a half-open socket, +`socket.connected()` still returns `true` (the lib hasn't noticed), so the 15s heartbeat keeps +firing into the void and **`EVENT_DISCONNECT` may never fire** → the infinite-reconnect logic +**never triggers** → the device sits on a dead socket = **"no retry."** Recovery depends +entirely on the socket.io-client-java Engine.IO ping-timeout firing; if it's slow or missed on +the MAXHUB's OkHttp/network stack, the client is stuck until the app/service restarts. + +This exactly matches #148: *server sees a clean/eventual disconnect, then nothing; the client +loops or stops retrying.* The two sub-cases (both consistent, distinguished only by MAXHUB +`logcat`): +- **(a) half-open undetected** → client never reconnects ("no retry"). +- **(b) client does reconnect** but a rapid reap→reconnect cycle trips the server flap-limiter + → 30-min quarantine → reconnects refused → "loops connecting/waiting, then offline." + +**Fire TV reconciliation:** on the *same network + server + APK*, the Fire TV survives — so +its network stack detects the half-open (or keeps the socket warm) and reconnects cleanly, +riding through the same edge behavior the MAXHUB can't. The "1.9.2 staging, days" row adds a +second reason (a single idle device with no fleet, no flapping, and possibly no aggressive +firewall in staging) — but the **clean same-network comparison isolates the difference to the +client's half-open handling.** + +--- + +## Phase 2 — server-side contributor ruling + +| Candidate | Verdict | +|---|---| +| **Per-IP / total connection cap** (SNAT'd fleet = one IP) | **CLEARED.** No `maxConnections`, no per-IP handshake limit. The flap-limiter keys on **identity** (device_id→fingerprint→token→anon), never IP — SNAT-safe. | +| **ScreenTinker's own proxy** (nginx/short WS read timeout) | **CLEARED.** The image runs `CMD ["node","server.js"]` — no proxy layer. Any WS timeout is on **Bold's edge**. | +| **Anon-bucket collapse under SNAT** | **CLEARED for reconnects.** A reconnect carries the saved `device_id` → per-device bucket, not `anon:global`. | +| **Flap-limiter / reconnect-throttle** | **IMPLICATED as an AMPLIFIER.** A reconnect on a fresh socket has `currentDeviceId=null` → `isRefreshConnect=false` → it **counts** toward the 20-connects/5-min limit. A repeated reap→reconnect cycle trips it → **30-min in-memory quarantine** → a recoverable blip becomes a 30-min offline. Because every device behind the edge gets reaped together, each trips its *own* limit at ~the same time → a **fleet-wide, synchronized-looking quarantine** (not via a shared bucket — via synchronized independent tripping). This is 1.9.2-only (beta5 has no flap-limiter). | + +--- + +## Phase 3 — mitigation spec (propose; do NOT implement yet) + +### Fixes from OUR side (make it self-heal regardless of Bold's edge) +1. **Client half-open watchdog (the primary fix).** Add an app-level liveness probe: send + `device:heartbeat` **with an ACK callback + a timeout** (e.g. ack expected within 10s); on a + missed ack, `socket.disconnect(); socket.connect()` (force a fresh reconnect). Equivalently, + track the last inbound server message and force-reconnect if none for >45s (> the 30s + pingInterval). This detects half-open death in ~15–25s regardless of the lib, and doubles as + the keepalive. Requires the server to ACK the heartbeat (or emit `device:heartbeat-ack`). + **This is an Android-APK change — it is NOT in v1.9.2.** +2. **Server SO_KEEPALIVE.** `socket.setKeepAlive(true, ~30000)` on accepted connections so the + OS surfaces a dead peer independently of the 60s Engine.IO ping-timeout — faster, more + reliable server-side half-open detection. +3. **Don't let the flap-limiter amplify a reap.** For an already-**paired device_id**, don't + escalate to the 30-min quarantine on reconnect churn (keep the soft reconnect-throttle for + loop protection, but a known device with a flaky link must keep being let back in). Scope + the exemption to paired device_ids so genuine anon/unidentified flappers are still limited — + **do not defeat the flap-limiter globally.** + +### Keepalive cadence (workaround — reduces the trigger, doesn't fix recovery) +- The client already emits every 15s; the server pings every 30s. If Bold's timeout is a plain + idle-timer, this **already** defeats it. If reaping persists, it's DPI/SSL-inspection — a + keepalive can't fix that. Optionally tighten server `pingInterval` to ~25s as cheap insurance + (still far below the flap threshold, no load concern). **The heartbeat-with-ack in (1) is the + real keepalive+detector; a bare keepalive is not sufficient.** + +### Fix vs. workaround +- **Fixes it (our side):** client heartbeat-with-ack + force-reconnect (1), server SO_KEEPALIVE + (2), flap-limiter paired-device exemption (3). With these, any silent drop self-heals in + ~15–60s and is never amplified into a 30-min lockout — independent of the firewall. +- **Bold should still set (their side):** raise/disable the Sophos WebSocket idle/session + timeout for the ScreenTinker host; **disable SSL/DPI inspection** for that host (DPI is the + most likely reason a 15s-active socket still gets reaped); confirm no reverse proxy with a + short WS read timeout. These reduce how often the reap fires, but are not a substitute for + client self-heal. + +--- + +## Questions for Dan to send Bold + +1. **Disconnect interval during quiet periods** (from #148): is it a round number + (60/120/300s)? A fixed round interval ⇒ idle/session-timeout confirmed. Irregular ⇒ + half-open/other. +2. **Sophos model + config:** the WebSocket/idle/session-timeout value on the ScreenTinker + host; is **SSL/deep-packet inspection** enabled for that host (this is the prime suspect for + reaping a socket that already has 15s traffic)? +3. **Reverse proxy?** Any nginx / HAProxy / Cloudflare Tunnel / load balancer in front of the + ScreenTinker Docker container, and its WebSocket read/idle timeout? +4. **MAXHUB `logcat` at the moment of a drop** (the decisive missing evidence): does it log + `EVENT_DISCONNECT` (client detected the drop → recovery is the throttle/quarantine story) or + **nothing** (half-open undetected → the client-watchdog fix is required)? +5. Does the same MAXHUB survive when pointed at a server **with no firewall in the path** (e.g. + direct/LAN)? Confirms edge involvement vs. a pure client timer bug. diff --git a/docs/148-half-open-fix.md b/docs/148-half-open-fix.md new file mode 100644 index 0000000..eef4cfe --- /dev/null +++ b/docs/148-half-open-fix.md @@ -0,0 +1,142 @@ +# #148 — half-open death: mechanism, synchronization, and fix ownership + +**Status: investigation + fix spec. No code changed, nothing deployed.** + +## Headline (corrects the leading assumption) + +The premise "the SERVER sits on a half-open socket and never detects it" is **empirically +false.** The Socket.IO server **actively pings and closes dead peers** — I confirmed it closes +a non-ponging socket in `pingInterval + pingTimeout` (**1005ms** in a 500/500 test → **~60s in +prod at 30/30**). And the server **hands those ping values to the client** in the Engine.IO +handshake, so a compliant client runs the *same* ~60s dead-server detector. + +Therefore: +- **The per-device fix is NOT "make the server detect half-open" — it already does.** +- The MAXHUB "no retry" is a **client-side recovery failure** (its Engine.IO ping-timeout + isn't recovering the socket) that a server change **cannot fully fix over a severed TCP** — + the server's close frame can't reach a client whose TCP is dead. +- The **simultaneity is Bold-edge or a reporting artifact**, not a 1.9.2 server mechanism. + +**A server-only 1.9.2.x patch can HELP (tighten detection via lower ping values, add +SO_KEEPALIVE, stop the flap-limiter amplifying, make offline↔socket consistent) but is NOT +proven sufficient alone.** Full resolution likely needs the client watchdog + Bold's edge fix. +This is stated against the Phase-B gate below. + +--- + +## Phase A — server-side detection (confirmed by test) + +| Behavior | Finding | +|---|---| +| Socket.IO `pingInterval`/`pingTimeout` | **30000 / 30000** (set, not default-off). Server sends a ping every 30s and **closes on missed pong** → dead-peer detection in **≤60s**. **Empirically verified** (`/tmp/halfopen.cjs`: 500/500 → closed at 1005ms). | +| Client parameterization | The handshake open-packet carries `pingInterval`/`pingTimeout`; `socket.io-client:2.1.0` (EIO4) uses them for its **own** ping-timeout → the client also has a ~60s dead-server detector, **driven by the server's values**. | +| SO_KEEPALIVE | **NOT set** anywhere. The OS never independently probes; Engine.IO ping is the only detector. | +| `mark-offline` vs socket close | The heartbeat checker flips DB status to `offline` at **45s** and **does NOT close the socket**; Engine.IO closes it separately at **≤60s**. So the "offline-but-open" divergence is **bounded to ~15s**, not indefinite — but the two are not explicitly consistent. | +| Server "sits forever" theory | **REFUTED.** The socket is gone within ~60s. | + +**Consequence:** because the client's detector is *parameterized by the server*, lowering the +server's ping values **tightens the client's half-open detection without an APK update** — the +one genuine server-only lever, and only useful **if the client's timer fires at all**. + +--- + +## Phase B — the synchronization GATE (explicitly addressed) + +Half-open explains the *per-device* hang. For fleet-wide simultaneity, the candidates: + +1. **Server tick / event-loop freeze** — a pause > pingTimeout would make **every** client's + ping-timeout fire at once (true simultaneous mass disconnect). **RULED OUT on 1.9.2:** the + chunked prune removed the freeze; alpha runs clean for days; #148 logs show no freeze/ + restart. (This *was* the beta5 mechanism — the 40–48s `ROW_NUMBER` prune freeze paused the + loop past pingTimeout → the whole fleet ping-timed-out together. It is the link between the + old "death spiral" and a genuine mass disconnect, and 1.9.2 fixed it.) +2. **Bold's edge — conntrack / session-table flush** behind the single SNAT IP: a periodic + firewall event (NAT table timeout, policy reload, session flush) severs **all flows behind + that IP simultaneously** → every device goes half-open at the same instant. True + simultaneity, edge-owned. +3. **Reporting artifact** — staggered client half-open deaths, then the **10s heartbeat checker + marks a batch offline on one tick** (plus the 5s deferral), so the CMS shows a synchronized + *offline wave* that does **not** correspond to a synchronized *sever*. + +**Verdict on the gate:** the per-device mechanism is real, and simultaneity is explainable — +but I **cannot yet discriminate (2) a true edge-sever from (3) a reporting artifact** without +Bold's data. What I *can* state firmly: **on 1.9.2 there is no server-side synchronizer** (the +freeze is gone), so a true simultaneous *sever* would have to be edge (2). The disconnect- +interval from #148 and whether the Fire TV *also* momentarily blips are the discriminators +(questions below). **This is the honest "not fully proven" the gate asks for — the fix must not +pretend the simultaneity is ours to fix.** + +**Fire TV reconciliation:** same network + server + APK, survives for days → its stack's +client-side ping-timeout fires and it reconnects cleanly (self-heals within ~60s; a brief blip +goes unnoticed). The MAXHUB on the identical path does not recover — the difference is +**client-side recovery**, not the server or the edge (both hit the Fire TV too). + +--- + +## Phase C — the reconnect amplifier (ours, under SNAT) + +- A **one-shot mass reconnect** from the single SNAT IP is **NOT refused**: each device carries + its own `device_id` → per-device flap bucket → one connect each, well under 20/5min. The + flap-limiter is identity-keyed (SNAT-safe), and reconnects carry `device_id` (not `anon`). + ✅ recovery from a single flush is not blocked. +- **BUT** a *repeated* flush→reconnect cycle (or a reconnect that immediately re-drops) makes a + device accumulate connects → trips its own **20/5min → 30-min quarantine** → reconnects + refused → **sustained offline**. Because all devices trip ~together, this reads as a + synchronized fleet-wide lockout. **This is 1.9.2-only** (beta5 has no flap-limiter) and turns + a recoverable blip into a long outage. It must be fixed (Phase D-4). + +--- + +## Does it exist in 1.9.2 as cut? / is the fix server-only? + +- The **server-detection gap does not exist** in 1.9.2 (it closes dead peers in ≤60s). +- What **does** exist in 1.9.2 and is worth patching: (a) no SO_KEEPALIVE; (b) 60s detection is + slower than it needs to be, and the *client* window is server-parameterized so we can tighten + it without an APK; (c) `mark-offline` and socket-close aren't explicitly consistent; (d) the + **flap-limiter can amplify** a reconnect storm into a fleet-wide quarantine. +- **Server-only reach:** items (a)–(d) are a legitimate **1.9.2.x server patch, no fleet APK + update** — and (b) tightens the *client's* detector for free. **But** if the MAXHUB's + client-side timer genuinely doesn't fire (true "no retry"), no server change wakes it over a + dead TCP → a **client watchdog is still required**, and the **edge synchronizer is Bold's**. + So: **ship the server patch (it strictly helps and de-risks the rollout), but do not tell + Bold it is guaranteed to fix #148 until the MAXHUB logcat shows the client recovers.** + +--- + +## Fix spec (propose; do NOT implement until green-lit) + +**Server (1.9.2.x, no APK):** +1. **Tighten ping cadence** — e.g. `PING_INTERVAL=20000`, `PING_TIMEOUT=20000` (→ ~40s + detection on both server AND client, since the client reads these). Do not go so low that + transient loop-lag false-disconnects healthy clients; gate/adjust with the existing loop-lag + band if lag is elevated. Env-driven, so tunable per deployment. +2. **SO_KEEPALIVE** — `httpServer.on('connection', s => s.setKeepAlive(true, 20000))` (or on + engine.io's transport) as OS-level defense-in-depth. +3. **Make mark-offline consistent with socket state** — when the heartbeat checker marks a + device offline, also `disconnect(true)` any lingering socket for it, so DB-offline can never + diverge into a silent half-open (and, on a one-directional break, the client is signalled). +4. **Flap-limiter amplifier fix** — do NOT escalate a **paired `device_id`** to the 30-min + quarantine on reconnect churn (keep the soft reconnect-throttle for loop protection; a known + device with a flaky link must keep being let back in). Scope strictly to paired devices so + genuine anon/unidentified flappers are still limited — do **not** weaken the flap-limiter + globally or reintroduce the load risk it exists to prevent. + +**Client (Android APK — the real per-device fix, separate release):** +5. **App-level liveness watchdog:** `device:heartbeat` with an **ACK + timeout**; on a missed + ack force `socket.disconnect(); socket.connect()`. Detects half-open in ~15–25s regardless + of the lib/OEM timer behavior. (Requires a server `device:heartbeat` ack.) + +**Bold (edge):** raise/disable the Sophos WS idle/session timeout for the host; **disable +SSL/DPI inspection** for it; confirm no reverse proxy with a short WS read timeout. + +### Tests +- **Server closes a dead peer within pingTimeout** — a client that completes the handshake then + stops ponging is disconnected within `pingInterval+pingTimeout` (already demonstrated: + `/tmp/halfopen.cjs`, 500/500 → 1005ms). Add as a regression at a short env timeout. +- **offline ⇒ closed** — after the heartbeat checker marks a device offline, assert no open + socket remains for it (Phase D-3). +- **SNAT mass reconnect not refused** — N distinct `device_id`s reconnecting once each from the + same IP are all admitted (flap-limiter identity-keyed); and a *repeated* cycle for a **paired** + device is not quarantined (Phase D-4). +- **Client (instrumented/emulator):** after a silent transport kill, the watchdog forces a + reconnect within the ack-timeout window. diff --git a/docs/148-mass-disconnect-analysis.md b/docs/148-mass-disconnect-analysis.md new file mode 100644 index 0000000..3e3382d --- /dev/null +++ b/docs/148-mass-disconnect-analysis.md @@ -0,0 +1,142 @@ +# #148 / #147 — "mass simultaneous disconnect" analysis (investigate-first) + +**Status: analysis only. No code changed, nothing deployed.** + +## Headline (the FIX-or-CARRY verdict Dan needs) + +**v1.9.2 does NOT fix #148/#147, and telling Bold to upgrade to it as "the fix" is not +supported by the evidence — it may make the visible symptom worse.** + +- The disconnect is **client-initiated and MAXHUB-specific**, per #148's own server logs and + the clean same-server comparison (a Fire TV Stick on the *same server, same APK, same + network* stays connected; the MAXHUB drops). The server does **not** initiate it. +- The bug reproduces on **every APK in the fleet (1.9.0 → 1.9.2)**, and **1.9.2's Android + player socket code is byte-identical to beta5** — the only android change between them is + the `versionCode` bump in `build.gradle.kts`. So the 1.9.2 APK will behave **identically**. +- Worse: **1.9.2 ADDS a flap-limiter with a 30-minute in-memory quarantine that beta5 does + not have.** A MAXHUB that flaps (which is exactly what #148 describes) will be throttled + *and then quarantined* more aggressively on 1.9.2 than on beta5 — i.e. it may "go offline + and stop retrying" **faster**. +- The one genuine server-side fix in 1.9.2 (the event-loop-freeze death spiral) is real but + addresses a **different** failure mode that is **not** present in #148 (there is no freeze, + no restart, no server-initiated close in the #148 logs). + +**Recommendation: walk back the "upgrade to 1.9.2 fixes this" advice.** The fix must target +the MAXHUB Android client's WebSocket lifecycle, not the server. + +--- + +## Phase 0 — the evidence (and the contradiction in the framing) + +There are two competing narratives, and the primary evidence resolves them: + +**#148 body + server logs (empirical):** +- "Fire TV Stick on the **same APK, same network, same server** stays connected indefinitely + — this is **MAXHUB-specific**." +- "Every disconnect is **client-initiated** — server sees a clean disconnect with the 5000ms + offline-transition deferral, then nothing." +- "No errors, no blocks, no quarantine, **no throttle except after rapid reconnect cycling** + (throttle correctly identified the flapping)." +- Trigger point: **after the initial content load completes** ("connect, register/pair, + receive content, then drop"), then loops connecting/waiting, then stops retrying. + +**Bold's later comment (the inference the task framing is built on):** +- "mass simultaneous disconnect across separate physical locations proving the trigger is + server-side." + +**These contradict, and the primary evidence wins:** a server-side mass event would be +*server-initiated* (the logs say client-initiated) and would drop the *Fire TV too* (it +doesn't). So "server-side" is not supported. + +### Disconnect timing / periodicity +There is **no fixed interval or wall-clock trigger** in #148. The trigger is a **per-device +lifecycle event — completion of the initial content load.** The *appearance* of periodicity / +simultaneity is a **reporting artifact** (see below), not a server clock. + +### Server-initiated vs missed-heartbeat — **missed-heartbeat, client-initiated** +The server never closes these sockets. Per the logs it observes the **client** close (clean +transport disconnect), starts the 5s offline-transition deferral (the #146 reconnect +containment), then marks the device offline because no heartbeat/registration follows. + +--- + +## Phase 1 — proof there is NO server-side mass-disconnect (1.9.2 / current main) + +Audited every path that can drop a device socket: + +| Mechanism | Can it close sockets? | Can it fire fleet-wide at once? | +|---|---|---| +| **Heartbeat checker** (`services/heartbeat.js`) | **No** — it `UPDATE devices SET status='offline'` in the DB (line 29) and deletes the in-memory conn; it **never** calls `disconnect()`. Line 45 is a *safety* `continue`-if-the-socket-is-still-live. | Marks *offline in the DB* in a batch each 10s tick, but does not disconnect anything. | +| **Flap limiter / reconnect throttle** | Yes — refuses a *register* and `disconnect(true)`s **that one socket** | Only per-identity: a device is refused only if **it** exceeds 20 connects/5min. Not global. | +| **Operator block** | Yes — one socket, only a `blocked=1` device | No | +| **Evict-prior-socket** | Yes — the device's **own** previous socket on reconnect | No | +| **`protectSocket`** (`safe-socket.js`) | Yes — the one socket whose handler threw | No | +| **Maintenance / prune** | No — chunked, yields; touches tables, not sockets | No | +| Global teardown (`io.disconnectSockets()`, namespace close, timer) | **Does not exist** | — | + +There is **no code path that disconnects many device sockets simultaneously.** The only +"batch, periodic" behavior is the heartbeat checker **marking devices offline in the DB** — +which is the server *reporting* that clients are gone, not *making* them gone. + +**Reconciling "mass simultaneous" without a server disconnect:** the MAXHUBs drop their own +sockets (staggered, but correlated in time because they finish the initial content load +around the same moment — e.g. after a content push or a shared power event). The heartbeat +checker then flips a *batch* of them to `offline` on a single 10s tick (plus the 5s +deferral), so the **CMS dashboard shows a synchronized offline wave** that never corresponds +to a synchronized server *disconnect*. That reporting artifact is the "mass simultaneous" +signal — and it behaves the same on beta5 and 1.9.2. + +--- + +## Phase 2 — FIX or CARRY (1.9.2 vs 1.9.2-beta5) + +| Path | beta5 | 1.9.2 (main) | Effect on #148 | +|---|---|---|---| +| **Android player socket/reconnect code** | — | **byte-identical** (only `versionCode` bumped) | **Neither fixes nor changes it.** The client bug is untouched. | +| **Heartbeat checker** | live-socket guard + marks offline, no close | same | No change to the offline-reporting artifact. | +| **`pruneStatusLog`** | whole-table `ROW_NUMBER` sort (40–48s freeze) | `chunkedDelete` (non-blocking) | Fixes the **freeze/death-spiral** — a *different* failure mode, **not seen in #148** (no freeze/restart in the logs). | +| **Flap limiter + 30-min quarantine** | **absent** | **present** (`lib/flap-limiter.js`, 20/5min → quarantine) | **CARRY / WORSEN:** a flapping MAXHUB is now throttled *and quarantined* — it will "stop retrying / go offline" **sooner** than on beta5. | + +**Verdict:** For the actual #148 mechanism (client-side drop), 1.9.2 is **inert** where it +matters (identical client code, no server disconnect) and **counterproductive** on the +secondary dynamic (the new flap-limiter quarantines the flapping client harder). + +### Reconciling the one success row (Fire TV, 1.9.2 staging, days) +The clean isolation is in #148 itself: **same server, same APK, same network → Fire TV +survives, MAXHUB drops.** The variable is the **device**, not the server. The task's other +row (beta7 APK, beta5-prod vs 1.9.2-staging) confounds device *and* server and therefore +can't isolate anything; the staging row also happens to be a **single idle device** (no +fleet, tiny `device_status_log`, no flapping to throttle) — which removes every *secondary* +aggravator too. Net: the survival tracks the **MAXHUB-vs-FireTV device difference**, with +"staging = one idle device" masking the flap-limiter aggravation on top. + +**Most likely client root cause** (to be confirmed with MAXHUB `logcat`, the "not yet +captured" item in #147): MAXHUB firmware/WebView power- or memory-management throttling or +killing the app's socket/JS timers **after the heavy initial content load**, so the Engine.IO +ping/pong lapses and the client closes — then its rapid reconnects trip the server throttle +(and, on 1.9.2, the flap quarantine), ending in "offline, stops retrying." Fire TV's +resource/power handling doesn't do this. + +### What would falsify this conclusion +- MAXHUB `logcat` showing the **server** sent a close/`device:auth-error`/throttle *before* + the client closed (would move blame server-ward). +- The Fire TV Stick *also* dropping under real fleet load on the same server (would reopen a + load/server-side theory). +Capturing the MAXHUB `logcat` at the moment of disconnect is the single highest-value missing +evidence. + +--- + +## Proposed fix (do NOT implement in this task) + +1. **Client (the real fix):** in the MAXHUB/Android player, harden the socket lifecycle after + initial sync — a foreground service + partial wakelock to keep the socket alive, an + app-level heartbeat/keepalive independent of the WebView timer, and a reconnect policy that + backs off but **never permanently stops retrying**. Confirm against MAXHUB `logcat` first. +2. **Server mitigation (reduces the *symptom*, not the cause):** since the flap limiter now + punishes a flapping MAXHUB harder, consider making it **more forgiving for + identified/paired devices** (raise `CONNECT_RATE_MAX`, or exempt a known device_id from + quarantine) so a client with a flaky socket keeps getting let back in instead of being + quarantined for 30 min. This makes 1.9.2 no worse than beta5 for this case while the client + fix is developed. +3. **Do not** tell Bold that upgrading the server to 1.9.2 resolves #148/#147.