#148 — half-open death: mechanism, synchronization, and fix ownership

Status: investigation + fix spec. No code changed, nothing deployed.

Headline (corrects the leading assumption)

The premise "the SERVER sits on a half-open socket and never detects it" is empirically false. The Socket.IO server actively pings and closes dead peers — I confirmed it closes a non-ponging socket in pingInterval + pingTimeout (1005ms in a 500/500 test → ~60s in prod at 30/30). And the server hands those ping values to the client in the Engine.IO handshake, so a compliant client runs the same ~60s dead-server detector.

Therefore:

The per-device fix is NOT "make the server detect half-open" — it already does.
The MAXHUB "no retry" is a client-side recovery failure (its Engine.IO ping-timeout isn't recovering the socket) that a server change cannot fully fix over a severed TCP — the server's close frame can't reach a client whose TCP is dead.
The simultaneity is Bold-edge or a reporting artifact, not a 1.9.2 server mechanism.

A server-only 1.9.2.x patch can HELP (tighten detection via lower ping values, add SO_KEEPALIVE, stop the flap-limiter amplifying, make offline↔socket consistent) but is NOT proven sufficient alone. Full resolution likely needs the client watchdog + Bold's edge fix. This is stated against the Phase-B gate below.

Phase A — server-side detection (confirmed by test)

Behavior	Finding
Socket.IO `pingInterval`/`pingTimeout`	30000 / 30000 (set, not default-off). Server sends a ping every 30s and closes on missed pong → dead-peer detection in ≤60s. Empirically verified (`/tmp/halfopen.cjs`: 500/500 → closed at 1005ms).
Client parameterization	The handshake open-packet carries `pingInterval`/`pingTimeout`; `socket.io-client:2.1.0` (EIO4) uses them for its own ping-timeout → the client also has a ~60s dead-server detector, driven by the server's values.
SO_KEEPALIVE	NOT set anywhere. The OS never independently probes; Engine.IO ping is the only detector.
`mark-offline` vs socket close	The heartbeat checker flips DB status to `offline` at 45s and does NOT close the socket; Engine.IO closes it separately at ≤60s. So the "offline-but-open" divergence is bounded to ~15s, not indefinite — but the two are not explicitly consistent.
Server "sits forever" theory	REFUTED. The socket is gone within ~60s.

Consequence: because the client's detector is parameterized by the server, lowering the server's ping values tightens the client's half-open detection without an APK update — the one genuine server-only lever, and only useful if the client's timer fires at all.

Phase B — the synchronization GATE (explicitly addressed)

Half-open explains the per-device hang. For fleet-wide simultaneity, the candidates:

Server tick / event-loop freeze — a pause > pingTimeout would make every client's ping-timeout fire at once (true simultaneous mass disconnect). RULED OUT on 1.9.2: the chunked prune removed the freeze; alpha runs clean for days; #148 logs show no freeze/ restart. (This was the beta5 mechanism — the 40–48s ROW_NUMBER prune freeze paused the loop past pingTimeout → the whole fleet ping-timed-out together. It is the link between the old "death spiral" and a genuine mass disconnect, and 1.9.2 fixed it.)
Bold's edge — conntrack / session-table flush behind the single SNAT IP: a periodic firewall event (NAT table timeout, policy reload, session flush) severs all flows behind that IP simultaneously → every device goes half-open at the same instant. True simultaneity, edge-owned.
Reporting artifact — staggered client half-open deaths, then the 10s heartbeat checker marks a batch offline on one tick (plus the 5s deferral), so the CMS shows a synchronized offline wave that does not correspond to a synchronized sever.

Verdict on the gate: the per-device mechanism is real, and simultaneity is explainable — but I cannot yet discriminate (2) a true edge-sever from (3) a reporting artifact without Bold's data. What I can state firmly: on 1.9.2 there is no server-side synchronizer (the freeze is gone), so a true simultaneous sever would have to be edge (2). The disconnect- interval from #148 and whether the Fire TV also momentarily blips are the discriminators (questions below). This is the honest "not fully proven" the gate asks for — the fix must not pretend the simultaneity is ours to fix.

Fire TV reconciliation: same network + server + APK, survives for days → its stack's client-side ping-timeout fires and it reconnects cleanly (self-heals within ~60s; a brief blip goes unnoticed). The MAXHUB on the identical path does not recover — the difference is client-side recovery, not the server or the edge (both hit the Fire TV too).

Phase C — the reconnect amplifier (ours, under SNAT)

A one-shot mass reconnect from the single SNAT IP is NOT refused: each device carries its own device_id → per-device flap bucket → one connect each, well under 20/5min. The flap-limiter is identity-keyed (SNAT-safe), and reconnects carry device_id (not anon). ✅ recovery from a single flush is not blocked.
BUT a repeated flush→reconnect cycle (or a reconnect that immediately re-drops) makes a device accumulate connects → trips its own 20/5min → 30-min quarantine → reconnects refused → sustained offline. Because all devices trip ~together, this reads as a synchronized fleet-wide lockout. This is 1.9.2-only (beta5 has no flap-limiter) and turns a recoverable blip into a long outage. It must be fixed (Phase D-4).

Does it exist in 1.9.2 as cut? / is the fix server-only?

The server-detection gap does not exist in 1.9.2 (it closes dead peers in ≤60s).
What does exist in 1.9.2 and is worth patching: (a) no SO_KEEPALIVE; (b) 60s detection is slower than it needs to be, and the client window is server-parameterized so we can tighten it without an APK; (c) mark-offline and socket-close aren't explicitly consistent; (d) the flap-limiter can amplify a reconnect storm into a fleet-wide quarantine.
Server-only reach: items (a)–(d) are a legitimate 1.9.2.x server patch, no fleet APK update — and (b) tightens the client's detector for free. But if the MAXHUB's client-side timer genuinely doesn't fire (true "no retry"), no server change wakes it over a dead TCP → a client watchdog is still required, and the edge synchronizer is Bold's. So: ship the server patch (it strictly helps and de-risks the rollout), but do not tell Bold it is guaranteed to fix #148 until the MAXHUB logcat shows the client recovers.

Fix spec (propose; do NOT implement until green-lit)

Server (1.9.2.x, no APK):

Tighten ping cadence — e.g. PING_INTERVAL=20000, PING_TIMEOUT=20000 (→ ~40s detection on both server AND client, since the client reads these). Do not go so low that transient loop-lag false-disconnects healthy clients; gate/adjust with the existing loop-lag band if lag is elevated. Env-driven, so tunable per deployment.
SO_KEEPALIVE — httpServer.on('connection', s => s.setKeepAlive(true, 20000)) (or on engine.io's transport) as OS-level defense-in-depth.
Make mark-offline consistent with socket state — when the heartbeat checker marks a device offline, also disconnect(true) any lingering socket for it, so DB-offline can never diverge into a silent half-open (and, on a one-directional break, the client is signalled).
Flap-limiter amplifier fix — do NOT escalate a paired device_id to the 30-min quarantine on reconnect churn (keep the soft reconnect-throttle for loop protection; a known device with a flaky link must keep being let back in). Scope strictly to paired devices so genuine anon/unidentified flappers are still limited — do not weaken the flap-limiter globally or reintroduce the load risk it exists to prevent.

Client (Android APK — the real per-device fix, separate release): 5. App-level liveness watchdog: device:heartbeat with an ACK + timeout; on a missed ack force socket.disconnect(); socket.connect(). Detects half-open in ~15–25s regardless of the lib/OEM timer behavior. (Requires a server device:heartbeat ack.)

Bold (edge): raise/disable the Sophos WS idle/session timeout for the host; disable SSL/DPI inspection for it; confirm no reverse proxy with a short WS read timeout.

Tests

Server closes a dead peer within pingTimeout — a client that completes the handshake then stops ponging is disconnected within pingInterval+pingTimeout (already demonstrated: /tmp/halfopen.cjs, 500/500 → 1005ms). Add as a regression at a short env timeout.
offline ⇒ closed — after the heartbeat checker marks a device offline, assert no open socket remains for it (Phase D-3).
SNAT mass reconnect not refused — N distinct device_ids reconnecting once each from the same IP are all admitted (flap-limiter identity-keyed); and a repeated cycle for a paired device is not quarantined (Phase D-4).
Client (instrumented/emulator): after a silent transport kill, the watchdog forces a reconnect within the ack-timeout window.

9.4 KiB Raw Blame History Unescape Escape