test(scrape): add abort exit 134 skip smoke; plan 041 closure

Extend run-discord-scrape-smoke with skip-abort target so OOM/abort
channel skip from plan 040 has offline regression coverage. Update
merge-readiness for 2026-05-30 and KotOR validation retry in progress.
This commit is contained in:
Copilot 2026-06-03 00:57:11 -05:00
parent 1608e7cfb0
commit 87284816d0
3 changed files with 83 additions and 5 deletions

View file

@ -0,0 +1,56 @@
---
title: feat: Abort-skip smoke and KotOR validation closure
type: feat
status: complete
date: 2026-05-30
origin: /lfg — close plan 040 R3/R4 with offline abort smoke and KotOR retry
---
# feat: Abort-skip smoke and KotOR validation closure
## Summary
Plan 040 landed OOM/abort channel skip (exit 134/137/139) but lacked offline regression coverage and KotOR host re-validation. Add a fake-CLI abort case to `run-discord-scrape-smoke.sh`, re-run KotOR validation after image rebuild, and stamp merge-readiness when 9/9 or documented skip.
## Requirements
| ID | Requirement |
|----|-------------|
| R1 | `run-discord-scrape-smoke.sh` covers channel export exit 134 (abort) — target completes, accessible channel merges |
| R2 | `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` — 19/19 pass |
| R3 | Rebuild compose image; `run-operator-validation.sh --target KotOR_discord_msgs` completes (may skip `yes_general`) |
| R4 | `docs/recurring-scrape-merge-readiness.md` updated with 2026-05-30 closure and KotOR row |
| R5 | PR #1538 body notes plan 041 closure |
## Files
- `scripts/tests/run-discord-scrape-smoke.sh` — fake CLI channel `134` exits 134; `skip-abort` target
- `docs/recurring-scrape-merge-readiness.md` — host validation table
- `docs/plans/2026-05-30-040-fix-skip-oom-channel-export-plan.md` — mark R3/R4 complete when done
## Test scenarios (R1)
1. Two-channel target: channel `111` exports; channel `134` fake CLI exits 134 with abort stderr
2. Scrape exits 0; destination for `111` has appended messages
3. No fallback `channels/134.json` created
4. Run summary includes SKIPPED for channel 134
## Verification
```bash
./scripts/tests/run-discord-scrape-smoke.sh
DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh
docker compose build # or podman-compose build
DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --target KotOR_discord_msgs \
--log-file logs/kotor-retry-20260530.log
```
## Risks
- `yes_general` may take ~35 min before abort; validation log is authoritative
- Large merges need ~22 GiB free on `/home`
## Out of scope
- Increasing container memory for full `yes_general` export
- New `/lfg` feature work beyond closure

View file

@ -1,10 +1,10 @@
# Recurring scrape — merge readiness # Recurring scrape — merge readiness
## Branch status (2026-05-29) ## Branch status (2026-05-30)
| Gate | Status | | Gate | Status |
|------|--------| |------|--------|
| Offline smokes (`run-all-smokes.sh`) | 19/19 pass | | Offline smokes (`run-all-smokes.sh`) | 19/19 pass (includes abort exit 134 skip regression) |
| Live proof (`run-operator-proof.sh --sync-gui --target eod_discord`) | Passed on maintainer host | | Live proof (`run-operator-proof.sh --sync-gui --target eod_discord`) | Passed on maintainer host |
| Monthly cron (`setup-cron.sh`) | Installed (`00 04 1 * *`); dry-run preflight OK for all enabled targets | | Monthly cron (`setup-cron.sh`) | Installed (`00 04 1 * *`); dry-run preflight OK for all enabled targets |
| Upstream CI (fork PR) | `action_required` until Tyrrrz approves workflow runs | | Upstream CI (fork PR) | `action_required` until Tyrrrz approves workflow runs |
@ -111,9 +111,9 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --sync-gui --per-target -
| expanded_kotor_discord | pass | pass | validation-resume | | expanded_kotor_discord | pass | pass | validation-resume |
| eod_discord | pass | pass | validation-resume | | eod_discord | pass | pass | validation-resume |
| DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden | | DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden |
| KotOR_discord_msgs | **retry** | — | `yes_general` CLI abort (OOM); fixed in plan 040 to skip channel on exit 134/137/139 | | KotOR_discord_msgs | **in progress** | — | plan 041 retry after abort-skip fix; log `logs/kotor-retry-20260530.log` |
**KotOR remediation (plan 040):** `run-discord-scrape.sh` skips channels when export exits 134/137/139 (abort/OOM) or log matches disk/forbidden patterns. Re-run: **KotOR remediation (plan 040041):** `run-discord-scrape.sh` skips channels when export exits 134/137/139 (abort/OOM) or log matches disk/forbidden patterns. Offline regression: `run-discord-scrape-smoke.sh` `skip-abort` target. Re-run:
```bash ```bash
docker compose build # or podman-compose build docker compose build # or podman-compose build
@ -122,7 +122,7 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --target KotOR_discord_ms
Large `yes_general` may still skip; export that channel separately with more container memory if needed. Large `yes_general` may still skip; export that channel separately with more container memory if needed.
**Disk:** ~22 GiB free on `/home` (2026-05-30); large channel merges still need headroom. **Disk:** ~65 GiB free on `/home` (2026-05-30); large channel merges still need headroom.
## CI note (fork PRs) ## CI note (fork PRs)

View file

@ -126,6 +126,14 @@ cat >"$CONFIG_PATH" <<JSON
"channel_ids": ["111", "403"], "channel_ids": ["111", "403"],
"guild_ids": [], "guild_ids": [],
"guild_name_patterns": [] "guild_name_patterns": []
},
{
"name": "skip-abort",
"kind": "guild",
"output_dir": "$ARCHIVE_ROOT/skip-abort",
"channel_ids": ["111", "134"],
"guild_ids": [],
"guild_name_patterns": []
} }
] ]
} }
@ -177,6 +185,11 @@ case "$subcommand" in
exit 1 exit 1
fi fi
if [[ "$channel" == "134" ]]; then
echo "Aborted (core dumped)" >&2
exit 134
fi
case "$mode" in case "$mode" in
initial) cp "$fixture_dir/append-existing.json" "$output" ;; initial) cp "$fixture_dir/append-existing.json" "$output" ;;
append) cp "$fixture_dir/append-incremental.json" "$output" ;; append) cp "$fixture_dir/append-incremental.json" "$output" ;;
@ -357,6 +370,15 @@ SKIP_DEST="$ARCHIVE_ROOT/skip-forbidden/$DEFAULT_FILE_NAME"
[[ "$(jq -r '.messages | length' "$SKIP_DEST")" == "3" ]] || { echo "expected skip-forbidden to append accessible channel" >&2; exit 1; } [[ "$(jq -r '.messages | length' "$SKIP_DEST")" == "3" ]] || { echo "expected skip-forbidden to append accessible channel" >&2; exit 1; }
[[ ! -e "$ARCHIVE_ROOT/skip-forbidden/channels/403.json" ]] || { echo "unexpected fallback file for skipped forbidden channel" >&2; exit 1; } [[ ! -e "$ARCHIVE_ROOT/skip-forbidden/channels/403.json" ]] || { echo "unexpected fallback file for skipped forbidden channel" >&2; exit 1; }
mkdir -p "$ARCHIVE_ROOT/skip-abort"
cp "$FIXTURE_DIR/append-existing.json" "$ARCHIVE_ROOT/skip-abort/$DEFAULT_FILE_NAME"
SKIP_ABORT_LOG="$TMP_DIR/skip-abort.log"
run_wrapper skip-abort append 2>"$SKIP_ABORT_LOG"
SKIP_ABORT_DEST="$ARCHIVE_ROOT/skip-abort/$DEFAULT_FILE_NAME"
[[ "$(jq -r '.messages | length' "$SKIP_ABORT_DEST")" == "3" ]] || { echo "expected skip-abort to append accessible channel" >&2; exit 1; }
[[ ! -e "$ARCHIVE_ROOT/skip-abort/channels/134.json" ]] || { echo "unexpected fallback file for skipped abort channel" >&2; exit 1; }
grep -q 'SKIPPED.*134' "$SKIP_ABORT_LOG" || { echo "expected SKIPPED line for abort channel 134" >&2; exit 1; }
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$REPO_ROOT/scripts/run-discord-scrape.sh" source "$REPO_ROOT/scripts/run-discord-scrape.sh"
SHRINK_EXISTING="$TMP_DIR/shrink-existing.json" SHRINK_EXISTING="$TMP_DIR/shrink-existing.json"