mirror of
https://github.com/Tyrrrz/DiscordChatExporter.git
synced 2026-06-10 00:02:37 -06:00
test(scrape): add abort exit 134 skip smoke; plan 041 closure
Extend run-discord-scrape-smoke with skip-abort target so OOM/abort channel skip from plan 040 has offline regression coverage. Update merge-readiness for 2026-05-30 and KotOR validation retry in progress.
This commit is contained in:
parent
1608e7cfb0
commit
87284816d0
|
|
@ -0,0 +1,56 @@
|
||||||
|
---
|
||||||
|
title: feat: Abort-skip smoke and KotOR validation closure
|
||||||
|
type: feat
|
||||||
|
status: complete
|
||||||
|
date: 2026-05-30
|
||||||
|
origin: /lfg — close plan 040 R3/R4 with offline abort smoke and KotOR retry
|
||||||
|
---
|
||||||
|
|
||||||
|
# feat: Abort-skip smoke and KotOR validation closure
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Plan 040 landed OOM/abort channel skip (exit 134/137/139) but lacked offline regression coverage and KotOR host re-validation. Add a fake-CLI abort case to `run-discord-scrape-smoke.sh`, re-run KotOR validation after image rebuild, and stamp merge-readiness when 9/9 or documented skip.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|-------------|
|
||||||
|
| R1 | `run-discord-scrape-smoke.sh` covers channel export exit 134 (abort) — target completes, accessible channel merges |
|
||||||
|
| R2 | `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` — 19/19 pass |
|
||||||
|
| R3 | Rebuild compose image; `run-operator-validation.sh --target KotOR_discord_msgs` completes (may skip `yes_general`) |
|
||||||
|
| R4 | `docs/recurring-scrape-merge-readiness.md` updated with 2026-05-30 closure and KotOR row |
|
||||||
|
| R5 | PR #1538 body notes plan 041 closure |
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `scripts/tests/run-discord-scrape-smoke.sh` — fake CLI channel `134` exits 134; `skip-abort` target
|
||||||
|
- `docs/recurring-scrape-merge-readiness.md` — host validation table
|
||||||
|
- `docs/plans/2026-05-30-040-fix-skip-oom-channel-export-plan.md` — mark R3/R4 complete when done
|
||||||
|
|
||||||
|
## Test scenarios (R1)
|
||||||
|
|
||||||
|
1. Two-channel target: channel `111` exports; channel `134` fake CLI exits 134 with abort stderr
|
||||||
|
2. Scrape exits 0; destination for `111` has appended messages
|
||||||
|
3. No fallback `channels/134.json` created
|
||||||
|
4. Run summary includes SKIPPED for channel 134
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/tests/run-discord-scrape-smoke.sh
|
||||||
|
DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh
|
||||||
|
docker compose build # or podman-compose build
|
||||||
|
DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --target KotOR_discord_msgs \
|
||||||
|
--log-file logs/kotor-retry-20260530.log
|
||||||
|
```
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- `yes_general` may take ~35 min before abort; validation log is authoritative
|
||||||
|
- Large merges need ~22 GiB free on `/home`
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Increasing container memory for full `yes_general` export
|
||||||
|
- New `/lfg` feature work beyond closure
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
# Recurring scrape — merge readiness
|
# Recurring scrape — merge readiness
|
||||||
|
|
||||||
## Branch status (2026-05-29)
|
## Branch status (2026-05-30)
|
||||||
|
|
||||||
| Gate | Status |
|
| Gate | Status |
|
||||||
|------|--------|
|
|------|--------|
|
||||||
| Offline smokes (`run-all-smokes.sh`) | 19/19 pass |
|
| Offline smokes (`run-all-smokes.sh`) | 19/19 pass (includes abort exit 134 skip regression) |
|
||||||
| Live proof (`run-operator-proof.sh --sync-gui --target eod_discord`) | Passed on maintainer host |
|
| Live proof (`run-operator-proof.sh --sync-gui --target eod_discord`) | Passed on maintainer host |
|
||||||
| Monthly cron (`setup-cron.sh`) | Installed (`00 04 1 * *`); dry-run preflight OK for all enabled targets |
|
| Monthly cron (`setup-cron.sh`) | Installed (`00 04 1 * *`); dry-run preflight OK for all enabled targets |
|
||||||
| Upstream CI (fork PR) | `action_required` until Tyrrrz approves workflow runs |
|
| Upstream CI (fork PR) | `action_required` until Tyrrrz approves workflow runs |
|
||||||
|
|
@ -111,9 +111,9 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --sync-gui --per-target -
|
||||||
| expanded_kotor_discord | pass | pass | validation-resume |
|
| expanded_kotor_discord | pass | pass | validation-resume |
|
||||||
| eod_discord | pass | pass | validation-resume |
|
| eod_discord | pass | pass | validation-resume |
|
||||||
| DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden |
|
| DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden |
|
||||||
| KotOR_discord_msgs | **retry** | — | `yes_general` CLI abort (OOM); fixed in plan 040 to skip channel on exit 134/137/139 |
|
| KotOR_discord_msgs | **in progress** | — | plan 041 retry after abort-skip fix; log `logs/kotor-retry-20260530.log` |
|
||||||
|
|
||||||
**KotOR remediation (plan 040):** `run-discord-scrape.sh` skips channels when export exits 134/137/139 (abort/OOM) or log matches disk/forbidden patterns. Re-run:
|
**KotOR remediation (plan 040–041):** `run-discord-scrape.sh` skips channels when export exits 134/137/139 (abort/OOM) or log matches disk/forbidden patterns. Offline regression: `run-discord-scrape-smoke.sh` `skip-abort` target. Re-run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose build # or podman-compose build
|
docker compose build # or podman-compose build
|
||||||
|
|
@ -122,7 +122,7 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --target KotOR_discord_ms
|
||||||
|
|
||||||
Large `yes_general` may still skip; export that channel separately with more container memory if needed.
|
Large `yes_general` may still skip; export that channel separately with more container memory if needed.
|
||||||
|
|
||||||
**Disk:** ~22 GiB free on `/home` (2026-05-30); large channel merges still need headroom.
|
**Disk:** ~65 GiB free on `/home` (2026-05-30); large channel merges still need headroom.
|
||||||
|
|
||||||
## CI note (fork PRs)
|
## CI note (fork PRs)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -126,6 +126,14 @@ cat >"$CONFIG_PATH" <<JSON
|
||||||
"channel_ids": ["111", "403"],
|
"channel_ids": ["111", "403"],
|
||||||
"guild_ids": [],
|
"guild_ids": [],
|
||||||
"guild_name_patterns": []
|
"guild_name_patterns": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "skip-abort",
|
||||||
|
"kind": "guild",
|
||||||
|
"output_dir": "$ARCHIVE_ROOT/skip-abort",
|
||||||
|
"channel_ids": ["111", "134"],
|
||||||
|
"guild_ids": [],
|
||||||
|
"guild_name_patterns": []
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
@ -177,6 +185,11 @@ case "$subcommand" in
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [[ "$channel" == "134" ]]; then
|
||||||
|
echo "Aborted (core dumped)" >&2
|
||||||
|
exit 134
|
||||||
|
fi
|
||||||
|
|
||||||
case "$mode" in
|
case "$mode" in
|
||||||
initial) cp "$fixture_dir/append-existing.json" "$output" ;;
|
initial) cp "$fixture_dir/append-existing.json" "$output" ;;
|
||||||
append) cp "$fixture_dir/append-incremental.json" "$output" ;;
|
append) cp "$fixture_dir/append-incremental.json" "$output" ;;
|
||||||
|
|
@ -357,6 +370,15 @@ SKIP_DEST="$ARCHIVE_ROOT/skip-forbidden/$DEFAULT_FILE_NAME"
|
||||||
[[ "$(jq -r '.messages | length' "$SKIP_DEST")" == "3" ]] || { echo "expected skip-forbidden to append accessible channel" >&2; exit 1; }
|
[[ "$(jq -r '.messages | length' "$SKIP_DEST")" == "3" ]] || { echo "expected skip-forbidden to append accessible channel" >&2; exit 1; }
|
||||||
[[ ! -e "$ARCHIVE_ROOT/skip-forbidden/channels/403.json" ]] || { echo "unexpected fallback file for skipped forbidden channel" >&2; exit 1; }
|
[[ ! -e "$ARCHIVE_ROOT/skip-forbidden/channels/403.json" ]] || { echo "unexpected fallback file for skipped forbidden channel" >&2; exit 1; }
|
||||||
|
|
||||||
|
mkdir -p "$ARCHIVE_ROOT/skip-abort"
|
||||||
|
cp "$FIXTURE_DIR/append-existing.json" "$ARCHIVE_ROOT/skip-abort/$DEFAULT_FILE_NAME"
|
||||||
|
SKIP_ABORT_LOG="$TMP_DIR/skip-abort.log"
|
||||||
|
run_wrapper skip-abort append 2>"$SKIP_ABORT_LOG"
|
||||||
|
SKIP_ABORT_DEST="$ARCHIVE_ROOT/skip-abort/$DEFAULT_FILE_NAME"
|
||||||
|
[[ "$(jq -r '.messages | length' "$SKIP_ABORT_DEST")" == "3" ]] || { echo "expected skip-abort to append accessible channel" >&2; exit 1; }
|
||||||
|
[[ ! -e "$ARCHIVE_ROOT/skip-abort/channels/134.json" ]] || { echo "unexpected fallback file for skipped abort channel" >&2; exit 1; }
|
||||||
|
grep -q 'SKIPPED.*134' "$SKIP_ABORT_LOG" || { echo "expected SKIPPED line for abort channel 134" >&2; exit 1; }
|
||||||
|
|
||||||
# shellcheck disable=SC1091
|
# shellcheck disable=SC1091
|
||||||
source "$REPO_ROOT/scripts/run-discord-scrape.sh"
|
source "$REPO_ROOT/scripts/run-discord-scrape.sh"
|
||||||
SHRINK_EXISTING="$TMP_DIR/shrink-existing.json"
|
SHRINK_EXISTING="$TMP_DIR/shrink-existing.json"
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue