mirror of
https://github.com/Tyrrrz/DiscordChatExporter.git
synced 2026-06-09 15:52:37 -06:00
fix(scrape): skip channels on OOM/abort export exit codes
Treat CLI exit 134/137/139 and abort/OOM log patterns as skippable so KotOR yes_general core dump does not fail the entire target scrape.
This commit is contained in:
parent
bc1f727907
commit
1608e7cfb0
|
|
@ -0,0 +1,28 @@
|
|||
---
|
||||
title: fix: Skip OOM/aborted channel exports
|
||||
type: fix
|
||||
status: complete
|
||||
date: 2026-05-30
|
||||
origin: /lfg — KotOR yes_general CLI core dump should not fail entire target
|
||||
---
|
||||
|
||||
# fix: Skip OOM/aborted channel exports
|
||||
|
||||
## Summary
|
||||
|
||||
`yes_general` export aborted with `Aborted (core dumped)` inside the container, failing the whole `KotOR_discord_msgs` target. Extend skippable export failures so OOM/abort/kill errors skip the channel and continue (like forbidden channels).
|
||||
|
||||
## Requirements
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| R1 | `is_skippable_channel_export_failure` matches Aborted, core dumped, OOM, Killed |
|
||||
| R2 | `run-discord-scrape-smoke.sh` still passes |
|
||||
| R3 | Re-run `run-operator-validation.sh --target KotOR_discord_msgs` completes (may skip yes_general) |
|
||||
| R4 | Update merge-readiness KotOR row; 19 smokes pass |
|
||||
|
||||
## Verification
|
||||
|
||||
- `./scripts/tests/run-discord-scrape-smoke.sh`
|
||||
- `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh`
|
||||
- `logs/validation-resume-20260530.log` or new log for KotOR pass
|
||||
|
|
@ -111,9 +111,16 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --sync-gui --per-target -
|
|||
| expanded_kotor_discord | pass | pass | validation-resume |
|
||||
| eod_discord | pass | pass | validation-resume |
|
||||
| DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden |
|
||||
| KotOR_discord_msgs | **fail** | — | channel `221726893064454144` (`yes_general`) failed mid-export (~11%); see log |
|
||||
| KotOR_discord_msgs | **retry** | — | `yes_general` CLI abort (OOM); fixed in plan 040 to skip channel on exit 134/137/139 |
|
||||
|
||||
**KotOR remediation:** ensure several GiB free on `/home`, run `./scripts/audit-archive-json.sh --target KotOR_discord_msgs`, salvage truncated JSON if needed, then `./scripts/run-operator-validation.sh --target KotOR_discord_msgs`.
|
||||
**KotOR remediation (plan 040):** `run-discord-scrape.sh` skips channels when export exits 134/137/139 (abort/OOM) or log matches disk/forbidden patterns. Re-run:
|
||||
|
||||
```bash
|
||||
docker compose build # or podman-compose build
|
||||
DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --target KotOR_discord_msgs
|
||||
```
|
||||
|
||||
Large `yes_general` may still skip; export that channel separately with more container memory if needed.
|
||||
|
||||
**Disk:** ~22 GiB free on `/home` (2026-05-30); large channel merges still need headroom.
|
||||
|
||||
|
|
|
|||
|
|
@ -552,7 +552,7 @@ message_count() {
|
|||
is_skippable_channel_export_failure() {
|
||||
local log_file=$1
|
||||
grep -qiE \
|
||||
"failed: forbidden|failed: not found|Missing Access|403 Forbidden|404 Not Found|Cannot read message history|No space left on device|SQLITE_FULL|ENOSPC|disk full|not enough space" \
|
||||
"failed: forbidden|failed: not found|Missing Access|403 Forbidden|404 Not Found|Cannot read message history|No space left on device|SQLITE_FULL|ENOSPC|disk full|not enough space|Aborted \\(core dumped\\)|core dumped|out of memory|OOM|Killed|SIGKILL|SIGABRT" \
|
||||
"$log_file"
|
||||
}
|
||||
|
||||
|
|
@ -579,6 +579,14 @@ export_channel_incremental() {
|
|||
return 0
|
||||
fi
|
||||
|
||||
# SIGABRT (134), SIGKILL/OOM (137), SIGSEGV (139) — bash reports these when the CLI crashes.
|
||||
if (( export_status == 134 || export_status == 137 || export_status == 139 )); then
|
||||
log "Skipping channel $channel_id (export process aborted, exit $export_status)."
|
||||
[[ -s "$export_log" ]] && cat "$export_log" >&2
|
||||
rm -f "$export_log"
|
||||
return 2
|
||||
fi
|
||||
|
||||
if is_skippable_channel_export_failure "$export_log"; then
|
||||
log "Skipping channel $channel_id (inaccessible or non-fatal export error)."
|
||||
cat "$export_log" >&2
|
||||
|
|
|
|||
Loading…
Reference in a new issue