Archive/DiscordChatExporter
1
0
Fork 0
mirror of https://github.com/Tyrrrz/DiscordChatExporter.git synced 2026-06-10 00:02:37 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat(scrape): complete validation resume (8/9 targets)

Browse source 
Resume per-target validation for five remaining servers; clarify
validation log labels (begin/done/failed). Document 8/9 pass in
merge-readiness; KotOR_discord_msgs fails on yes_general export.
This commit is contained in:
Copilot 2026-05-29 23:35:35 -05:00
parent b089137c52
commit bc1f727907
3 changed files with 44 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
docs/plans/2026-05-29-039-feat-complete-validation-resume-plan.md Normal file
View file

@ -0,0 +1,28 @@
---
title: feat: Complete interrupted full-target validation
type: feat
status: complete
date: 2026-05-30
origin: /lfg — 2026-05-30 validation stalled on KotOR_discord_msgs; finish remaining targets
---
# feat: Complete interrupted full-target validation
## Summary
The 2026-05-30 `full-validation-latest.log` run completed 4/9 targets then stalled during `KotOR_discord_msgs` scrape. Resume validation per remaining target, fix misleading per-target log labels, and finalize merge-readiness table.
## Requirements
| ID | Requirement |
|----|-------------|
| R1 | Run `run-operator-validation.sh --target NAME` for each incomplete enabled target; append to `logs/validation-resume-20260530.log` |
| R2 | `docs/recurring-scrape-merge-readiness.md` table shows final pass/fail per target |
| R3 | `run-operator-validation.sh` logs `Per-target begin` / `Per-target done` instead of premature `Per-target pass` |
| R4 | `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` passes |
| R5 | PR #1538 updated |
## Verification
- `grep 'Operator validation finished' logs/validation-resume-20260530.log` (per target)
- 19 offline smokes

22
docs/recurring-scrape-merge-readiness.md
View file

@ -99,23 +99,23 @@ DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh --sync-gui --per-target -
--log-file logs/full-validation-latest.log --log-file logs/full-validation-latest.log
``` ```
**2026-05-30 run** (log `logs/full-validation-latest.log`, started `01:35:29Z`): **Combined 2026-05-30 validation** (`logs/full-validation-latest.log` + `logs/validation-resume-20260530.log`):
| Target | Scrape | Audit | Notes | | Target | Scrape | Audit | Notes |
|--------|--------|-------|-------| |--------|--------|-------|-------|
| ror_orig_discord | pass | pass | | | ror_orig_discord | pass | pass | full-validation run |
| ror_new_discord | pass | pass | | | ror_new_discord | pass | pass | full-validation run |
| openkotor_discord_msgs | pass | pass | | | openkotor_discord_msgs | pass | pass | full-validation run |
| KotOR_Speedrun_Discord | pass | pass | 7 channels skipped (forbidden) | | KotOR_Speedrun_Discord | pass | pass | 7 channels skipped (forbidden) |
| KotOR_discord_msgs | in progress / long-running | — | 27 channels; allow time on first full scrape | | holocron_toolset_discord | pass | pass | validation-resume |
| holocron_toolset_discord | pending | — | | | expanded_kotor_discord | pass | pass | validation-resume |
| expanded_kotor_discord | pending | — | | | eod_discord | pass | pass | validation-resume |
| eod_discord | pending | — | | | DS_Discord_msgs | pass | pass | validation-resume; some channels forbidden |
| DS_Discord_msgs | pending | — | | | KotOR_discord_msgs | **fail** | — | channel `221726893064454144` (`yes_general`) failed mid-export (~11%); see log |
Re-check completion: `grep 'Per-target summary\\|Operator validation finished' logs/full-validation-latest.log` **KotOR remediation:** ensure several GiB free on `/home`, run `./scripts/audit-archive-json.sh --target KotOR_discord_msgs`, salvage truncated JSON if needed, then `./scripts/run-operator-validation.sh --target KotOR_discord_msgs`.
**Disk:** ~25 GiB free on `/home` at 2026-05-30; still allow headroom for KotOR `yes_general`-scale merges. **Disk:** ~22 GiB free on `/home` (2026-05-30); large channel merges still need headroom.
## CI note (fork PRs) ## CI note (fork PRs)

6
scripts/run-operator-validation.sh
View file

@ -94,8 +94,9 @@ scrape_per_target() {
mapfile -t target_names < <(enabled_targets) mapfile -t target_names < <(enabled_targets)
for name in "${target_names[@]}"; do for name in "${target_names[@]}"; do
[[ -n "$name" ]] || continue [[ -n "$name" ]] || continue
log_step "Per-target pass: $name" log_step "Per-target begin: $name"
if ! run_step "run-documents-scrape ($name)" "$DOCUMENTS_SCRAPE" "${scrape_args[@]}" --target "$name"; then if ! run_step "run-documents-scrape ($name)" "$DOCUMENTS_SCRAPE" "${scrape_args[@]}" --target "$name"; then
log_step "Per-target failed: $name (scrape)"
failures=$((failures + 1)) failures=$((failures + 1))
if (( CONTINUE_ON_ERROR == 0 )); then if (( CONTINUE_ON_ERROR == 0 )); then
return 1 return 1
@ -103,12 +104,15 @@ scrape_per_target() {
continue continue
fi fi
if (( DRY_RUN )); then if (( DRY_RUN )); then
log_step "Per-target done: $name (dry-run)"
ok=$((ok + 1)) ok=$((ok + 1))
continue continue
fi fi
if run_step "audit-archive-json ($name)" "$AUDIT_JSON" --config "$CONFIG_PATH" --target "$name"; then if run_step "audit-archive-json ($name)" "$AUDIT_JSON" --config "$CONFIG_PATH" --target "$name"; then
log_step "Per-target done: $name (ok)"
ok=$((ok + 1)) ok=$((ok + 1))
else else
log_step "Per-target failed: $name (audit)"
failures=$((failures + 1)) failures=$((failures + 1))
if (( CONTINUE_ON_ERROR == 0 )); then if (( CONTINUE_ON_ERROR == 0 )); then
return 1 return 1