Commit graph

20 commits

Author SHA1 Message Date
Copilot ee62078f5b fix(scrape): skip SIGTERM/SIGINT export aborts like OOM
Stopping validation with kill/Ctrl+C returned exit 143/130 and failed
the whole target instead of SKIPPED + preserve partial. Added smoke for
exit 143; gitignore .dce-scrape.lock.
2026-06-03 06:06:15 -05:00
Copilot b9bb4bbe64 fix(host): flock scrape lock prevents concurrent container exports
Overlapping run-operator-validation invocations spawned twin yes_general
exports and repeated OOM skips. Host scrape now holds .dce-scrape.lock;
smokes bypass via DCE_SKIP_SCRAPE_LOCK. Added lock smoke (20/20 pass).
2026-06-03 06:03:47 -05:00
Copilot 928c0ef682 fix(audit): exclude .dce-temp partial exports from JSON audit
Operator validation failed when yes_general OOM left truncated exports
under .dce-temp. Audit and archive verification now skip in-progress temps;
smoke covers the partial-temp case. KotOR audit passes with temps present.
2026-06-03 05:59:54 -05:00
Copilot 8b54b6a498 test(scrape): preserve-partial smoke; fix host token-file precedence
Add offline regression for OOM skip preserving partial export temps.
Host wrapper now prefers DISCORD_TOKEN_FILE over inherited shell tokens
and always writes explicit compose env for auth-retry. All 19 smokes pass.
2026-06-03 05:52:39 -05:00
Copilot 87537eb8b0 fix(scrape): preserve partial temps on OOM; large-file salvage merge
OOM/aborted channel exports no longer delete partial temp downloads.
Salvage uses grep boundary repair with python merge/validate for files
over 64 MiB. Retain stale temps when merge fails instead of discarding.
2026-06-03 05:35:22 -05:00
Copilot 87284816d0 test(scrape): add abort exit 134 skip smoke; plan 041 closure
Extend run-discord-scrape-smoke with skip-abort target so OOM/abort
channel skip from plan 040 has offline regression coverage. Update
merge-readiness for 2026-05-30 and KotOR validation retry in progress.
2026-06-03 00:57:11 -05:00
Copilot 1608e7cfb0 fix(scrape): skip channels on OOM/abort export exit codes
Treat CLI exit 134/137/139 and abort/OOM log patterns as skippable
so KotOR yes_general core dump does not fail the entire target scrape.
2026-06-03 00:44:06 -05:00
Copilot bc1f727907 feat(scrape): complete validation resume (8/9 targets)
Resume per-target validation for five remaining servers; clarify
validation log labels (begin/done/failed). Document 8/9 pass in
merge-readiness; KotOR_discord_msgs fails on yes_general export.
2026-05-29 23:35:35 -05:00
Copilot b089137c52 docs(scrape): record per-target validation outcomes (plan 037)
Document full-validation-latest.log results in merge-readiness:
four targets scrape+audit pass; KotOR_discord_msgs and remainder
documented as pending while long-running validation continues.
2026-05-29 21:56:00 -05:00
Copilot 0b242ddfc4 docs(scrape): stamp merge-ready after host validation
Document offline/live/cron gates; align operator checklist with
run-operator-proof.
2026-05-29 16:37:57 -05:00
Boden a4f080e6d9 docs(scrape): record live operator proof on eod_discord
Host validation passed with podman-compose and GUI token sync; note disk
headroom before large archive merges.
2026-05-29 16:36:02 -05:00
Boden 65c9fb2206 feat(scrape): operator proof script and podman-compose smoke fix
Add run-operator-proof for one-target handoff/scrape/prove flows.
Prefer podman-compose on Podman hosts but honor DCE_DOCKER_BIN overrides
so offline smokes keep using fake compose shims.
2026-05-29 16:20:25 -05:00
Boden 9c22a3efee docs(scrape): track GUI zip bridge doc in source repo
Add docs/gui-zip-recurring-scrape-bridge.md and cross-links so GUI-only
users have a versioned quick-start beside the linux-x64 zip folder.
2026-05-29 16:08:36 -05:00
Boden c0818715a8 feat(scrape): add operator-handoff verification script
Single entrypoint runs disk summary, verify-operator-ready, and
run-documents-scrape --dry-run before cron or full scrapes.
2026-05-29 16:03:22 -05:00
Boden 44eadee634 feat(scrape): disk preflight on host runner for cron jobs
run-discord-scrape-host.sh runs verify --disk-only before preflight/scrape
so setup-cron monthly jobs fail fast when archive roots are low on space.
Harden bootstrap smoke to surface failures when dry-run fails.
2026-05-29 16:00:11 -05:00
Boden 1142e376b5 fix(scrape): disk preflight before compose and skippable disk errors
Fail fast when archive or repo paths lack free space (DCE_MIN_FREE_MB),
treat disk-full export failures as skippable channels, and add an offline
disk-space smoke. Smokes default DCE_MIN_FREE_MB=0 so CI stays portable.
2026-05-29 15:27:39 -05:00
Boden 76b4231d7a feat(scrape): per-target validation with continue-on-error
Run scrape and audit per enabled server independently; log summary
counts. Full host validation started via --per-target --continue-on-error.
2026-05-29 14:20:37 -05:00
Boden 1742a9d41e feat(scrape): add run-operator-validation orchestrator
Sync GUI token, verify readiness, run documents scrape, and audit JSON
with timestamped logs. Live eod_discord validation passed on host.
2026-05-29 14:19:04 -05:00
Boden 00bcbc5b21 feat(scrape): add verify-operator-ready host checks
One command validates compose, auth, config, and seeded archives before
bootstrap or cron. Includes offline smoke test (14 smokes total).
2026-05-29 14:16:10 -05:00
Boden 927d5e9607 docs(scrape): add merge readiness index and doc cross-links
Single reviewer/operator page for the recurring scrape feature with
validation commands; link from root and .docs indexes.
2026-05-29 14:14:44 -05:00