Copilot
7171d7b6e3
feat(scrape): add KotOR yes_general catch-up wrapper (plan 083)
...
One-command operator entry for channel 221726893064454144 with salvage-before-scrape,
log/summary defaults, and dry-run/validation/prove modes. Smoke suite now 24/24.
2026-06-03 11:55:22 -05:00
Copilot
3d65c0e8e5
feat(scrape): cron opt-in salvage-before-scrape
...
setup-cron.sh forwards --salvage-before-scrape to documents scrape for
operators recovering from OOM partials on scheduled runs.
2026-06-03 11:35:50 -05:00
Copilot
759e33efe9
feat(scrape): add --log-file tee to documents scrape
...
Live runs auto-write logs/documents-scrape-UTC.log and pair JSON
summary with the log basename; optional --log-file overrides the path.
2026-06-03 11:21:59 -05:00
Copilot
c8ed19d26b
feat(scrape): per-target JSON summaries in multi-target loops
...
Validation --per-target and multi-target proof now pass --summary-file
per scrape so each target gets its own operator-*-<target>-UTC summary.
2026-06-03 11:08:44 -05:00
Copilot
a929be48e8
feat(scrape): add print-scrape-summary CLI for JSON artifacts
...
Pretty-print version-1 scrape summary files with totals table, --oom-only
filter, and stdin support for operator validation/proof outputs.
2026-06-03 10:45:55 -05:00
Copilot
dbc887d81c
feat(scrape): JSON summary export for operator proof runs
...
Auto-enable DCE_RUN_SUMMARY_* when proof scrapes, support --log-file,
and recover summary JSON from the teed proof log when file write fails.
2026-06-03 10:35:48 -05:00
Copilot
5cfb2ed144
feat(scrape): host compose passthrough for JSON summary
...
Mount logs/ in compose, map DCE_RUN_SUMMARY_FILE to /logs, and auto-enable
JSON summary beside operator-validation log files when scraping.
2026-06-03 10:18:33 -05:00
Copilot
8ca55f299b
feat(scrape): per-target container_memory in scrape config
...
Single --target runs apply optional container_memory from
scrape-targets.json when global DCE_CONTAINER_MEMORY is unset.
KotOR_discord_msgs defaults to 8g; scrape.env still overrides.
2026-06-03 09:55:33 -05:00
Copilot
69ce1ca539
feat(scrape): optional DCE_CONTAINER_MEMORY compose mem_limit
...
Operators can raise the scrape container memory cap for large channel
catch-up (e.g. yes_general) via scrape.env without changing default runs.
2026-06-03 09:23:37 -05:00
Copilot
ad5384ecc1
docs(scrape): add salvage and lock operator playbook
...
Document scrape-lock-status, reclaim-stale, and salvage-before flags in
operator checklist, merge-readiness, and GUI bridge guide.
2026-06-03 07:10:18 -05:00
Copilot
0b242ddfc4
docs(scrape): stamp merge-ready after host validation
...
Document offline/live/cron gates; align operator checklist with
run-operator-proof.
2026-05-29 16:37:57 -05:00
Boden
3fd42d924e
feat(scrape): sync GUI bridge doc into linux-x64 folder
...
Add sync-gui-bridge-doc.sh to copy the versioned bridge markdown beside
the GUI zip; includes offline smoke test (18 total via run-all-smokes).
2026-05-29 16:10:17 -05:00
Boden
9c22a3efee
docs(scrape): track GUI zip bridge doc in source repo
...
Add docs/gui-zip-recurring-scrape-bridge.md and cross-links so GUI-only
users have a versioned quick-start beside the linux-x64 zip folder.
2026-05-29 16:08:36 -05:00
Boden
c0818715a8
feat(scrape): add operator-handoff verification script
...
Single entrypoint runs disk summary, verify-operator-ready, and
run-documents-scrape --dry-run before cron or full scrapes.
2026-05-29 16:03:22 -05:00
Boden
00bcbc5b21
feat(scrape): add verify-operator-ready host checks
...
One command validates compose, auth, config, and seeded archives before
bootstrap or cron. Includes offline smoke test (14 smokes total).
2026-05-29 14:16:10 -05:00
Boden
927d5e9607
docs(scrape): add merge readiness index and doc cross-links
...
Single reviewer/operator page for the recurring scrape feature with
validation commands; link from root and .docs indexes.
2026-05-29 14:14:44 -05:00
Boden
89091d76ef
feat(scrape): add run-all-smokes.sh and align CI docs
...
Single entrypoint for offline smokes; CI uses it instead of listing each
script. Update setup guide and operator checklist with full smoke table.
2026-05-29 14:13:31 -05:00
Boden
a2aeaaab9c
fix(scrape): audit and salvage corrupt archive JSON
...
Add audit-archive-json and salvage-truncated-export helpers, skip invalid
JSON during prove snapshots, and wire an audit smoke test into CI.
2026-05-29 14:09:46 -05:00
Boden
a0db7aec52
feat(scrape): sync token from GUI Settings.dat for live exports
...
Add sync-token-from-gui.sh; bootstrap points to it when channels are
forbidden. Verified live incremental scrape on eod_discord with GUI token.
2026-05-29 14:05:45 -05:00
Boden
8c7ae90f3f
feat(scrape): warn on bot-token preflight and document cron install
...
Bootstrap surfaces forbidden-channel guidance; operator checklist
documents the installed monthly cron job.
2026-05-29 14:03:01 -05:00
Boden
058aaa0960
feat(scrape): add bootstrap CI smoke and operator checklist
...
Document bootstrap-first workflow; verify dry-run in CI; live
bootstrap preflight validated against Documents archives.
2026-05-29 14:02:06 -05:00
