GUI bridge and troubleshooting lead with run-kotor-yes-general-catchup.sh;
merge-readiness HEAD updated; bridge sync smoke asserts wrapper and 24/24 gate.
One-command operator entry for channel 221726893064454144 with salvage-before-scrape,
log/summary defaults, and dry-run/validation/prove modes. Smoke suite now 24/24.
Single --target runs apply optional container_memory from
scrape-targets.json when global DCE_CONTAINER_MEMORY is unset.
KotOR_discord_msgs defaults to 8g; scrape.env still overrides.
Classify aborted/OOM export skips as SKIPPED (OOM/aborted) in the run
summary with salvage/memory guidance; verify-operator-ready shows
configured DCE_CONTAINER_MEMORY.
Document container OOM skips, scrape-lock contention, partial temp
salvage, and DCE_CONTAINER_MEMORY in the troubleshooting guide and
GUI bridge quick-start.
* Stream asset downloads to avoid OutOfMemoryException on large files
ExportAssetDownloader used HttpClient.GetAsync(url, ct), which defaults to
HttpCompletionOption.ResponseContentRead and buffers the entire response body
into memory before returning. Downloading a large attachment therefore tried to
grow a single in-memory buffer to the full file size and threw
OutOfMemoryException (HttpContent.LoadIntoBufferAsync -> GrowAndWrite).
Pass HttpCompletionOption.ResponseHeadersRead so the body is streamed straight
to disk via CopyToAsync in bounded chunks, matching how DiscordClient already
issues its requests.
* Update ExportAssetDownloader.cs
---------
Co-authored-by: Oleksii Holub <1935960+Tyrrrz@users.noreply.github.com>
Centralize archive-root lock path, held checks, holder formatting, and
reclaim helpers in scripts/lib/scrape-lock.sh. Source it from the host
runner, lock status script, and operator wrappers to remove duplicated
logic. Update documents-scrape smoke fake repo to include the new lib.
Add archive-root lock preflight and --salvage-before-scrape to
run-documents-scrape.sh so direct operator invocations match validation
safety and KotOR catch-up workflow.
Wire --salvage-only through operator-handoff and run-operator-proof, and
refuse run-operator-validation scrape steps while the archive-root lock
is actively held.
Introduce read-only scrape-lock-status.sh and surface lock state during
operator-handoff so shared Documents archives show active or stale holders
before starting another scrape.
Expose --salvage-only and --salvage-before-scrape on run-operator-validation.sh
and --salvage-before-scrape on run-operator-proof.sh so operators can merge stale
.dce-temp exports before or instead of incremental Discord scrapes.
Expose run-discord-scrape.sh salvage and run-documents-scrape.sh
--salvage-only so operators can merge quiescent .dce-temp partials without
re-downloading from Discord after stopping a long or OOM-aborted run.
Serialize scrapes across repo checkouts that share the same Documents
archive_root, write lock meta with pid/cmd, and reclaim when the holder
process is dead.
Complete the operator CLI chain so focused yes_general catch-up can run
end-to-end with --target and --channel on handoff dry-run, prove scrape,
and operator-proof orchestration.
Avoid salvaging export.json while a channel export is still writing
(default: skip temps modified within 120s). Retry truncate+merge once
when merge fails on a quiescent partial temp. Adds active-skip smoke
and ages stale fixture mtime so salvage tests stay deterministic.
podman-compose and docker compose allocate a pseudo-TTY by default;
always passing -T block-buffered export progress. Omit -T for operator
runs and set DCE_COMPOSE_TTY=0 only for cron log append. Adds compose
TTY smokes and cron job env assertion.
Operators can narrow KotOR yes_general catch-up to a single channel ID
without re-scraping every guild channel. Adds dry-run and compose arg
capture smokes for the passthrough path.
Use tee in run_subcommand_with_retry so validation logs show live export
progress instead of buffering until compose exits. Add streaming smoke
assertion and skip scrape lock in documents-scrape smoke when prove runs
against the real repo root during parallel validation.
Stopping validation with kill/Ctrl+C returned exit 143/130 and failed
the whole target instead of SKIPPED + preserve partial. Added smoke for
exit 143; gitignore .dce-scrape.lock.
Operator validation failed when yes_general OOM left truncated exports
under .dce-temp. Audit and archive verification now skip in-progress temps;
smoke covers the partial-temp case. KotOR audit passes with temps present.
