mirror of
https://github.com/Tyrrrz/DiscordChatExporter.git
synced 2026-06-09 15:52:37 -06:00
feat(operators): pass --channel through handoff, prove, and proof
Complete the operator CLI chain so focused yes_general catch-up can run end-to-end with --target and --channel on handoff dry-run, prove scrape, and operator-proof orchestration.
This commit is contained in:
parent
ae120c916f
commit
8057a4443a
|
|
@ -0,0 +1,121 @@
|
|||
---
|
||||
title: "feat: Pass --channel through remaining operator wrappers"
|
||||
type: feat
|
||||
status: complete
|
||||
date: 2026-06-04
|
||||
origin: /lfg — plan 049 wired --channel on documents-scrape and operator-validation; handoff, prove, and operator-proof still reject it, blocking focused yes_general catch-up
|
||||
deepened: 2026-06-04
|
||||
---
|
||||
|
||||
# feat: Pass --channel through remaining operator wrappers
|
||||
|
||||
## Summary
|
||||
|
||||
Complete the operator CLI chain by accepting repeatable `--channel ID` in `operator-handoff.sh`, `prove-incremental-append.sh`, and `run-operator-proof.sh`, forwarding to downstream scripts the same way plan 049 did for documents-scrape and operator-validation.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
Operators re-running KotOR `yes_general` (`221726893064454144`) need single-channel workflows end-to-end:
|
||||
|
||||
```bash
|
||||
./scripts/run-operator-proof.sh --target KotOR_discord_msgs --channel 221726893064454144
|
||||
```
|
||||
|
||||
Plan 049 fixed `run-documents-scrape.sh` and `run-operator-validation.sh`, but `operator-handoff` dry-run, `prove-incremental-append` scrape step, and `run-operator-proof` orchestration still die on `--channel`.
|
||||
|
||||
## Requirements
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| R1 | `operator-handoff.sh` accepts repeatable `--channel ID` and forwards to `run-documents-scrape.sh --dry-run` |
|
||||
| R2 | `prove-incremental-append.sh` accepts repeatable `--channel ID` and forwards to `run-discord-scrape-host.sh scrape` |
|
||||
| R3 | `run-operator-proof.sh` accepts repeatable `--channel ID` and forwards to handoff, documents scrape, and prove |
|
||||
| R4 | Usage text on all three scripts documents `--channel` requires exactly one `--target` (enforced downstream) |
|
||||
| R5 | Smokes cover `--channel` acceptance on handoff dry-run path; prove smoke documents passthrough via arg capture or dry-run stub where feasible |
|
||||
| R6 | `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` passes |
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Mirror plan 049 passthrough pattern**: collect `--channel` into an array during option parsing; append to downstream invocations unchanged.
|
||||
- **Snapshot scope unchanged**: `prove-incremental-append` still snapshots all archives under the target; single-channel scrape must not shrink any existing archive (existing compare logic).
|
||||
- **Handoff only dry-runs scrape**: `--channel` affects the dry-run invocation only; verify-operator-ready remains target-wide.
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. operator-handoff --channel passthrough
|
||||
|
||||
**Goal:** Handoff dry-run accepts and forwards `--channel`.
|
||||
|
||||
**Requirements:** R1, R4
|
||||
|
||||
**Files:**
|
||||
- `scripts/operator-handoff.sh`
|
||||
- `scripts/tests/operator-handoff-smoke.sh`
|
||||
|
||||
**Approach:** Parse `--channel` into `CHANNEL_ARGS`; pass `"${CHANNEL_ARGS[@]}"` to `"$DOCUMENTS_SCRAPE" --dry-run`.
|
||||
|
||||
**Test scenarios:**
|
||||
- Handoff with `--target demo --channel 111...` on temp config completes with "Handoff complete" (dry-run path; no live Discord).
|
||||
|
||||
**Verification:** `operator-handoff-smoke.sh` passes.
|
||||
|
||||
### U2. prove-incremental-append --channel passthrough
|
||||
|
||||
**Goal:** Prove script forwards `--channel` to host scrape runner.
|
||||
|
||||
**Requirements:** R2, R4
|
||||
|
||||
**Files:**
|
||||
- `scripts/prove-incremental-append.sh`
|
||||
- `scripts/tests/prove-incremental-append-smoke.sh` (optional arg-capture if host stubbed; at minimum usage/help acceptance)
|
||||
|
||||
**Approach:** Parse `--channel` into array; append to `"$HOST_RUNNER" scrape ...` invocation.
|
||||
|
||||
**Test scenarios:**
|
||||
- `--help` or dry-run parse path accepts `--channel` without "Unknown option".
|
||||
- When fake host runner captures argv, `--channel` appears on scrape command (if smoke infrastructure allows; otherwise extend handoff-style config-only test).
|
||||
|
||||
**Verification:** `prove-incremental-append-smoke.sh` passes.
|
||||
|
||||
### U3. run-operator-proof --channel passthrough
|
||||
|
||||
**Goal:** End-to-end operator proof orchestrator forwards `--channel` to all three steps.
|
||||
|
||||
**Requirements:** R3, R4
|
||||
|
||||
**Files:**
|
||||
- `scripts/run-operator-proof.sh`
|
||||
|
||||
**Approach:** Parse `--channel` once; pass to `"$HANDOFF"`, `"$DOCUMENTS"`, and `"$PROVE"` when `--target` is set.
|
||||
|
||||
**Test scenarios:**
|
||||
- `--dry-run` with `--target demo --channel 111...` completes without unknown-option error.
|
||||
|
||||
**Verification:** Manual or documents-scrape-smoke pattern; full proof requires auth (out of scope for smoke).
|
||||
|
||||
### U4. Full smoke gate
|
||||
|
||||
**Goal:** Confirm no regressions across operator chain.
|
||||
|
||||
**Requirements:** R6
|
||||
|
||||
**Files:** (none — verification only)
|
||||
|
||||
**Verification:** `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` → 20/20 (or current count) pass.
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### In scope
|
||||
|
||||
- CLI passthrough on three remaining operator wrappers
|
||||
- Targeted smoke updates
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- Stopping stale full-target validation processes
|
||||
- Running live yes_general catch-up inside LFG
|
||||
- Container memory tuning for large channels
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Downstream `--channel` + single `--target` validation in `run-discord-scrape.sh` remains authoritative; wrappers do not re-validate channel membership.
|
||||
|
|
@ -10,17 +10,22 @@ CONFIG_PATH="${DCE_CONFIG_FILE:-$REPO_ROOT/config/scrape-targets.json}"
|
|||
VERIFY_READY="$REPO_ROOT/scripts/verify-operator-ready.sh"
|
||||
DOCUMENTS_SCRAPE="$REPO_ROOT/scripts/run-documents-scrape.sh"
|
||||
SKIP_DF=0
|
||||
TARGET=""
|
||||
CHANNEL_ARGS=()
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage:
|
||||
$(basename "$0") [--config PATH] [--skip-df]
|
||||
$(basename "$0") [--config PATH] [--skip-df] [--target NAME] [--channel ID]
|
||||
|
||||
Run operator handoff checks before cron install or a full scrape:
|
||||
1. Free-space summary (archive_root + repo)
|
||||
2. verify-operator-ready (jq, compose, auth, archives)
|
||||
3. run-documents-scrape --dry-run (archive paths only)
|
||||
|
||||
--target NAME Limit dry-run scrape plan to one configured target
|
||||
--channel ID With exactly one --target, limit dry-run to channel ID (repeatable)
|
||||
|
||||
Environment:
|
||||
DCE_MIN_FREE_MB Minimum MiB free (default 1024 in verify-operator-ready)
|
||||
EOF
|
||||
|
|
@ -63,6 +68,16 @@ main() {
|
|||
SKIP_DF=1
|
||||
shift
|
||||
;;
|
||||
--target)
|
||||
[[ $# -ge 2 ]] || die "Missing value for --target."
|
||||
TARGET=$2
|
||||
shift 2
|
||||
;;
|
||||
--channel)
|
||||
[[ $# -ge 2 ]] || die "Missing value for --channel."
|
||||
CHANNEL_ARGS+=(--channel "$2")
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
|
|
@ -86,10 +101,16 @@ main() {
|
|||
fi
|
||||
|
||||
"$VERIFY_READY" --config "$CONFIG_PATH"
|
||||
"$DOCUMENTS_SCRAPE" --dry-run --config "$CONFIG_PATH"
|
||||
local -a dry_run_args=(--dry-run --config "$CONFIG_PATH")
|
||||
[[ -n "$TARGET" ]] && dry_run_args+=(--target "$TARGET")
|
||||
dry_run_args+=("${CHANNEL_ARGS[@]}")
|
||||
"$DOCUMENTS_SCRAPE" "${dry_run_args[@]}"
|
||||
|
||||
printf '\nHandoff complete. Safe to run:\n'
|
||||
printf ' ./scripts/run-documents-scrape.sh\n'
|
||||
printf ' ./scripts/run-documents-scrape.sh'
|
||||
[[ -n "$TARGET" ]] && printf ' --target %s' "$TARGET"
|
||||
((${#CHANNEL_ARGS[@]})) && printf ' %s' "${CHANNEL_ARGS[*]}"
|
||||
printf '\n'
|
||||
printf ' ./scripts/setup-cron.sh --dry-run\n'
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ SNAPSHOT_DIR=""
|
|||
usage() {
|
||||
cat <<EOF
|
||||
Usage:
|
||||
$(basename "$0") --target NAME [--config PATH]
|
||||
$(basename "$0") --target NAME [--config PATH] [--channel ID]
|
||||
$(basename "$0") --target NAME --snapshot-only --snapshot-file PATH [--config PATH]
|
||||
$(basename "$0") --compare-snapshots BEFORE.tsv AFTER.tsv
|
||||
|
||||
|
|
@ -21,6 +21,8 @@ run one incremental scrape, then assert:
|
|||
- archive file paths are unchanged (no parallel channels/ fallbacks)
|
||||
- message counts never shrink
|
||||
|
||||
--channel ID Limit incremental scrape to channel ID (repeatable; requires --target)
|
||||
|
||||
Requires valid Discord auth (scrape.env, exported DISCORD_TOKEN, or token file).
|
||||
EOF
|
||||
}
|
||||
|
|
@ -119,6 +121,7 @@ main() {
|
|||
local snapshot_file=""
|
||||
local compare_before=""
|
||||
local compare_after=""
|
||||
local -a channel_args=()
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
|
|
@ -149,6 +152,11 @@ main() {
|
|||
compare_after=$3
|
||||
shift 3
|
||||
;;
|
||||
--channel)
|
||||
[[ $# -ge 2 ]] || die "Missing value for --channel."
|
||||
channel_args+=(--channel "$2")
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
|
|
@ -198,7 +206,7 @@ main() {
|
|||
"$REPO_ROOT/config/scrape-targets.json"|config/scrape-targets.json|./config/scrape-targets.json) ;;
|
||||
*) container_config="$CONFIG_PATH" ;;
|
||||
esac
|
||||
"$HOST_RUNNER" scrape --config "$container_config" --target "$target"
|
||||
"$HOST_RUNNER" scrape --config "$container_config" --target "$target" "${channel_args[@]}"
|
||||
|
||||
snapshot_archives "$output_dir" "$after_file"
|
||||
compare_snapshots "$before_file" "$after_file"
|
||||
|
|
|
|||
|
|
@ -16,17 +16,20 @@ source "$SCRIPT_DIR/lib/scrape-run-plan.sh"
|
|||
TARGET=""
|
||||
SYNC_GUI_FLAG=0
|
||||
DRY_RUN=0
|
||||
CHANNEL_ARGS=()
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage:
|
||||
$(basename "$0") [--target NAME] [--config PATH] [--sync-gui] [--dry-run]
|
||||
$(basename "$0") [--target NAME] [--channel ID] [--config PATH] [--sync-gui] [--dry-run]
|
||||
|
||||
End-to-end operator proof:
|
||||
operator-handoff → incremental scrape → prove-incremental-append
|
||||
|
||||
When --target is omitted, all enabled targets in the config are processed.
|
||||
|
||||
--channel ID With exactly one --target, limit scrape/prove to channel ID (repeatable)
|
||||
|
||||
Logs append to logs/operator-proof-<timestamp>.log
|
||||
EOF
|
||||
}
|
||||
|
|
@ -57,6 +60,11 @@ main() {
|
|||
DRY_RUN=1
|
||||
shift
|
||||
;;
|
||||
--channel)
|
||||
[[ $# -ge 2 ]] || die "Missing value for --channel."
|
||||
CHANNEL_ARGS+=(--channel "$2")
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
|
|
@ -98,7 +106,10 @@ main() {
|
|||
"$SYNC_GUI" --force
|
||||
fi
|
||||
|
||||
"$HANDOFF" --config "$CONFIG_PATH"
|
||||
local -a handoff_args=(--config "$CONFIG_PATH")
|
||||
[[ -n "$TARGET" ]] && handoff_args+=(--target "$TARGET")
|
||||
handoff_args+=("${CHANNEL_ARGS[@]}")
|
||||
"$HANDOFF" "${handoff_args[@]}"
|
||||
|
||||
if (( DRY_RUN == 1 )); then
|
||||
printf '\nDry run complete (no Discord scrape).\n'
|
||||
|
|
@ -107,7 +118,9 @@ main() {
|
|||
|
||||
for name in "${targets[@]}"; do
|
||||
printf '\n--- Target: %s ---\n' "$name"
|
||||
if "$DOCUMENTS" --config "$CONFIG_PATH" --target "$name" && "$PROVE" --config "$CONFIG_PATH" --target "$name"; then
|
||||
local -a scrape_args=(--config "$CONFIG_PATH" --target "$name")
|
||||
scrape_args+=("${CHANNEL_ARGS[@]}")
|
||||
if "$DOCUMENTS" "${scrape_args[@]}" && "$PROVE" "${scrape_args[@]}"; then
|
||||
succeeded=$((succeeded + 1))
|
||||
printf 'Operator proof passed for %s\n' "$name"
|
||||
else
|
||||
|
|
|
|||
|
|
@ -50,4 +50,20 @@ if [[ "$handoff_status" -ne 0 ]] || ! grep -q 'Handoff complete' <<<"$handoff_ou
|
|||
exit 1
|
||||
fi
|
||||
|
||||
set +e
|
||||
channel_output=$(
|
||||
DCE_MIN_FREE_MB=0 \
|
||||
DCE_CONFIG_FILE="$CONFIG_PATH" \
|
||||
DCE_ENV_FILE="$ENV_PATH" \
|
||||
"$HANDOFF" --config "$CONFIG_PATH" --skip-df --target demo --channel 111111111111111111 2>&1
|
||||
)
|
||||
channel_status=$?
|
||||
set -e
|
||||
|
||||
if [[ "$channel_status" -ne 0 ]] || ! grep -q 'Handoff complete' <<<"$channel_output"; then
|
||||
printf 'operator-handoff --channel failed (status=%s)\n' "$channel_status" >&2
|
||||
printf '%s\n' "$channel_output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf 'operator-handoff-smoke: ok\n'
|
||||
|
|
|
|||
|
|
@ -75,4 +75,9 @@ if "$PROVE" --compare-snapshots "$AFTER" "$BEFORE" 2>/dev/null; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
if ! "$PROVE" --help 2>&1 | grep -q -- '--channel'; then
|
||||
printf 'ERROR: prove --help should document --channel\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf 'prove-incremental-append-smoke: ok\n'
|
||||
|
|
|
|||
Loading…
Reference in a new issue