feat(scrape): add run-all-smokes.sh and align CI docs

Single entrypoint for offline smokes; CI uses it instead of listing each
script. Update setup guide and operator checklist with full smoke table.
This commit is contained in:
Boden 2026-05-29 14:13:31 -05:00
parent 10cd2a534d
commit 89091d76ef
5 changed files with 130 additions and 23 deletions

View file

@ -321,14 +321,24 @@ Space requirements:
## Smoke test validation ## Smoke test validation
Run the full local suite from the repo root (requires `jq`; `container-smoke.sh` also needs Docker/Podman and a writable `archive_root` from `config/scrape-targets.json`): Run the full offline suite from the repo root (requires `jq`):
```bash ```bash
chmod +x scripts/*.sh scripts/tests/*.sh ./scripts/run-all-smokes.sh
for script in scripts/tests/*.sh; do ```
echo "==> $script"
"$script" With Docker/Podman, include the container smoke:
done
```bash
./scripts/run-all-smokes.sh --include-container
```
**Archive integrity helpers** (not smokes; run against live `~/Documents` trees):
```bash
./scripts/audit-archive-json.sh --target KotOR_discord_msgs
./scripts/salvage-truncated-export.sh path/to/export.json # truncated JSON only
./scripts/prove-incremental-append.sh --target NAME # live grow-only proof (needs token)
``` ```
| Script | CI (`recurring-scrape-smoke`) | Notes | | Script | CI (`recurring-scrape-smoke`) | Notes |
@ -342,9 +352,13 @@ done
| `gh-approve-pr-runs-smoke.sh` | yes | Fork PR workflow helper | | `gh-approve-pr-runs-smoke.sh` | yes | Fork PR workflow helper |
| `documents-scrape-smoke.sh` | yes | Unified Documents workflow | | `documents-scrape-smoke.sh` | yes | Unified Documents workflow |
| `verify-documents-auth-smoke.sh` | yes | Archive verify + auth bootstrap | | `verify-documents-auth-smoke.sh` | yes | Archive verify + auth bootstrap |
| `container-smoke.sh` | no (local) | Docker build + `help` / `list-targets` | | `scrape-here-smoke.sh` | yes | Workspace bridge launcher |
| `bootstrap-recurring-scrape-smoke.sh` | yes | Bootstrap dry-run |
| `audit-archive-json-smoke.sh` | yes | Invalid JSON detection |
| `prove-incremental-append-smoke.sh` | yes | Offline prove snapshot/compare |
| `container-smoke.sh` | no (local) | Docker build + `help` / `list-targets`; use `--include-container` |
GitHub Actions runs the CI-marked scripts on every push/PR via `.github/workflows/main.yml` job `recurring-scrape-smoke`. GitHub Actions runs `./scripts/run-all-smokes.sh` via `.github/workflows/main.yml` job `recurring-scrape-smoke`.
## Next Steps ## Next Steps

View file

@ -65,21 +65,7 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y jq run: sudo apt-get update && sudo apt-get install -y jq
- name: Run recurring scrape smoke tests - name: Run recurring scrape smoke tests
run: | run: ./scripts/run-all-smokes.sh
chmod +x scripts/*.sh scripts/tests/*.sh
./scripts/tests/run-discord-scrape-smoke.sh
./scripts/tests/error-path-smoke.sh
./scripts/tests/cron-idempotency-smoke.sh
./scripts/tests/end-to-end-preflight-smoke.sh
./scripts/tests/setup-cron-smoke.sh
./scripts/tests/run-discord-scrape-host-smoke.sh
./scripts/tests/gh-approve-pr-runs-smoke.sh
./scripts/tests/documents-scrape-smoke.sh
./scripts/tests/verify-documents-auth-smoke.sh
./scripts/tests/scrape-here-smoke.sh
./scripts/tests/bootstrap-recurring-scrape-smoke.sh
./scripts/tests/audit-archive-json-smoke.sh
./scripts/tests/prove-incremental-append-smoke.sh
test: test:
# Tests need access to secrets, so we can't run them against PRs because of limited trust # Tests need access to secrets, so we can't run them against PRs because of limited trust

View file

@ -0,0 +1,28 @@
---
title: feat: Unified smoke runner and validation docs
type: feat
status: complete
date: 2026-05-29
origin: Repeated /lfg — consolidate test entrypoint and align docs with CI
---
# feat: Unified smoke runner and validation docs
## Summary
Recurring scrape is feature-complete. Add one authoritative smoke runner script, wire CI to it, and update setup docs for audit/salvage/prove offline modes.
## Requirements
| ID | Requirement |
|----|-------------|
| R1 | `scripts/run-all-smokes.sh` runs all `scripts/tests/*.sh` except `container-smoke.sh` by default |
| R2 | `--include-container` runs container smoke for local Docker/Podman validation |
| R3 | CI `recurring-scrape-smoke` job uses `run-all-smokes.sh` |
| R4 | `.docs/Recurring-Scrape-Setup.md` smoke table lists all 12 smokes + audit/salvage/prove notes |
| R5 | Operator checklist references `run-all-smokes.sh` |
## Verification
- `./scripts/run-all-smokes.sh`
- `./scripts/run-all-smokes.sh --include-container` when Docker available (optional local)

View file

@ -33,4 +33,10 @@ Installed jobs are marked `# BEGIN discord-scrape` in `crontab -l`. Logs append
See `../DiscordChatExporter.linux-x64/RECURRING-SCRAPE.md` or run `../DiscordChatExporter.linux-x64/bootstrap-recurring-scrape.sh`. See `../DiscordChatExporter.linux-x64/RECURRING-SCRAPE.md` or run `../DiscordChatExporter.linux-x64/bootstrap-recurring-scrape.sh`.
Validate scripts after changes:
```bash
./scripts/run-all-smokes.sh
```
Full detail: [.docs/Recurring-Scrape-Setup.md](../.docs/Recurring-Scrape-Setup.md) Full detail: [.docs/Recurring-Scrape-Setup.md](../.docs/Recurring-Scrape-Setup.md)

73
scripts/run-all-smokes.sh Executable file
View file

@ -0,0 +1,73 @@
#!/usr/bin/env bash
set -Eeuo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)
REPO_ROOT="${DCE_REPO_ROOT:-$(cd "$SCRIPT_DIR/.." && pwd -P)}"
INCLUDE_CONTAINER=0
usage() {
cat <<EOF
Usage:
$(basename "$0") [--include-container]
Run every script in scripts/tests/ (offline smokes by default).
Use --include-container to also run container-smoke.sh (needs Docker/Podman).
EOF
}
die() {
printf 'ERROR: %s\n' "$*" >&2
exit 1
}
main() {
while (($#)); do
case "$1" in
--include-container)
INCLUDE_CONTAINER=1
shift
;;
--help|-h)
usage
exit 0
;;
*)
die "Unknown option: $1"
;;
esac
done
local tests_dir="$REPO_ROOT/scripts/tests"
[[ -d "$tests_dir" ]] || die "Missing tests directory: $tests_dir"
chmod +x "$REPO_ROOT"/scripts/*.sh "$tests_dir"/*.sh 2>/dev/null || true
local script_path failures=0 ran=0
for script_path in "$tests_dir"/*.sh; do
[[ -f "$script_path" ]] || continue
local base
base=$(basename "$script_path")
if [[ "$base" == "container-smoke.sh" && "$INCLUDE_CONTAINER" -eq 0 ]]; then
printf 'SKIP: %s (pass --include-container to run)\n' "$base"
continue
fi
printf '==> %s\n' "$base"
if ! "$script_path"; then
failures=$((failures + 1))
fi
ran=$((ran + 1))
done
if (( ran == 0 )); then
die "No smoke scripts found under $tests_dir"
fi
if (( failures > 0 )); then
die "$failures smoke script(s) failed."
fi
printf 'All %d smoke script(s) passed.\n' "$ran"
}
main "$@"