DiscordChatExporter/docs/plans/2026-05-27-003-feat-recurring-scrape-finalization-validation-plan.md

title: feat: Finalize and validate recurring Discord scrape automation type: feat status: completed date: 2026-05-27

feat: Finalize and validate recurring Discord scrape automation

Summary

The feat/recurring-cli-scrape branch has implemented the core recurring scraper infrastructure (scripts, Docker build, cron setup, smoke tests, and fixtures). This plan focuses on comprehensive validation and production hardening: verifying append-only safety end-to-end, testing all failure paths, ensuring documentation completeness, validating cron idempotency under stress, and creating a deployment readiness checklist.

The implementation stays in the wrapper/script layer and does not require changes to the core C# exporter. The validation approach is practical and executable: smoke-test suite coverage, edge-case scenario validation, cross-environment testing, and live iteration proofs.

---

Problem Frame

The recurring scraper is feature-complete but requires production-hardening before it can be trusted with real token + existing archive roots. The hard part is gaining confidence that:

• Append-only merge logic preserves existing history under all conditions (including partial failures, interrupted runs, conflicting local state)
• Error handling fails closed consistently across auth, config, target resolution, and archive-safety boundaries
• The cron installation mechanism stays idempotent across repeated setup runs with evolving target configurations
• Operator-facing documentation aligns with actual behavior, with clear setup, troubleshooting, and recovery paths
• The preflight validation path covers every safety requirement before unattended runs

---

Assumptions

This plan builds from the existing implementation, test fixtures, and smoke-test scaffolding already on the feat/recurring-cli-scrape branch. The items below represent validation-focused bets that should be confirmed during execution.

• The scripts run-discord-scrape.sh, setup-cron.sh, and run-discord-scrape-host.sh are the authoritative recurring-scraper implementations; the CLI project itself is unchanged.
• Smoke tests are the primary validation vehicle; formal integration tests are deferred to a future repo test suite if it emerges.
• The append-only merge logic in run-discord-scrape.sh is the critical data-safety contract and warrants the deepest validation coverage.
• Host cron remains the scheduler of record and the focus for idempotency and lock validation.
• README.md will be updated to surface the recurring-scraper capability at the repo's entry point, not buried in sub-documentation.
• Preflight validation is run-time-sufficient rather than compile-time-guaranteed; the shell layer cannot prove static correctness, only demonstrate runtime success.

---

Requirements

• R1. All append-only merge scenarios in the existing fixtures (append-existing.json, append-incremental.json, wrong-channel.json) pass automated validation with clear pass/fail signals.
• R2. Error handling paths cover: missing token, invalid config, unresolvable targets, mismatched channel identity, missing preflight, and failed docker operations—each tested with expected failure messages and no silent data loss.
• R3. Cron installation mechanism stays idempotent across repeated setup runs with different schedule and target selections; existing unrelated crontab entries are preserved.
• R4. Preflight validation exercises the full runtime path (source-built container startup, authenticated discovery, config/token visibility) and produces clear pass/fail output before cron is installed.
• R5. Documentation (README.md, .docs/Docker.md, .docs/Scheduling-Linux.md) describes the operator contract accurately: supported config keys, safety guarantees, failure modes, and recovery procedures.
• R6. Smoke-test suite runs reliably in CI and local environments; test fixtures remain deterministic and do not depend on external state (real Discord tokens, live servers, etc.).
• R7. The host-retry auth flow (added in commit 090884f) is validated: retry behavior is predictable, error messages are clear, and the retry logic does not mask underlying token/auth issues.

---

Scope Boundaries

• Implementation is frozen on this plan; only validation, documentation updates, and smoke-test enhancements are in scope. No new features or architectural changes.
• No performance optimization or refactoring of script logic unless it directly supports a validation goal.
• No changes to the core C# exporter or CLI project; the wrapper layer remains the only target.
• No cross-platform scheduler support beyond the existing Linux cron focus; macOS/Windows scheduling deferred.

Deferred to Follow-Up Work

• Full integration test suite in the repo's existing test infrastructure (if one emerges).
• Performance profiling or optimization of incremental export and merge logic.
• Cross-platform scheduler parity (Windows Task Scheduler, macOS launchd).
• Rehydrating edited messages or reactions on already-archived history.

---

Context & Research

Relevant Code and Patterns

• scripts/run-discord-scrape.sh — Core append-only merge and error handling logic.
• scripts/setup-cron.sh — Cron installation, idempotency, and preflight orchestration.
• scripts/run-discord-scrape-host.sh — Host-side lock and cron invocation wrapper.
• scripts/tests/ — Existing smoke-test suite (container-smoke.sh, run-discord-scrape-smoke.sh, setup-cron-smoke.sh, run-discord-scrape-host-smoke.sh).
• scripts/tests/test-fixtures/ — Fixture JSON files for append/merge validation.
• config/scrape-targets.json — Target configuration with guild_ids, channel_ids, output_dir, and schedule.
• Dockerfile and docker-compose.yml — Source-built container and compose configuration.
• STRATEGY.md — Product-level goals and tracks for the recurring scraper.
• .docs/Docker.md and .docs/Scheduling-Linux.md — Existing operator documentation (to be reviewed and updated).

Institutional Learnings

• No prior institutional learnings found; this is a first-time recurring-scraper implementation.

External References

• Bash best practices: error handling, set -e, trap handlers, fd locking.
• Docker build and compose best practices from existing repo patterns.
• cron idempotency patterns from Linux sysadmin practice.

---

Key Technical Decisions

• Validation-first approach: Smoke tests and fixtures are the validation vehicle rather than formal unit tests; this keeps the barrier low for shell-based integration work.
• Append-only safety is non-negotiable: Every merge scenario in the fixtures must pass, and new edge cases discovered during validation trigger fixture additions.
• Fail-closed by default: Ambiguous or unsafe state stops the affected target and never silently overwrites archives; error messages are explicit about why.
• Idempotency is enforced at the cron layer: Repeated setup runs should converge to a stable state; this is testable with fixture crontabs.
• Documentation drives trust: README.md and .docs/ materials are updated to reflect actual behavior; discrepancies are resolved by updating implementation, not documentation.
• Host cron is the authority: The recurring workflow does not attempt to override host timezone, scheduling, or lock semantics; all of those are host responsibilities.

---

Open Questions

Resolved During Planning

• What level of validation is sufficient before declaring the feature production-ready? Pass all smoke tests, cover error paths, validate end-to-end preflight, update documentation.
• Should new merge-logic edge cases discovered during validation add to the fixture set or remain one-off test runs? Add to fixtures so they're part of the permanent regression suite.

Deferred to Implementation

• How should the smoke-test suite be invoked in CI/CD? The implementer should decide whether to wire the tests into an existing repo test runner or keep them as standalone scripts for now.
• Should the host-retry auth flow be validated with a real Discord token or purely with mocked responses? Implementer choice; mocked responses are sufficient for validation, but real-token testing may catch subtle timeout/retry edge cases.

---

High-Level Technical Design

This illustrates the intended validation approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.

Validation Flow

┌─────────────────────────────────────────────────────────────┐
│ Validation Checklist (All items must pass before release)   │
├─────────────────────────────────────────────────────────────┤
│ 1. Append-Only Merge Validation                             │
│    ├─ All fixtures pass (append-existing, incremental, etc) │
│    ├─ Edge case: partial write + retry = correct merge      │
│    └─ Edge case: concurrent appends don't corrupt           │
│ 2. Error Handling Validation                                │
│    ├─ Missing token → clear error, no archive touch        │
│    ├─ Invalid config → setup stops before cron install     │
│    ├─ Unresolvable target → logs and continues next target │
│    └─ Channel mismatch → archive preserved, target skipped  │
│ 3. Cron Idempotency Validation                              │
│    ├─ Install, then reinstall → one managed block only      │
│    ├─ Update schedule → only managed block changes          │
│    └─ Remove → managed block gone, other entries survive    │
│ 4. Preflight Validation                                     │
│    ├─ Container builds from source                          │
│    ├─ Auth layer is reachable with token                    │
│    ├─ Config discovery works                                │
│    └─ Lock mechanism is functional                          │
│ 5. Documentation Validation                                 │
│    ├─ README.md mentions recurring-scraper capability       │
│    ├─ Setup instructions are clear and complete             │
│    ├─ Error modes are documented                            │
│    └─ Recovery procedures are provided                      │
│ 6. Smoke Test Reliability Validation                        │
│    ├─ All tests pass locally                                │
│    ├─ Tests pass in CI (if integrated)                      │
│    ├─ Tests are deterministic (no timing/state issues)      │
│    └─ Fixtures are self-contained (no external deps)        │
└─────────────────────────────────────────────────────────────┘

---

Implementation Units

U1. Deepen append-only merge test coverage

Goal: Validate that the merge logic preserves existing local history under all plausible edge cases and failure scenarios.

Requirements: R1, R6

Dependencies: None

Files:

• Modify: scripts/tests/run-discord-scrape-smoke.sh
• Modify: scripts/tests/test-fixtures/append-existing.json
• Create: scripts/tests/test-fixtures/append-partial-write.json
• Create: scripts/tests/test-fixtures/append-concurrent-conflict.json
• Create: scripts/tests/validation-checklist.md

Approach:

• Review the existing append-only merge logic in run-discord-scrape.sh and identify all paths where data could be lost or corrupted.
• Enhance the smoke-test suite with additional fixture scenarios: partial writes interrupted mid-merge, concurrent export attempts, timestamp edge cases, empty incremental exports.
• Add validation assertions to confirm that existing JSON structure and message count are preserved after each merge scenario.
• Document the test scenarios clearly so operators understand what safety guarantees they have.

Execution note: Start by running the existing fixtures and understanding the current merge logic flow, then identify edge cases and add fixture scenarios.

Patterns to follow:

• scripts/tests/run-discord-scrape-smoke.sh — existing test structure
• scripts/tests/test-fixtures/append-*.json — fixture naming and structure
• scripts/run-discord-scrape.sh — merge logic implementation to understand

Test scenarios:

• Happy path: existing archive + incremental new messages = merged archive with all messages, sorted by ID.
• Happy path: first export creates a new archive with correct structure and metadata.
• Edge case: incremental export with zero new messages leaves the existing archive unchanged (byte-for-byte).
• Edge case: overlapping message IDs between existing and incremental are deduplicated.
• Edge case: missing incremental file after export attempt leaves the existing archive unchanged.
• Error path: corrupted destination JSON fails that target without attempting merge.
• Error path: channel metadata mismatch (guildId, channelId mismatch) aborts merge and preserves existing archive.
• Integration: a fixture that removes older messages from the incremental export still produces a merged archive with original history intact.
• Integration: repeated merges of the same incremental file (simulating a retry) produce identical results (idempotent).

Verification:

• All fixture scenarios pass and produce deterministic, reproducible results.
• Error paths produce explicit failure messages and never silently replace archives.
• Smoke-test output clearly signals pass/fail for each scenario.

---

U2. Validate error handling across all failure modes

Goal: Ensure that the recurring scraper fails safely and clearly when token is missing, config is invalid, targets cannot be resolved, or archive state is ambiguous.

Requirements: R2, R4

Dependencies: None

Files:

• Create: scripts/tests/error-path-smoke.sh
• Create: scripts/tests/test-configs/invalid-output-dir.json
• Create: scripts/tests/test-configs/missing-guild.json
• Create: scripts/tests/test-configs/duplicate-output-dir.json
• Modify: scripts/tests/validation-checklist.md

Approach:

• Map all error conditions from the plan (missing token, invalid config, unresolvable target, channel mismatch, etc.).
• Write a dedicated error-path smoke test that exercises each condition with expected failure messages.
• Verify that each error condition stops the affected target without silencing other targets or mutating crontab.
• Document the expected error messages so operators can troubleshoot.

Patterns to follow:

• scripts/run-discord-scrape.sh — error handling patterns (set -e, trap handlers, explicit error messages)
• scripts/tests/run-discord-scrape-smoke.sh — test structure for validation

Test scenarios:

• Error path: missing DISCORD_TOKEN env variable → setup fails with clear message before cron install.
• Error path: invalid output_dir (outside approved root) → config validation rejects it before any export.
• Error path: duplicate output_dir across targets → validation fails before setup.
• Error path: guild_id not found or not accessible → target is skipped with a clear log message.
• Error path: channel mismatch in existing archive → that target fails without archive replacement.
• Error path: docker compose build fails → setup stops before cron install.
• Error path: host lock already held (another run in progress) → cron command logs and exits gracefully.

Verification:

• Each error condition produces a clear, actionable error message.
• No silent data loss or archive corruption occurs.
• Unrelated targets are not affected by a single target's failure.

---

U3. Test cron idempotency and lifecycle management

Goal: Verify that the cron installation mechanism stays stable and idempotent across repeated setup runs, schedule changes, and removals.

Requirements: R3, R4

Dependencies: None

Files:

• Create: scripts/tests/cron-idempotency-smoke.sh
• Create: scripts/tests/test-crontabs/fixture-with-unrelated-entries.txt
• Modify: scripts/tests/validation-checklist.md

Approach:

• Create a smoke test that exercises the full cron lifecycle: install, reinstall with new schedule, update targets, remove.
• Use fixture crontabs (text files representing a pre-existing user's crontab) to ensure unrelated entries are preserved.
• Verify that setup converges to a single managed block and is safe to re-run.
• Test the --dry-run and --remove paths to ensure they work as expected.

Patterns to follow:

• scripts/setup-cron.sh — cron lifecycle implementation
• Existing cron testing patterns in the branch

Test scenarios:

• Happy path: initial install creates one managed cron block with monthly default schedule.
• Happy path: rerunning setup with same config produces no changes (idempotent).
• Happy path: rerunning with new schedule replaces only the managed block and preserves unrelated entries.
• Happy path: --dry-run shows the intended managed block without touching the live crontab.
• Happy path: --remove deletes only the managed block and leaves unrelated entries intact.
• Edge case: pre-existing fixture crontab with many unrelated entries survives a full lifecycle (install → update → remove).
• Error path: failed preflight leaves crontab untouched.

Verification:

• Cron installation mechanism converges to a stable, idempotent state.
• Unrelated crontab entries are always preserved.
• Dry-run and remove operations work as expected.

---

U4. Validate preflight and end-to-end setup path

Goal: Ensure the preflight validation covers all runtime requirements and proves the recurring scraper is ready before cron is installed.

Requirements: R4, R5, R7

Dependencies: U1, U2, U3

Files:

• Create: scripts/tests/end-to-end-preflight-smoke.sh
• Modify: .docs/Scheduling-Linux.md — preflight section
• Modify: scripts/tests/validation-checklist.md

Approach:

• Design and execute a smoke test that runs the full preflight path: container build, config visibility, auth token validation, discovery success.
• Verify that a successful preflight leads to cron install and a failed preflight leaves crontab untouched.
• Document the preflight path clearly for operators so they understand what's being validated.
• Test the host-retry auth flow (commit 090884f) to ensure retries are predictable and don't mask real auth failures.

Patterns to follow:

• scripts/setup-cron.sh — preflight orchestration
• scripts/tests/container-smoke.sh — container validation patterns

Test scenarios:

• Happy path: preflight succeeds with valid token and config → cron install proceeds.
• Happy path: preflight shows accessible targets and estimated schedule clearly.
• Error path: missing DISCORD_TOKEN → preflight fails before cron install.
• Error path: docker build fails → setup stops before cron install.
• Error path: config not visible or invalid → setup stops before cron install.
• Integration: full lifecycle (preflight → install → dry-run → remove) succeeds end-to-end.

Verification:

• Preflight validation is comprehensive and covers all safety requirements.
• Failed preflight prevents cron installation.
• Successful preflight gives operators clear confidence in the runtime setup.

---

U5. Complete and align documentation with implementation

Goal: Ensure README.md and .docs/ materials describe the operator contract accurately: setup, configuration, failure modes, and recovery procedures.

Requirements: R5, R6

Dependencies: U1, U2, U3, U4

Files:

• Modify: Readme.md
• Modify: .docs/Docker.md
• Modify: .docs/Scheduling-Linux.md
• Create: .docs/Recurring-Scrape-Setup.md
• Create: .docs/Recurring-Scrape-Troubleshooting.md

Approach:

• Add a high-level section to README.md that mentions the recurring-scraper capability and links to detailed setup docs.
• Review .docs/Docker.md and .docs/Scheduling-Linux.md for accuracy against the current implementation; update descriptions, examples, and error messages to match behavior.
• Create two new documents: a quick-start setup guide (Recurring-Scrape-Setup.md) and a troubleshooting guide (Recurring-Scrape-Troubleshooting.md).
• Ensure all documented flags, defaults, and safety constraints match the implemented behavior.

Patterns to follow:

• .docs/Docker.md and .docs/Scheduling-Linux.md — existing documentation style and structure
• Readme.md — high-level feature descriptions

Test scenarios:

• Test expectation: none -- documentation-only unit. Review should confirm that documented flags, examples, and safety guarantees match the implemented behavior.

Verification:

• README.md surfaces the recurring-scraper feature prominently.
• .docs/Recurring-Scrape-Setup.md provides clear, step-by-step instructions for first-time setup.
• .docs/Recurring-Scrape-Troubleshooting.md covers the most common failure modes and recovery steps.
• All documented error messages, defaults, and config keys match the implementation.
• External readers can set up the recurring scraper from the documentation without needing to reverse-engineer the scripts.

---

U6. Create production-readiness checklist and sign-off

Goal: Produce a clear, verifiable checklist that confirms the feature is production-ready for release.

Requirements: R1-R7

Dependencies: U1, U2, U3, U4, U5

Files:

• Create: docs/recurring-scrape-production-checklist.md
• Modify: docs/plans/2026-05-27-003-feat-recurring-scrape-finalization-validation-plan.md — add final sign-off section

Approach:

• Compile all validation results (smoke-test pass rates, edge-case coverage, error-handling validation, idempotency proof, documentation alignment) into a single production-readiness checklist.
• Include specific test commands and expected outcomes so future reviewers or maintainers can re-validate if needed.
• Document any known limitations or deferred follow-up work.
• Provide clear sign-off criteria: all tests pass, all error paths verified, all documentation updated and reviewed.

Patterns to follow:

• Existing validation-checklist.md sections from U1-U5

Test scenarios:

• Test expectation: none -- summary/attestation document. Review should confirm all prior units' validation results are captured and organized.

Verification:

• The checklist is comprehensive, specific, and verifiable.
• Future maintainers can reproduce the validation by following the checklist.
• Sign-off criteria are clear and leave no ambiguity about readiness.

---

System-Wide Impact

• Interaction graph: Host cron, Docker Compose, wrapper scripts, CLI, and local archives form a tightly coupled system; validation must exercise the full stack.
• Error propagation: Config/setup failures stop before cron mutation; target-level failures stop that target without affecting others; clear error messages guide operator troubleshooting.
• State lifecycle risks: Fixture crontabs, temporary merge files, and existing archives must remain coherent across repeated validation runs and interruptions.
• Integration coverage: Smoke tests validate source-built container, authenticated discovery, append-only merge, cron idempotency, and preflight path—all together, not in isolation.
• Documentation parity: Operator docs must match implementation; discrepancies are resolved by updating implementation, not softening documentation claims.
• Unchanged invariants: The upstream CLI remains the exporter of record; this plan does not modify core C# behavior, only validates the wrapper layer's safety.

---

Risks & Dependencies

Risk	Mitigation
Append-only merge logic still has unidentified edge cases	Deepen fixture coverage (U1); add edge cases discovered during validation to permanent fixture set
Error messages are unclear or missing, leading to operator confusion	Validate all error paths (U2); review error messages for clarity and actionability
Cron installation drifts and produces duplicate blocks after repeated setup runs	Test idempotency thoroughly with fixture crontabs (U3); verify managed-block markers are stable
Preflight validation passes but runtime fails, leaving cron in broken state	Run end-to-end smoke test that covers full lifecycle (U4); test host-retry auth flow for robustness
Documentation describes old behavior or missing config keys	Review docs against implementation (U5); cross-check with actual script output and error messages
Smoke tests are unreliable or time-sensitive, causing false failures in CI	Keep fixtures deterministic and self-contained (U6); avoid real Discord tokens or external dependencies

---

Documentation Plan

• README.md — Add recurring-scraper overview and link to detailed docs.
• .docs/Recurring-Scrape-Setup.md — Step-by-step first-time setup guide.
• .docs/Recurring-Scrape-Troubleshooting.md — Common issues and recovery steps.
• .docs/Docker.md and .docs/Scheduling-Linux.md — Update for accuracy and alignment with implementation.
• docs/recurring-scrape-production-checklist.md — Final validation results and readiness sign-off.

---

Operational & Rollout Notes

• The recurring scraper requires explicit operator action to install (via setup-cron.sh); no automatic deployment or background updates.
• Host cron is the scheduler of record; the operator owns the schedule, retention, and log rotation.
• The preflight validation path is designed to be safe for operators to run with real tokens and existing archives before committing to cron.
• Recovery from a failed run is manual (inspect logs, fix config, re-run setup or individual target exports).

---

Sources & References

• Related code: scripts/run-discord-scrape.sh
• Related code: scripts/setup-cron.sh
• Related code: scripts/run-discord-scrape-host.sh
• Related code: scripts/tests/ (smoke-test suite and fixtures)
• Related code: Dockerfile and docker-compose.yml
• Related docs: STRATEGY.md
• Related docs: .docs/Docker.md, .docs/Scheduling-Linux.md
• Existing plan: docs/plans/2026-05-24-001-feat-recurring-cli-scrape-automation-plan.md