From 927d5e96074ab8796e94e23c18a692df597d0ef4 Mon Sep 17 00:00:00 2001 From: Boden Date: Fri, 29 May 2026 14:14:44 -0500 Subject: [PATCH] docs(scrape): add merge readiness index and doc cross-links Single reviewer/operator page for the recurring scrape feature with validation commands; link from root and .docs indexes. --- .docs/Readme.md | 4 +++ Readme.md | 2 +- ...026-05-29-021-docs-merge-readiness-plan.md | 25 +++++++++++++ docs/recurring-scrape-merge-readiness.md | 35 +++++++++++++++++++ docs/recurring-scrape-operator-checklist.md | 2 ++ 5 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 docs/plans/2026-05-29-021-docs-merge-readiness-plan.md create mode 100644 docs/recurring-scrape-merge-readiness.md diff --git a/.docs/Readme.md b/.docs/Readme.md index 01f86e51..bfa4fe55 100644 --- a/.docs/Readme.md +++ b/.docs/Readme.md @@ -15,6 +15,10 @@ - [Windows](Scheduling-Windows.md) - [macOS](Scheduling-MacOS.md) - [Linux](Scheduling-Linux.md) +- Recurring append-only scrape (Docker + cron, this fork): + - [Setup guide](Recurring-Scrape-Setup.md) + - [Troubleshooting](Recurring-Scrape-Troubleshooting.md) + - [Merge readiness](../docs/recurring-scrape-merge-readiness.md) ## Video tutorial diff --git a/Readme.md b/Readme.md index b4a677b4..cf69f4ba 100644 --- a/Readme.md +++ b/Readme.md @@ -81,7 +81,7 @@ To learn more about the war and how you can help, [click here](https://tyrrrz.me ## See also -- [**Recurring Exports**](.docs/Recurring-Scrape-Setup.md) — automated scheduled exports using cron (Linux/macOS). From the source repo run `./scripts/bootstrap-recurring-scrape.sh` (verify, build, preflight). If you only have the GUI zip (`DiscordChatExporter.linux-x64`), use `./bootstrap-recurring-scrape.sh` or `scripts/scrape-here.sh` in the sibling source repository. +- [**Recurring Exports**](.docs/Recurring-Scrape-Setup.md) — append-only incremental JSON exports via Docker/cron (Linux/macOS). From the source repo: `./scripts/bootstrap-recurring-scrape.sh`, then `./scripts/run-documents-scrape.sh`; validate with `./scripts/run-all-smokes.sh`. GUI zip users: `../DiscordChatExporter.linux-x64/RECURRING-SCRAPE.md`. Maintainer summary: [docs/recurring-scrape-merge-readiness.md](docs/recurring-scrape-merge-readiness.md). - [**Documented solutions**](docs/solutions/) — searchable learnings (append-only scrape, Docker/cron workflow); YAML frontmatter: `module`, `tags`, `problem_type` - [**Chat Analytics**](https://github.com/mlomb/chat-analytics) — solution for analyzing chat patterns of Discord users, using exports produced by **DiscordChatExporter**. - [**DiscordChatExporter-frontend**](https://github.com/slatinsky/DiscordChatExporter-frontend) — convenient viewer for exports produced by **DiscordChatExporter**. diff --git a/docs/plans/2026-05-29-021-docs-merge-readiness-plan.md b/docs/plans/2026-05-29-021-docs-merge-readiness-plan.md new file mode 100644 index 00000000..5d1baae2 --- /dev/null +++ b/docs/plans/2026-05-29-021-docs-merge-readiness-plan.md @@ -0,0 +1,25 @@ +--- +title: docs: Merge readiness index and doc cross-links +type: docs +status: complete +date: 2026-05-29 +origin: Repeated /lfg — feature stack complete; surface merge/operator entrypoints +--- + +# docs: Merge readiness index and doc cross-links + +## Summary + +Recurring scrape automation is implemented and tested. Add a merge-readiness doc for reviewers and wire documentation indexes so operators find setup, troubleshooting, and validation in one hop. + +## Requirements + +| ID | Requirement | +|----|-------------| +| R1 | `docs/recurring-scrape-merge-readiness.md` summarizes feature, validation commands, operator flow | +| R2 | `.docs/Readme.md` links recurring scrape setup and troubleshooting | +| R3 | Root `Readme.md` See also mentions `run-all-smokes.sh` validation | + +## Verification + +- `./scripts/run-all-smokes.sh` passes diff --git a/docs/recurring-scrape-merge-readiness.md b/docs/recurring-scrape-merge-readiness.md new file mode 100644 index 00000000..59123b5c --- /dev/null +++ b/docs/recurring-scrape-merge-readiness.md @@ -0,0 +1,35 @@ +# Recurring scrape — merge readiness + +Fork branch `feat/recurring-cli-scrape` adds append-only, Docker-based incremental exports with optional monthly cron. Intended for personal archive trees under a configurable `archive_root` (for example `~/Documents/*`). + +## What ships + +- **Config:** `config/scrape-targets.json` — per-server `output_dir`, optional `channel_ids`, `enabled` flags +- **Core:** `scripts/run-discord-scrape.sh` — incremental `--after`, merge-by-id, fail-closed path safety +- **Host:** `scripts/run-discord-scrape-host.sh`, `scripts/run-documents-scrape.sh`, `scripts/bootstrap-recurring-scrape.sh` +- **Auth:** `scrape.env`, `scripts/setup-scrape-auth.sh`, `scripts/sync-token-from-gui.sh` +- **Cron:** `scripts/setup-cron.sh` (`--interval monthly` default) +- **Integrity:** `scripts/audit-archive-json.sh`, `scripts/salvage-truncated-export.sh`, `scripts/prove-incremental-append.sh` +- **CI:** `.github/workflows/main.yml` job `recurring-scrape-smoke` runs `./scripts/run-all-smokes.sh` + +## Validate before merge + +```bash +./scripts/run-all-smokes.sh +./scripts/run-all-smokes.sh --include-container # optional; needs Docker/Podman +``` + +## Operator quick path + +```bash +cp scrape.env.example scrape.env # or ./scripts/sync-token-from-gui.sh --force +./scripts/bootstrap-recurring-scrape.sh +./scripts/run-documents-scrape.sh +./scripts/setup-cron.sh --dry-run +``` + +Detail: [.docs/Recurring-Scrape-Setup.md](../.docs/Recurring-Scrape-Setup.md) · [operator checklist](recurring-scrape-operator-checklist.md) · [troubleshooting](../.docs/Recurring-Scrape-Troubleshooting.md) + +## CI note (fork PRs) + +Upstream workflows may show `action_required` for cross-repo PRs from `th3w1zard1/DiscordChatExporter` until a maintainer approves workflow runs. Local `run-all-smokes.sh` is the authoritative offline gate. diff --git a/docs/recurring-scrape-operator-checklist.md b/docs/recurring-scrape-operator-checklist.md index 907ed8a7..1817a62a 100644 --- a/docs/recurring-scrape-operator-checklist.md +++ b/docs/recurring-scrape-operator-checklist.md @@ -39,4 +39,6 @@ Validate scripts after changes: ./scripts/run-all-smokes.sh ``` +Merge / review summary: [recurring-scrape-merge-readiness.md](recurring-scrape-merge-readiness.md) + Full detail: [.docs/Recurring-Scrape-Setup.md](../.docs/Recurring-Scrape-Setup.md)