mirror of
https://github.com/Tyrrrz/DiscordChatExporter.git
synced 2026-06-10 00:02:37 -06:00
e9a3fea9d1
Document container OOM skips, scrape-lock contention, partial temp salvage, and DCE_CONTAINER_MEMORY in the troubleshooting guide and GUI bridge quick-start.
89 lines
3.4 KiB
Markdown
89 lines
3.4 KiB
Markdown
# Recurring scrape — GUI zip users
|
|
|
|
The **DiscordChatExporter.linux-x64** (or similar) GUI zip is only the desktop app. Docker, cron, and append-only JSON merges live in the **source repo** checked out beside the zip:
|
|
|
|
```text
|
|
DiscordChatExporter.linux-x64/ ← GUI (this folder)
|
|
DiscordChatExporter/ ← source repo (scripts, Docker, config)
|
|
```
|
|
|
|
Sync to the GUI zip folder: `./scripts/sync-gui-bridge-doc.sh` (writes `../DiscordChatExporter.linux-x64/RECURRING-SCRAPE.md` by default).
|
|
|
|
## Quick start (run from source repo)
|
|
|
|
```bash
|
|
cd ../DiscordChatExporter # or your clone path
|
|
./scripts/operator-handoff.sh # disk + verify + archive dry-run
|
|
./scripts/sync-token-from-gui.sh --force # token from GUI Settings.dat
|
|
# or: cp scrape.env.example scrape.env and set DISCORD_TOKEN (never commit)
|
|
./scripts/bootstrap-recurring-scrape.sh # verify, build image, preflight
|
|
./scripts/run-documents-scrape.sh # incremental append-only scrape
|
|
./scripts/run-operator-proof.sh --sync-gui --target eod_discord # one-target handoff + scrape + grow-only proof
|
|
./scripts/setup-cron.sh --dry-run # preview monthly job (default 1st @ 04:00)
|
|
./scripts/setup-cron.sh # install cron after handoff passes
|
|
```
|
|
|
|
Optional integrity tools:
|
|
|
|
```bash
|
|
./scripts/audit-archive-json.sh
|
|
./scripts/scrape-lock-status.sh # show archive-root scrape lock
|
|
./scripts/scrape-lock-status.sh --reclaim-stale # clear dead-holder lock artifacts
|
|
# ./scripts/salvage-truncated-export.sh path/to/export.json
|
|
```
|
|
|
|
### Stuck or crashed export (partial `.dce-temp`)
|
|
|
|
After stopping a long run, merge quiescent partial exports before re-downloading history:
|
|
|
|
```bash
|
|
./scripts/scrape-lock-status.sh
|
|
./scripts/scrape-lock-status.sh --reclaim-stale # when state is stale
|
|
|
|
# Merge partial temps only (no Discord)
|
|
./scripts/operator-handoff.sh --salvage-only --target KotOR_discord_msgs --channel 221726893064454144
|
|
|
|
# Salvage then incremental catch-up (with audit + log)
|
|
# For large yes_general catch-up, set DCE_CONTAINER_MEMORY=8g in scrape.env first.
|
|
DCE_MIN_FREE_MB=0 ./scripts/run-operator-validation.sh \
|
|
--salvage-before-scrape \
|
|
--target KotOR_discord_msgs \
|
|
--channel 221726893064454144 \
|
|
--log-file logs/kotor-yes-general-$(date -u +%Y%m%d-%H%M%S).log
|
|
```
|
|
|
|
Or direct documents scrape:
|
|
|
|
```bash
|
|
./scripts/run-documents-scrape.sh \
|
|
--salvage-before-scrape \
|
|
--target KotOR_discord_msgs \
|
|
--channel 221726893064454144
|
|
```
|
|
|
|
If a temp is still being written, stop the export first. To merge an active temp after confirming nothing is writing: `DCE_SALVAGE_ACTIVE_TEMPS=1`.
|
|
|
|
Archives: `config/scrape-targets.json` (typically `~/Documents/*` per target `output_dir`).
|
|
|
|
**Disk:** Free several GiB on `/home` and archive roots before large scrapes (`DCE_MIN_FREE_MB`, default 1024).
|
|
|
|
**Validate scripts:** `DCE_MIN_FREE_MB=0 ./scripts/run-all-smokes.sh` (21 offline smokes)
|
|
|
|
**Podman (Fedora):** install `podman-compose` when `docker compose` cannot reach the socket; scripts auto-prefer it.
|
|
|
|
## Wrappers in the GUI zip folder
|
|
|
|
From `DiscordChatExporter.linux-x64/`:
|
|
|
|
```bash
|
|
./bootstrap-recurring-scrape.sh --dry-run
|
|
```
|
|
|
|
That script delegates to `../DiscordChatExporter/scripts/bootstrap-recurring-scrape.sh`.
|
|
|
|
## More detail
|
|
|
|
- [Recurring-Scrape-Setup.md](../.docs/Recurring-Scrape-Setup.md)
|
|
- [recurring-scrape-operator-checklist.md](recurring-scrape-operator-checklist.md)
|
|
- [recurring-scrape-merge-readiness.md](recurring-scrape-merge-readiness.md)
|