mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2025-08-17 09:02:15 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity

readme refactor

Browse source
This commit is contained in:
ed 2022-10-24 18:48:12 +02:00
parent 4b2b4ed52d
commit 75e5e53276
5 changed files with 230 additions and 167 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

191
README.md
View file

@ -10,7 +10,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
* server only needs `py2.7` or `py3.3+`, all dependencies optional * server only needs `py2.7` or `py3.3+`, all dependencies optional
* browse/upload with [IE4](#browser-support) / netscape4.0 on win3.11 (heh) * browse/upload with [IE4](#browser-support) / netscape4.0 on win3.11 (heh)
* *resumable* uploads need `firefox 34+` / `chrome 41+` / `safari 7+` * protocols: [http](#the-browser) // [ftp](#ftp-server) // [webdav](#webdav-server) // [smb/cifs](#smb-server)
try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running from a basement in finland try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running from a basement in finland
@ -30,16 +30,16 @@ try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running fro
* [quickstart](#quickstart) - download **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** and you're all set! * [quickstart](#quickstart) - download **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** and you're all set!
* [on servers](#on-servers) - you may also want these, especially on servers * [on servers](#on-servers) - you may also want these, especially on servers
* [on debian](#on-debian) - recommended additional steps on debian * [on debian](#on-debian) - recommended additional steps on debian
* [notes](#notes) - general notes * [features](#features)
* [status](#status) - feature summary
* [testimonials](#testimonials) - small collection of user feedback * [testimonials](#testimonials) - small collection of user feedback
* [motivations](#motivations) - project goals / philosophy * [motivations](#motivations) - project goals / philosophy
* [future plans](#future-plans) - some improvement ideas * [notes](#notes) - general notes
* [bugs](#bugs) * [bugs](#bugs)
* [general bugs](#general-bugs) * [general bugs](#general-bugs)
* [not my bugs](#not-my-bugs) * [not my bugs](#not-my-bugs)
* [FAQ](#FAQ) - "frequently" asked questions * [FAQ](#FAQ) - "frequently" asked questions
* [accounts and volumes](#accounts-and-volumes) - per-folder, per-user permissions * [accounts and volumes](#accounts-and-volumes) - per-folder, per-user permissions
* [shadowing](#shadowing) - hiding specific subfolders
* [the browser](#the-browser) - accessing a copyparty server using a web-browser * [the browser](#the-browser) - accessing a copyparty server using a web-browser
* [tabs](#tabs) - the main tabs in the ui * [tabs](#tabs) - the main tabs in the ui
* [hotkeys](#hotkeys) - the browser has the following hotkeys * [hotkeys](#hotkeys) - the browser has the following hotkeys
@ -78,7 +78,6 @@ try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running fro
* [client examples](#client-examples) - interact with copyparty using non-browser clients * [client examples](#client-examples) - interact with copyparty using non-browser clients
* [mount as drive](#mount-as-drive) - a remote copyparty server as a local filesystem * [mount as drive](#mount-as-drive) - a remote copyparty server as a local filesystem
* [up2k](#up2k) - quick outline of the up2k protocol, see [uploading](#uploading) for the web-client * [up2k](#up2k) - quick outline of the up2k protocol, see [uploading](#uploading) for the web-client
* [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
* [performance](#performance) - defaults are usually fine - expect `8 GiB/s` download, `1 GiB/s` upload * [performance](#performance) - defaults are usually fine - expect `8 GiB/s` download, `1 GiB/s` upload
* [client-side](#client-side) - when uploading files * [client-side](#client-side) - when uploading files
* [security](#security) - some notes on hardening * [security](#security) - some notes on hardening
@ -96,16 +95,10 @@ try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running fro
* [install recommended deps](#install-recommended-deps) * [install recommended deps](#install-recommended-deps)
* [optional gpl stuff](#optional-gpl-stuff) * [optional gpl stuff](#optional-gpl-stuff)
* [sfx](#sfx) - the self-contained "binary" * [sfx](#sfx) - the self-contained "binary"
* [sfx repack](#sfx-repack) - reduce the size of an sfx by removing features
* [copyparty.exe](#copypartyexe) * [copyparty.exe](#copypartyexe)
* [install on android](#install-on-android) * [install on android](#install-on-android)
* [reporting bugs](#reporting-bugs) - ideas for context to include in bug reports * [reporting bugs](#reporting-bugs) - ideas for context to include in bug reports
* [building](#building) * [devnotes](#devnotes) - for build instructions etc, see [./docs/devnotes.md](./docs/devnotes.md)
* [dev env setup](#dev-env-setup)
* [just the sfx](#just-the-sfx)
* [complete release](#complete-release)
* [todo](#todo) - roughly sorted by priority
* [discarded ideas](#discarded-ideas)
## quickstart ## quickstart
@ -147,23 +140,7 @@ recommended additional steps on debian which enable audio metadata and thumbnai
(skipped `pyheif-pillow-opener` because apparently debian is too old to build it) (skipped `pyheif-pillow-opener` because apparently debian is too old to build it)
## notes ## features
general notes:
* paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
* because no browsers currently implement the media-query to do this properly orz
browser-specific:
* iPhone/iPad: use Firefox to download files
* Android-Chrome: increase "parallel uploads" for higher speed (android bug)
* Android-Firefox: takes a while to select files (their fix for ☝️)
* Desktop-Firefox: ~~may use gigabytes of RAM if your files are massive~~ *seems to be OK now*
* Desktop-Firefox: may stop you from deleting files you've uploaded until you visit `about:memory` and click `Minimize memory usage`
## status
feature summary
* backend stuff * backend stuff
* ☑ sanic multipart parser * ☑ sanic multipart parser
@ -233,20 +210,18 @@ project goals / philosophy
* no build steps; modify the js/python without needing node.js or anything like that * no build steps; modify the js/python without needing node.js or anything like that
## future plans ## notes
some improvement ideas general notes:
* paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
* because no browsers currently implement the media-query to do this properly orz
* the JS is a mess -- a preact rewrite would be nice browser-specific:
* preferably without build dependencies like webpack/babel/node.js, maybe a python thing to assemble js files into main.js * iPhone/iPad: use Firefox to download files
* good excuse to look at using virtual lists (browsers start to struggle when folders contain over 5000 files) * Android-Chrome: increase "parallel uploads" for higher speed (android bug)
* the UX is a mess -- a proper design would be nice * Android-Firefox: takes a while to select files (their fix for ☝️)
* very organic (much like the python/js), everything was an afterthought * Desktop-Firefox: ~~may use gigabytes of RAM if your files are massive~~ *seems to be OK now*
* true for both the layout and the visual flair * Desktop-Firefox: may stop you from deleting files you've uploaded until you visit `about:memory` and click `Minimize memory usage`
* something like the tron board-room ui (or most other hollywood ones, like ironman) would be :100:
* some of the python files are way too big
* `up2k.py` ended up doing all the file indexing / db management
* `httpcli.py` should be separated into modules in general
# bugs # bugs
@ -349,6 +324,13 @@ examples:
anyone trying to bruteforce a password gets banned according to `--ban-pw`; default is 24h ban for 9 failed attempts in 1 hour anyone trying to bruteforce a password gets banned according to `--ban-pw`; default is 24h ban for 9 failed attempts in 1 hour
## shadowing
hiding specific subfolders by mounting another volume on top of them
for example `-v /mnt::r -v /var/empty:web/certs:r` mounts the server folder `/mnt` as the webroot, but another volume is mounted at `/web/certs` -- so visitors can only see the contents of `/mnt` and `/mnt/web` (at URLs `/` and `/web`), but not `/mnt/web/certs` because URL `/web/certs` is mapped to `/var/empty`
# the browser # the browser
accessing a copyparty server using a web-browser accessing a copyparty server using a web-browser
@ -749,11 +731,11 @@ unsafe, slow, not recommended for wan, enable with `--smb` for read-only or `--
dependencies: `python3 -m pip install --user -U impacket==0.10.0` dependencies: `python3 -m pip install --user -U impacket==0.10.0`
* newer versions of impacket will hopefully work just fine but there is monkeypatching so maybe not * newer versions of impacket will hopefully work just fine but there is monkeypatching so maybe not
some big warnings specific to SMB/CIFS, in decreasing importance: some **BIG WARNINGS** specific to SMB/CIFS, in decreasing importance:
* not entirely confident that read-only is read-only * not entirely confident that read-only is read-only
* the smb backend is not fully integrated with vfs, meaning there could be security issues (path traversal). Please use `--smb-port` (see below) and [./bin#prisonpartysh](prisonparty) * the smb backend is not fully integrated with vfs, meaning there could be security issues (path traversal). Please use `--smb-port` (see below) and [prisonparty](./bin/prisonparty.sh)
* account passwords work per-volume as expected, but account permissions are ignored; all accounts have access to all volumes, and `--smbw` gives all accounts write-access everywhere * account passwords work per-volume as expected, but account permissions are ignored; all accounts have access to all volumes, and `--smbw` gives all accounts write-access everywhere
* shadowing (hiding the contents in subfolders by creating overlapping volumes) probably works as expected but no guarantees * [shadowing](#shadowing) probably works as expected but no guarantees
and some minor issues, and some minor issues,
* hot-reload of server config (`/?reload=cfg`) only works for volumes, not account passwords * hot-reload of server config (`/?reload=cfg`) only works for volumes, not account passwords
@ -1153,22 +1135,7 @@ regarding the frequent server log message during uploads;
* on this http connection, `2.77 GiB` transferred, `102.9 MiB/s` average, `948` chunks handled * on this http connection, `2.77 GiB` transferred, `102.9 MiB/s` average, `948` chunks handled
* client says `4` uploads OK, `0` failed, `3` busy, `1` queued, `10042 MiB` total size, `7198 MiB` and `00:01:09` left * client says `4` uploads OK, `0` failed, `3` busy, `1` queued, `10042 MiB` total size, `7198 MiB` and `00:01:09` left
design detail: [why chunk-hashes](#./docs/devnotes/#why-chunk-hashes)
## why chunk-hashes
a single sha512 would be better, right?
this is due to `crypto.subtle` [not yet](https://github.com/w3c/webcrypto/issues/73) providing a streaming api (or the option to seed the sha512 hasher with a starting hash)
as a result, the hashes are much less useful than they could have been (search the server by sha512, provide the sha512 in the response http headers, ...)
however it allows for hashing multiple chunks in parallel, greatly increasing upload speed from fast storage (NVMe, raid-0 and such)
* both the [browser uploader](#uploading) and the [commandline one](https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py) does this now, allowing for fast uploading even from plaintext http
hashwasm would solve the streaming issue but reduces hashing speed for sha512 (xxh128 does 6 GiB/s), and it would make old browsers and [iphones](https://bugs.webkit.org/show_bug.cgi?id=228552) unsupported
* blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
# performance # performance
@ -1401,25 +1368,7 @@ these are standalone programs and will never be imported / evaluated by copypart
the self-contained "binary" [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) will unpack itself and run copyparty, assuming you have python installed of course the self-contained "binary" [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) will unpack itself and run copyparty, assuming you have python installed of course
you can reduce the sfx size by repacking it; see [./docs/devnotes/#sfx-repack](#./docs/devnotes/#sfx-repack)
## sfx repack
reduce the size of an sfx by removing features
if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL)
* `393k` size of original sfx.py as of v1.1.3
* `310k` after `./scripts/make-sfx.sh re no-cm`
* `269k` after `./scripts/make-sfx.sh re no-cm no-hl`
the features you can opt to drop are
* `cm`/easymde, the "fancy" markdown editor, saves ~82k
* `hl`, prism, the syntax hilighter, saves ~41k
* `fnt`, source-code-pro, the monospace font, saves ~9k
* `dd`, the custom mouse cursor for the media player tray tab, saves ~2k
for the `re`pack to work, first run one of the sfx'es once to unpack it
**note:** you can also just download and run [scripts/copyparty-repack.sh](scripts/copyparty-repack.sh) -- this will grab the latest copyparty release from github and do a few repacks; works on linux/macos (and windows with msys2 or WSL)
## copyparty.exe ## copyparty.exe
@ -1464,86 +1413,6 @@ journalctl -aS '48 hour ago' -u copyparty | grep -C10 FILENAME | tee bug.log
if there's a wall of base64 in the log (thread stacks) then please include that, especially if you run into something freezing up or getting stuck, for example `OperationalError('database is locked')` -- alternatively you can visit `/?stack` to see the stacks live, so http://127.0.0.1:3923/?stack for example if there's a wall of base64 in the log (thread stacks) then please include that, especially if you run into something freezing up or getting stuck, for example `OperationalError('database is locked')` -- alternatively you can visit `/?stack` to see the stacks live, so http://127.0.0.1:3923/?stack for example
# building # devnotes
## dev env setup for build instructions etc, see [./docs/devnotes.md](./docs/devnotes.md)
you need python 3.9 or newer due to type hints
the rest is mostly optional; if you need a working env for vscode or similar
```sh
python3 -m venv .venv
. .venv/bin/activate
pip install jinja2 strip_hints # MANDATORY
pip install mutagen # audio metadata
pip install pyftpdlib # ftp server
pip install Pillow pyheif-pillow-opener pillow-avif-plugin # thumbnails
pip install black==21.12b0 click==8.0.2 bandit pylint flake8 isort mypy # vscode tooling
```
## just the sfx
first grab the web-dependencies from a previous sfx (assuming you don't need to modify something in those):
```sh
rm -rf copyparty/web/deps
curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
python3 x.py --version
rm x.py
mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/
```
then build the sfx using any of the following examples:
```sh
./scripts/make-sfx.sh # regular edition
./scripts/make-sfx.sh gz no-cm # gzip-compressed + no fancy markdown editor
```
## complete release
also builds the sfx so skip the sfx section above
in the `scripts` folder:
* run `make -C deps-docker` to build all dependencies
* run `./rls.sh 1.2.3` which uploads to pypi + creates github release + sfx
# todo
roughly sorted by priority
* nothing! currently
## discarded ideas
* reduce up2k roundtrips
* start from a chunk index and just go
* terminate client on bad data
* not worth the effort, just throw enough conncetions at it
* single sha512 across all up2k chunks?
* crypto.subtle cannot into streaming, would have to use hashwasm, expensive
* separate sqlite table per tag
* performance fixed by skipping some indexes (`+mt.k`)
* audio fingerprinting
* only makes sense if there can be a wasm client and that doesn't exist yet (except for olaf which is agpl hence counts as not existing)
* `os.copy_file_range` for up2k cloning
* almost never hit this path anyways
* up2k partials ui
* feels like there isn't much point
* cache sha512 chunks on client
* too dangerous -- overtaken by turbo mode
* comment field
* nah
* look into android thumbnail cache file format
* absolutely not
* indexedDB for hashes, cfg enable/clear/sz, 2gb avail, ~9k for 1g, ~4k for 100m, 500k items before autoeviction
* blank hashlist when up-ok to skip handshake
* too many confusing side-effects
* hls framework for Someone Else to drop code into :^)
* probably not, too much stuff to consider -- seeking, start at offset, task stitching (probably np-hard), conditional passthru, rate-control (especially multi-consumer), session keepalive, cache mgmt...

153
docs/devnotes.md Normal file
View file

@ -0,0 +1,153 @@
## devnotes toc
* top
* [future plans](#future-plans) - some improvement ideas
* [design](#design)
* [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
* [sfx repack](#sfx-repack) - reduce the size of an sfx by removing features
* [building](#building)
* [dev env setup](#dev-env-setup)
* [just the sfx](#just-the-sfx)
* [complete release](#complete-release)
* [todo](#todo) - roughly sorted by priority
* [discarded ideas](#discarded-ideas)
## future plans
some improvement ideas
* the JS is a mess -- a preact rewrite would be nice
* preferably without build dependencies like webpack/babel/node.js, maybe a python thing to assemble js files into main.js
* good excuse to look at using virtual lists (browsers start to struggle when folders contain over 5000 files)
* the UX is a mess -- a proper design would be nice
* very organic (much like the python/js), everything was an afterthought
* true for both the layout and the visual flair
* something like the tron board-room ui (or most other hollywood ones, like ironman) would be :100:
* some of the python files are way too big
* `up2k.py` ended up doing all the file indexing / db management
* `httpcli.py` should be separated into modules in general
# design
## why chunk-hashes
a single sha512 would be better, right?
this is due to `crypto.subtle` [not yet](https://github.com/w3c/webcrypto/issues/73) providing a streaming api (or the option to seed the sha512 hasher with a starting hash)
as a result, the hashes are much less useful than they could have been (search the server by sha512, provide the sha512 in the response http headers, ...)
however it allows for hashing multiple chunks in parallel, greatly increasing upload speed from fast storage (NVMe, raid-0 and such)
* both the [browser uploader](#uploading) and the [commandline one](https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py) does this now, allowing for fast uploading even from plaintext http
hashwasm would solve the streaming issue but reduces hashing speed for sha512 (xxh128 does 6 GiB/s), and it would make old browsers and [iphones](https://bugs.webkit.org/show_bug.cgi?id=228552) unsupported
* blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
# sfx repack
reduce the size of an sfx by removing features
if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL)
* `393k` size of original sfx.py as of v1.1.3
* `310k` after `./scripts/make-sfx.sh re no-cm`
* `269k` after `./scripts/make-sfx.sh re no-cm no-hl`
the features you can opt to drop are
* `cm`/easymde, the "fancy" markdown editor, saves ~82k
* `hl`, prism, the syntax hilighter, saves ~41k
* `fnt`, source-code-pro, the monospace font, saves ~9k
* `dd`, the custom mouse cursor for the media player tray tab, saves ~2k
for the `re`pack to work, first run one of the sfx'es once to unpack it
**note:** you can also just download and run [scripts/copyparty-repack.sh](scripts/copyparty-repack.sh) -- this will grab the latest copyparty release from github and do a few repacks; works on linux/macos (and windows with msys2 or WSL)
# building
## dev env setup
you need python 3.9 or newer due to type hints
the rest is mostly optional; if you need a working env for vscode or similar
```sh
python3 -m venv .venv
. .venv/bin/activate
pip install jinja2 strip_hints # MANDATORY
pip install mutagen # audio metadata
pip install pyftpdlib # ftp server
pip install Pillow pyheif-pillow-opener pillow-avif-plugin # thumbnails
pip install black==21.12b0 click==8.0.2 bandit pylint flake8 isort mypy # vscode tooling
```
## just the sfx
first grab the web-dependencies from a previous sfx (assuming you don't need to modify something in those):
```sh
rm -rf copyparty/web/deps
curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
python3 x.py --version
rm x.py
mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/
```
then build the sfx using any of the following examples:
```sh
./scripts/make-sfx.sh # regular edition
./scripts/make-sfx.sh gz no-cm # gzip-compressed + no fancy markdown editor
```
## complete release
also builds the sfx so skip the sfx section above
in the `scripts` folder:
* run `make -C deps-docker` to build all dependencies
* run `./rls.sh 1.2.3` which uploads to pypi + creates github release + sfx
# todo
roughly sorted by priority
* nothing! currently
## discarded ideas
* reduce up2k roundtrips
* start from a chunk index and just go
* terminate client on bad data
* not worth the effort, just throw enough conncetions at it
* single sha512 across all up2k chunks?
* crypto.subtle cannot into streaming, would have to use hashwasm, expensive
* separate sqlite table per tag
* performance fixed by skipping some indexes (`+mt.k`)
* audio fingerprinting
* only makes sense if there can be a wasm client and that doesn't exist yet (except for olaf which is agpl hence counts as not existing)
* `os.copy_file_range` for up2k cloning
* almost never hit this path anyways
* up2k partials ui
* feels like there isn't much point
* cache sha512 chunks on client
* too dangerous -- overtaken by turbo mode
* comment field
* nah
* look into android thumbnail cache file format
* absolutely not
* indexedDB for hashes, cfg enable/clear/sz, 2gb avail, ~9k for 1g, ~4k for 100m, 500k items before autoeviction
* blank hashlist when up-ok to skip handshake
* too many confusing side-effects
* hls framework for Someone Else to drop code into :^)
* probably not, too much stuff to consider -- seeking, start at offset, task stitching (probably np-hard), conditional passthru, rate-control (especially multi-consumer), session keepalive, cache mgmt...

3
docs/notes.sh
View file

@ -227,9 +227,6 @@ for d in /usr /var; do find $d -type f -size +30M 2>/dev/null; done | while IFS=
brew install python@2 brew install python@2
pip install virtualenv pip install virtualenv
# readme toc
cat README.md | awk 'function pr() { if (!h) {return}; if (/^ *[*!#|]/||!s) {printf "%s\n",h;h=0;return}; if (/.../) {printf "%s - %s\n",h,$0;h=0}; }; /^#/{s=1;pr()} /^#* *(install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0} /^#/{lv=length($1);sub(/[^ ]+ /,"");bab=$0;gsub(/ /,"-",bab);gsub(/\./,"",bab); h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);next} !h{next} {sub(/ .*/,"");sub(/[:;,]$/,"")} {pr()}' > toc; grep -E '^## readme toc' -B1000 -A2 <README.md >p1; grep -E '^## quickstart' -B2 -A999999 <README.md >p2; (cat p1; grep quickstart -A1000 <toc; cat p2) >README.md; rm p1 p2 toc
# fix firefox phantom breakpoints, # fix firefox phantom breakpoints,
# suggestions from bugtracker, doesnt work (debugger is not attachable) # suggestions from bugtracker, doesnt work (debugger is not attachable)
devtools settings >> advanced >> enable browser chrome debugging + enable remote debugging devtools settings >> advanced >> enable browser chrome debugging + enable remote debugging

6
scripts/sfx.py
View file

@ -231,11 +231,11 @@ def hashfile(fn):
def unpack(): def unpack():
"""unpacks the tar yielded by `data`""" """unpacks the tar yielded by `data`"""
name = "pe-copyparty." name = "pe-copyparty"
try: try:
name += str(os.geteuid()) name += "." + str(os.geteuid())
except: except:
name += "0" pass
tag = "v" + str(STAMP) tag = "v" + str(STAMP)
top = tempfile.gettempdir() top = tempfile.gettempdir()

44
scripts/toc.sh Executable file
View file

@ -0,0 +1,44 @@
#!/bin/bash
set -e
for f in README.md docs/devnotes.md; do
cat $f | awk '
function pr() {
if (!h) {return};
if (/^ *[*!#|]/||!s) {
printf "%s\n",h;
h=0;
return
};
if (/.../) {
printf "%s - %s\n",h,$0;
h=0
};
};
/^#/{s=1;pr()} /^#* *(install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0}
/^#/{
lv=length($1);
sub(/[^ ]+ /,"");
bab=$0;
gsub(/ /,"-",bab);
gsub(/\./,"",bab);
h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);
next
}
!h{next}
{sub(/ .*/,"");sub(/[:;,]$/,"")}
{pr()}
' > toc
grep -E '^#+ [^ ]+ toc$' -B1000 -A2 <$f >p1
h2="$(awk '/^#+ [^ ]+ toc$/{o=1;next} o&&/^#/{print;exit}' <$f)"
grep -F "$h2" -B2 -A999999 <$f >p2
(cat p1; grep -F "${h2#* }" -A1000 <toc; cat p2) >$f
rm p1 p2 toc
done