mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2025-08-17 09:02:15 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity
copyparty/bin
History
ed 132a83501e add chunk stitching; twice as fast long-distance uploads: 
rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East,  `1.6x` from US-West to Finland )

the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body

a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value

this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies

the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes

a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case

on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
2024-07-21 23:35:37 +00:00
..
handlers 404/403 can be handled with plugins 2023-07-07 21:33:40 +00:00
hooks hooks: add cache-warmer 2024-07-17 21:00:59 +00:00
mtag rehost deps from a flaky server 2024-04-12 21:49:01 +00:00
dbtool.py make linters happier 2023-11-30 17:33:07 +00:00
partyfuse-streaming.py cosmetic 2022-11-26 22:27:09 +00:00
partyfuse.py make linters happier 2023-11-30 17:33:07 +00:00
partyfuse2.py cosmetic 2022-11-26 22:27:09 +00:00
partyjournal.py utcfromtimestamp was deprecated and nobody told me, 2023-10-20 23:41:58 +00:00
prisonparty.sh prisonparty: prevent overlapping setup/teardown 2023-11-25 14:03:41 +00:00
README.md rename up2k.py (client) to u2c.py 2023-05-07 15:35:56 +00:00
u2c.py add chunk stitching; twice as fast long-distance uploads: 2024-07-21 23:35:37 +00:00
unforget.py make linters happier 2023-11-30 17:33:07 +00:00
up2k.sh u2cli: add eta, errorhandling, better windows support 2021-10-01 22:31:24 +02:00

README.md

u2c.py

  • command-line up2k client (webm)
  • file uploads, file-search, autoresume of aborted/broken uploads
  • sync local folder to server
  • generally faster than browsers
  • if something breaks just restart it

partyjournal.py

produces a chronological list of all uploads by collecting info from up2k databases and the filesystem

  • outputs a standalone html file
  • optional mapping from IP-addresses to nicknames

partyfuse.py

  • mount a copyparty server as a local filesystem (read-only)
  • supports Windows! -- expect 194 MiB/s sequential read
  • supports Linux -- expect 117 MiB/s sequential read
  • supports macos -- expect 85 MiB/s sequential read

filecache is default-on for windows and macos;

  • macos readsize is 64kB, so speed ~32 MiB/s without the cache
  • windows readsize varies by software; explorer=1M, pv=32k

note that copyparty should run with -ed to enable dotfiles (hidden otherwise)

also consider using ../docs/rclone.md instead for 5x performance

to run this on windows:

  • install winfsp and python 3
    • add python 3.x to PATH (it asks during install)
  • python -m pip install --user fusepy
  • python ./partyfuse.py n: http://192.168.1.69:3923/

10% faster in msys2, 700% faster if debug prints are enabled:

  • pacman -S mingw64/mingw-w64-x86_64-python{,-pip}
  • /mingw64/bin/python3 -m pip install --user fusepy
  • /mingw64/bin/python3 ./partyfuse.py [...]

you could replace winfsp with dokan, let me know if you figure out how
(winfsp's sshfs leaks, doesn't look like winfsp itself does, should be fine)

partyfuse2.py

  • mount a copyparty server as a local filesystem (read-only)
  • does the same thing except more correct, samba approves
  • supports Linux -- expect 18 MiB/s (wait what)
  • supports Macos -- probably

partyfuse-streaming.py

  • pretend this doesn't exist

mtag/

  • standalone programs which perform misc. file analysis
  • copyparty can Popen programs like these during file indexing to collect additional metadata

dbtool.py

upgrade utility which can show db info and help transfer data between databases, for example when a new version of copyparty is incompatible with the old DB and automatically rebuilds the DB from scratch, but you have some really expensive -mtp parsers and want to copy over the tags from the old db

for that example (upgrading to v0.11.20), first launch the new version of copyparty like usual, let it make a backup of the old db and rebuild the new db until the point where it starts running mtp (colored messages as it adds the mtp tags), that's when you hit CTRL-C and patch in the old mtp tags from the old db instead

so assuming you have -mtp parsers to provide the tags key and .bpm:

cd /mnt/nas/music/.hist
~/src/copyparty/bin/dbtool.py -ls up2k.db
~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -cmp
~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -rm-mtp-flag -copy key
~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -rm-mtp-flag -copy .bpm -vac

prisonparty.sh

  • run copyparty in a chroot, preventing any accidental file access
  • creates bindmounts for /bin, /lib, and so on, see sysdirs=