mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2025-08-17 17:12:13 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity
3613 commits 4 branches 344 tags 55 MiB
Commit graph

286 commits

Author SHA1 Message Date
ed ef1c4b7a20 this guy didn't make it in 2024-08-11 14:55:51 +00:00
ed 6c94a63f1c add hook side-effects; closes #86 
hooks can now interrupt or redirect actions, and initiate
related actions, by printing json on stdout with commands

mainly to mitigate limitations such as sharex/sharex#3992

xbr/xau can redirect uploads to other destinations with `reloc`
and most hooks can initiate indexing or deletion of additional
files by giving a list of vpaths in json-keys `idx` or `del`

there are limitations;
* xbu/xau effects don't apply to ftp, tftp, smb
* xau will intentionally fail if a reloc destination exists
* xau effects do not apply to up2k

also provides more details for hooks:
* xbu/xau: basic-uploader vpath with filename
* xbr/xar: add client ip
 2024-08-11 14:52:32 +00:00
ed 0da719f4c2 up2k: shrink request headers 
v1.13.5 made some proxies angry with its massive chunklists

when stitching chunks, only list the first chunk hash in full,
and include a truncated hash for the consecutive chunks

should be enough for logfiles to make sense
and to smoketest that clients are behaving
 2024-08-08 18:24:18 +00:00
ed 8916bce306 u2c fixes: 
* `--sz` was num.chunks, not the intended MiB
* crash on exit with `-z` and no modified files
* summary upload elapsed-time could exceed wallclock
 2024-07-26 19:28:47 +00:00
ed 99edba4fd9 change xm examples to reject users without write-access; #68 2024-07-25 19:23:08 +00:00
ed 987bce2182 u2c fixes: 
* don't stitch across deduplicated blocks
* print speed/time for hash/upload
* more compact json in handshakes
 2024-07-22 20:55:32 +00:00
ed 132a83501e add chunk stitching; twice as fast long-distance uploads: 
rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East,  `1.6x` from US-West to Finland )

the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body

a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value

this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies

the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes

a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case

on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
 2024-07-21 23:35:37 +00:00
ed d26a944d95 hooks: add cache-warmer 2024-07-17 21:00:59 +00:00
ed 803e156509 hooks: improve torrent downloader 2024-07-14 17:57:36 +00:00
ed bd3b3863ae hooks: add bittorrent downloader 2024-07-13 01:37:17 +02:00
ed ffd4f9c8b9 hooks: describe examples better 2024-07-13 01:32:26 +02:00
ed bef9617638 u2c.exe: explain that https is disabled 2024-06-01 22:26:47 +00:00
ed 87c60a1ec9 ensure OS signals hit main-thread as intended; 
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads

also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
 2024-05-09 22:28:16 +00:00
ed 439cb7f85b u2c: add --ow (previously part of --dr) 2024-04-20 16:36:10 +00:00
ed f1e947f37d rehost deps from a flaky server 2024-04-12 21:49:01 +00:00
ed 32553e4520 fix building mtp deps on python 3.12 2024-03-16 13:59:08 +00:00
ed fd552842d4 fix other possible division-by-zeros; 
u2c: also fix exe detection
 2024-02-17 23:19:11 +00:00
ed b2e233403d u2c: apply exclude-filter to deletion too 
if a file gets synced and you later add an exclude-filter for it,
delete the file from the server as if it doesn't exist locally
 2024-01-27 18:49:25 +00:00
ed 55eb692134 up2k: add option to touch existing files to match local 2024-01-24 20:36:41 +00:00
ed 86419b8f47 suboptimizations and some future safeguards 2024-01-10 23:20:42 +01:00
ed b4031e8d43 forgot to bump this... oh well, at least the exe is correct 2023-12-08 02:16:40 +00:00
ed 2077dca66f u2c: when deleting from server, heed request size limit 2023-12-08 00:54:57 +00:00
ed 8c52b88767 make linters happier 2023-11-30 17:33:07 +00:00
ed 72e355eb2c prisonparty: prevent overlapping setup/teardown 2023-11-25 14:03:41 +00:00
ed 43d409a5d9 prisonparty accepts user/group names 2023-11-25 13:40:21 +00:00
ed edd3e53ab3 prisonparty: support zfs-ubuntu 
* when bind-mounting, resolve any symlinks ($v/) and read target inode;
   for example merged /bin and /usr/bin
* add failsafe in case this test should break in new exciting ways;
   inspect `mount` for any instances of the jailed path
   (not /proc/mounts since that has funny space encoding)
* unmount in a while-loop because xargs freaks out if one of them fail
   * and systemd doesn't give us a /dev/stderr to write to anyways
 2023-11-25 02:16:48 +00:00
ed c6744dc483 u2c: configurable retry delay 2023-11-11 14:46:00 +00:00
ed b9997d677d u2c: give up on files with bitflips 2023-11-11 14:30:46 +00:00
ed 10defe6aef u2c: make -x case-insensitive 2023-11-11 14:02:01 +00:00
ed fc658e5b9e utcfromtimestamp was deprecated and nobody told me, 
not even the deprecationwarning that got silently generated burning
20~30% of all CPU-time without actually displaying it anywhere, nice

python 3.12.0 is now only 5% slower than 3.11.6

also fixes some other, less-performance-fatal deprecations
 2023-10-20 23:41:58 +00:00
ed fcc3336760 v1.9.8 2023-10-06 17:50:35 +00:00
ed 01a851da28 mtp-deps: fix building on archlinux 2023-09-24 23:17:26 +00:00
ed e76a50cb9d add indexer benchmark + bump default num cores from 4 to 5 
and make the mtag deps build better on fedora
 2023-09-22 20:40:52 +00:00
ed 4fbd6853f4 add msg-log.py initially by @clach04, closes #35 2023-09-12 19:56:05 +00:00
ed 14497b2425 docs: 
* mention cloudflare-specific nginx config

versus.md:
* seafile has a size limit on zip downloads
* seafile and nextcloud are slow at uploading many small files

u2c: improve error message in funky environments
 2023-08-25 21:57:26 +00:00
ed 7df890d964 wget: only allow http/https/ftp/ftps (#50): 
these are all the protocols that are currently supported by wget,
so this has no practical effect aside from making sure we won't
suddenly get file:// support or something (which would be bad)
 2023-08-20 09:47:50 +00:00
ed 033b50ae1b u2c: exclude files by regex 2023-08-15 00:45:12 +02:00
ed 5d8cb34885 404/403 can be handled with plugins 2023-07-07 21:33:40 +00:00
ed e767230383 very-bad-idea: prefer mpv / streamlink; closes #42 2023-06-28 21:25:40 +00:00
ed 90601314d6 better explain why very-bad-idea is a very bad idea 2023-06-27 22:30:14 +00:00
ed deca082623 v1.7.1 2023-05-07 18:34:39 +00:00
ed 4295923b76 rename up2k.py (client) to u2c.py 2023-05-07 15:37:52 +00:00
ed 572aa4b26c rename up2k.py (client) to u2c.py 2023-05-07 15:35:56 +00:00
ed a84c583b2c ok that wasn't enough 2023-04-27 19:06:35 +00:00
ed d3ccd3f174 v1.6.15 2023-04-26 23:00:55 +00:00
ed 673b4f7e23 option to show symlink's lastmod instead of deref; 
mainly motivated by u2cli's folder syncing in turbo mode
which would un-turbo on most dupes due to wrong lastmod

disabled by default for regular http listings
(to avoid confusion in most regular usecases),
enable per-request with urlparam lt

enabled by default for single-level webdav listings
(because rclone hits the same issue as u2cli),
can be disabled with arg --dav-rt or volflag davrt

impossible to enable for recursive webdav listings
 2023-04-26 18:54:21 +00:00
ed d11e02da49 u2cli: avoid dns lookups while uploading 2023-04-26 18:46:42 +00:00
ed 2459965ca8 u2cli: dont enter delete stage if something failed 2023-04-20 20:40:09 +00:00
ed d6a0a738ce add windows example + update docs + some cosmetics 2023-04-12 22:06:44 +00:00
ed e7f3e475a2 more accurate bpm detector 2023-03-31 21:20:37 +00:00
ed 5f4ea27586 new hook: exif stripper 2023-03-26 22:19:15 +00:00
ed 099cc97247 hooks: more correct usage examples 2023-03-26 22:18:48 +00:00
ed 4cbffec0ec u2cli: show more errors + drop --ws (does nothing) 2023-03-23 23:47:41 +00:00
ed 5485c6d7ca prisonparty: FFmpeg runs faster with /dev/urandom 2023-03-19 18:32:35 +00:00
ed b0cc396bca v1.6.8 2023-03-12 16:10:07 +00:00
ed ae463518f6 u2cli: send upload stats to server + fix py2.6 support 2023-03-11 21:39:56 +00:00
ed bf04e7c089 update some docs 2023-03-09 22:11:39 +00:00
ed b39ff92f34 u2cli: support long paths on win7 2023-03-08 22:27:13 +00:00
ed 5ceb3a962f build up2k.exe 2023-03-07 22:58:14 +00:00
ed c98fff1647 fix chunkpost-handshake race (affects --no-dedup only); 
a handshake arriving in the middle of the final chunk could cause
dupes to become empty -- worst case leading to loss of data
 2023-03-05 19:45:50 +00:00
ed b179e2b031 prisonparty: ignore unresolvable mount paths; 
allows startup even if some locations are missing,
for example if a server rebooted and some disks aren't up yet
 2023-02-26 22:11:15 +00:00
ed 5cb2e33353 update readmes + fix typo 2023-02-26 19:22:54 +00:00
ed 05e0c2ec9e add xiu (batching hook; runs on idle after uploads) + 
bunch of tweaks/fixes for hooks
 2023-02-26 18:23:32 +00:00
ed 3e561976db optimize docker build times (884 to 379 sec) 2023-02-19 14:19:35 +00:00
ed 1526a4e084 add docker packaging 2023-02-10 23:02:01 +00:00
ed a4b56c74c7 support long filepaths on win7 + misc windows fixes 2023-02-10 18:37:37 +00:00
ed 76e5eeea3f prisonparty: fix reload signal 2023-02-05 00:00:18 +00:00
ed 78d2f69ed5 prisonparty: support opus transcoding on debian 
libblas.so and liblapack.so are symlinks into /etc/alternatives
 2023-01-31 20:50:59 +00:00
ed 2f15bb5085 include filesize in notification 2023-01-31 19:03:13 +00:00
ed 87ad23fb93 docs + chmod 2023-01-29 18:28:53 +00:00
ed 5f54d534e3 hook/notify: add android support 2023-01-29 15:14:22 +00:00
ed 0be1e43451 mention mtp in the hooks readme 2023-01-28 00:07:50 +00:00
ed bcd91b1809 add eventhook examples 2023-01-27 23:55:57 +00:00
ed 460709e6f3 upgrade wget downloader to use event hooks 2023-01-22 23:45:11 +00:00
ed f8e3e87a52 add event hooks 2023-01-22 23:35:31 +00:00
ed ee46f832b1 u2cli: add option -ns for slow terminals 2023-01-17 23:29:51 +00:00
ed dd7fab1352 u2cli: properly retry failed handshakes 2023-01-13 07:17:41 +00:00
ed b4c7282956 password from file 2022-12-20 13:28:48 +00:00
ed 28232656a9 folder-sync optimizations 2022-12-13 18:56:40 +00:00
ed 447ed5ab37 windows fixes 2022-12-12 21:59:50 +00:00
ed af59808611 u2cli: always compare toplevel in syncs 2022-12-12 07:16:05 +01:00
ed e3406a9f86 dont cls by default 2022-12-11 22:46:21 +00:00
ed 138c2bbcbb o no 2022-12-11 18:30:29 +00:00
ed bc7af1d1c8 u2cli: add basic folder sync 2022-12-11 17:41:10 +00:00
ed 0cef062084 misc cleanup 2022-12-01 21:44:31 +00:00
ed 3a800585bc u2cli: server is allowed to reject dupes 2022-11-29 22:09:32 +00:00
ed f39e370e2a cosmetic 2022-11-26 22:27:09 +00:00
ed 6c962ec7d3 rename copyparty-fuse to partyfuse 2022-11-26 20:01:20 +00:00
ed 6bc1bc542f rename copyparty-fuse to partyfuse 2022-11-26 19:53:41 +00:00
ed 2170ee8da4 improve scheduling 2022-11-04 20:28:05 +00:00
ed 89d1f52235 cursory slowloris / buggy-webdav-client detector 2022-11-01 22:18:20 +00:00
ed 54e1c8d261 remove 697 GiB upload filesize limit 2022-10-30 12:51:20 +00:00
ed 0063021012 mtp-deps: add fedora support 2022-10-29 21:38:08 +00:00
ed 4bcd30da6b cleaner daemon instancing 2022-10-23 12:05:44 +02:00
ed 1c66d06702 cleanup versionchecks 2022-09-25 21:31:47 +02:00
ed 6db72d7166 optimizations / cleanup 2022-09-15 01:18:19 +02:00
ed 590c763659 add unforgetti beam 2022-09-07 08:09:32 +02:00
ed 20dca2bea5 mtp: add guestbook reader 2022-09-05 20:23:59 +02:00
ed 059e93cdcf u2cli: fix py3.5 support + better deps warning 2022-09-05 18:24:18 +02:00
ed 788db47b95 option to let mtp's keep stdout/stderr 2022-09-04 01:42:28 +02:00