mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2026-06-21 05:32:25 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity
4646 commits 4 branches 380 tags 94 MiB
Commit graph

332 commits

Author SHA1 Message Date
ed 99edba4fd9 change xm examples to reject users without write-access; #68 2024-07-25 19:23:08 +00:00
ed 987bce2182 u2c fixes: 
* don't stitch across deduplicated blocks
* print speed/time for hash/upload
* more compact json in handshakes
 2024-07-22 20:55:32 +00:00
ed 132a83501e add chunk stitching; twice as fast long-distance uploads: 
rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East,  `1.6x` from US-West to Finland )

the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body

a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value

this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies

the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes

a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case

on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
 2024-07-21 23:35:37 +00:00
ed d26a944d95 hooks: add cache-warmer 2024-07-17 21:00:59 +00:00
ed 803e156509 hooks: improve torrent downloader 2024-07-14 17:57:36 +00:00
ed bd3b3863ae hooks: add bittorrent downloader 2024-07-13 01:37:17 +02:00
ed ffd4f9c8b9 hooks: describe examples better 2024-07-13 01:32:26 +02:00
ed bef9617638 u2c.exe: explain that https is disabled 2024-06-01 22:26:47 +00:00
ed 87c60a1ec9 ensure OS signals hit main-thread as intended; 
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads

also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
 2024-05-09 22:28:16 +00:00
ed 439cb7f85b u2c: add --ow (previously part of --dr) 2024-04-20 16:36:10 +00:00
ed f1e947f37d rehost deps from a flaky server 2024-04-12 21:49:01 +00:00
ed 32553e4520 fix building mtp deps on python 3.12 2024-03-16 13:59:08 +00:00
ed fd552842d4 fix other possible division-by-zeros; 
u2c: also fix exe detection
 2024-02-17 23:19:11 +00:00
ed b2e233403d u2c: apply exclude-filter to deletion too 
if a file gets synced and you later add an exclude-filter for it,
delete the file from the server as if it doesn't exist locally
 2024-01-27 18:49:25 +00:00
ed 55eb692134 up2k: add option to touch existing files to match local 2024-01-24 20:36:41 +00:00
ed 86419b8f47 suboptimizations and some future safeguards 2024-01-10 23:20:42 +01:00
ed b4031e8d43 forgot to bump this... oh well, at least the exe is correct 2023-12-08 02:16:40 +00:00
ed 2077dca66f u2c: when deleting from server, heed request size limit 2023-12-08 00:54:57 +00:00
ed 8c52b88767 make linters happier 2023-11-30 17:33:07 +00:00
ed 72e355eb2c prisonparty: prevent overlapping setup/teardown 2023-11-25 14:03:41 +00:00
ed 43d409a5d9 prisonparty accepts user/group names 2023-11-25 13:40:21 +00:00
ed edd3e53ab3 prisonparty: support zfs-ubuntu 
* when bind-mounting, resolve any symlinks ($v/) and read target inode;
   for example merged /bin and /usr/bin
* add failsafe in case this test should break in new exciting ways;
   inspect `mount` for any instances of the jailed path
   (not /proc/mounts since that has funny space encoding)
* unmount in a while-loop because xargs freaks out if one of them fail
   * and systemd doesn't give us a /dev/stderr to write to anyways
 2023-11-25 02:16:48 +00:00
ed c6744dc483 u2c: configurable retry delay 2023-11-11 14:46:00 +00:00
ed b9997d677d u2c: give up on files with bitflips 2023-11-11 14:30:46 +00:00
ed 10defe6aef u2c: make -x case-insensitive 2023-11-11 14:02:01 +00:00
ed fc658e5b9e utcfromtimestamp was deprecated and nobody told me, 
not even the deprecationwarning that got silently generated burning
20~30% of all CPU-time without actually displaying it anywhere, nice

python 3.12.0 is now only 5% slower than 3.11.6

also fixes some other, less-performance-fatal deprecations
 2023-10-20 23:41:58 +00:00
ed fcc3336760 v1.9.8 2023-10-06 17:50:35 +00:00
ed 01a851da28 mtp-deps: fix building on archlinux 2023-09-24 23:17:26 +00:00
ed e76a50cb9d add indexer benchmark + bump default num cores from 4 to 5 
and make the mtag deps build better on fedora
 2023-09-22 20:40:52 +00:00
ed 4fbd6853f4 add msg-log.py initially by @clach04, closes #35 2023-09-12 19:56:05 +00:00
ed 14497b2425 docs: 
* mention cloudflare-specific nginx config

versus.md:
* seafile has a size limit on zip downloads
* seafile and nextcloud are slow at uploading many small files

u2c: improve error message in funky environments
 2023-08-25 21:57:26 +00:00
ed 7df890d964 wget: only allow http/https/ftp/ftps (#50): 
these are all the protocols that are currently supported by wget,
so this has no practical effect aside from making sure we won't
suddenly get file:// support or something (which would be bad)
 2023-08-20 09:47:50 +00:00
ed 033b50ae1b u2c: exclude files by regex 2023-08-15 00:45:12 +02:00
ed 5d8cb34885 404/403 can be handled with plugins 2023-07-07 21:33:40 +00:00
ed e767230383 very-bad-idea: prefer mpv / streamlink; closes #42 2023-06-28 21:25:40 +00:00
ed 90601314d6 better explain why very-bad-idea is a very bad idea 2023-06-27 22:30:14 +00:00
ed deca082623 v1.7.1 2023-05-07 18:34:39 +00:00
ed 4295923b76 rename up2k.py (client) to u2c.py 2023-05-07 15:37:52 +00:00
ed 572aa4b26c rename up2k.py (client) to u2c.py 2023-05-07 15:35:56 +00:00
ed a84c583b2c ok that wasn't enough 2023-04-27 19:06:35 +00:00
ed d3ccd3f174 v1.6.15 2023-04-26 23:00:55 +00:00
ed 673b4f7e23 option to show symlink's lastmod instead of deref; 
mainly motivated by u2cli's folder syncing in turbo mode
which would un-turbo on most dupes due to wrong lastmod

disabled by default for regular http listings
(to avoid confusion in most regular usecases),
enable per-request with urlparam lt

enabled by default for single-level webdav listings
(because rclone hits the same issue as u2cli),
can be disabled with arg --dav-rt or volflag davrt

impossible to enable for recursive webdav listings
 2023-04-26 18:54:21 +00:00
ed d11e02da49 u2cli: avoid dns lookups while uploading 2023-04-26 18:46:42 +00:00
ed 2459965ca8 u2cli: dont enter delete stage if something failed 2023-04-20 20:40:09 +00:00
ed d6a0a738ce add windows example + update docs + some cosmetics 2023-04-12 22:06:44 +00:00
ed e7f3e475a2 more accurate bpm detector 2023-03-31 21:20:37 +00:00
ed 5f4ea27586 new hook: exif stripper 2023-03-26 22:19:15 +00:00
ed 099cc97247 hooks: more correct usage examples 2023-03-26 22:18:48 +00:00
ed 4cbffec0ec u2cli: show more errors + drop --ws (does nothing) 2023-03-23 23:47:41 +00:00
ed 5485c6d7ca prisonparty: FFmpeg runs faster with /dev/urandom 2023-03-19 18:32:35 +00:00
ed b0cc396bca v1.6.8 2023-03-12 16:10:07 +00:00
ed ae463518f6 u2cli: send upload stats to server + fix py2.6 support 2023-03-11 21:39:56 +00:00
ed bf04e7c089 update some docs 2023-03-09 22:11:39 +00:00
ed b39ff92f34 u2cli: support long paths on win7 2023-03-08 22:27:13 +00:00
ed 5ceb3a962f build up2k.exe 2023-03-07 22:58:14 +00:00
ed c98fff1647 fix chunkpost-handshake race (affects --no-dedup only); 
a handshake arriving in the middle of the final chunk could cause
dupes to become empty -- worst case leading to loss of data
 2023-03-05 19:45:50 +00:00
ed b179e2b031 prisonparty: ignore unresolvable mount paths; 
allows startup even if some locations are missing,
for example if a server rebooted and some disks aren't up yet
 2023-02-26 22:11:15 +00:00
ed 5cb2e33353 update readmes + fix typo 2023-02-26 19:22:54 +00:00
ed 05e0c2ec9e add xiu (batching hook; runs on idle after uploads) + 
bunch of tweaks/fixes for hooks
 2023-02-26 18:23:32 +00:00
ed 3e561976db optimize docker build times (884 to 379 sec) 2023-02-19 14:19:35 +00:00
ed 1526a4e084 add docker packaging 2023-02-10 23:02:01 +00:00
ed a4b56c74c7 support long filepaths on win7 + misc windows fixes 2023-02-10 18:37:37 +00:00
ed 76e5eeea3f prisonparty: fix reload signal 2023-02-05 00:00:18 +00:00
ed 78d2f69ed5 prisonparty: support opus transcoding on debian 
libblas.so and liblapack.so are symlinks into /etc/alternatives
 2023-01-31 20:50:59 +00:00
ed 2f15bb5085 include filesize in notification 2023-01-31 19:03:13 +00:00
ed 87ad23fb93 docs + chmod 2023-01-29 18:28:53 +00:00
ed 5f54d534e3 hook/notify: add android support 2023-01-29 15:14:22 +00:00
ed 0be1e43451 mention mtp in the hooks readme 2023-01-28 00:07:50 +00:00
ed bcd91b1809 add eventhook examples 2023-01-27 23:55:57 +00:00
ed 460709e6f3 upgrade wget downloader to use event hooks 2023-01-22 23:45:11 +00:00
ed f8e3e87a52 add event hooks 2023-01-22 23:35:31 +00:00
ed ee46f832b1 u2cli: add option -ns for slow terminals 2023-01-17 23:29:51 +00:00
ed dd7fab1352 u2cli: properly retry failed handshakes 2023-01-13 07:17:41 +00:00
ed b4c7282956 password from file 2022-12-20 13:28:48 +00:00
ed 28232656a9 folder-sync optimizations 2022-12-13 18:56:40 +00:00
ed 447ed5ab37 windows fixes 2022-12-12 21:59:50 +00:00
ed af59808611 u2cli: always compare toplevel in syncs 2022-12-12 07:16:05 +01:00
ed e3406a9f86 dont cls by default 2022-12-11 22:46:21 +00:00
ed 138c2bbcbb o no 2022-12-11 18:30:29 +00:00
ed bc7af1d1c8 u2cli: add basic folder sync 2022-12-11 17:41:10 +00:00
ed 0cef062084 misc cleanup 2022-12-01 21:44:31 +00:00
ed 3a800585bc u2cli: server is allowed to reject dupes 2022-11-29 22:09:32 +00:00
ed f39e370e2a cosmetic 2022-11-26 22:27:09 +00:00
ed 6c962ec7d3 rename copyparty-fuse to partyfuse 2022-11-26 20:01:20 +00:00
ed 6bc1bc542f rename copyparty-fuse to partyfuse 2022-11-26 19:53:41 +00:00
ed 2170ee8da4 improve scheduling 2022-11-04 20:28:05 +00:00
ed 89d1f52235 cursory slowloris / buggy-webdav-client detector 2022-11-01 22:18:20 +00:00
ed 54e1c8d261 remove 697 GiB upload filesize limit 2022-10-30 12:51:20 +00:00
ed 0063021012 mtp-deps: add fedora support 2022-10-29 21:38:08 +00:00
ed 4bcd30da6b cleaner daemon instancing 2022-10-23 12:05:44 +02:00
ed 1c66d06702 cleanup versionchecks 2022-09-25 21:31:47 +02:00
ed 6db72d7166 optimizations / cleanup 2022-09-15 01:18:19 +02:00
ed 590c763659 add unforgetti beam 2022-09-07 08:09:32 +02:00
ed 20dca2bea5 mtp: add guestbook reader 2022-09-05 20:23:59 +02:00
ed 059e93cdcf u2cli: fix py3.5 support + better deps warning 2022-09-05 18:24:18 +02:00
ed 788db47b95 option to let mtp's keep stdout/stderr 2022-09-04 01:42:28 +02:00
ed 89d1af7f33 this actually serves a purpose but please dont ask 2022-09-03 20:19:16 +02:00
ed 5d19f23372 accurate num.cores detection 2022-08-29 19:24:48 +02:00
ed 993213e2c0 mtp/vidchk: support stuff like rag-prep 2022-08-24 23:25:03 +02:00
ed 49470c05fa well that was dumb 2022-08-23 00:03:04 +02:00
ed d48a7d2398 provide tagparsers with uploader info 2022-08-15 20:23:17 +02:00
ed 8ea1f4a751 idx multimedia format/container type 2022-08-15 17:56:13 +02:00
ed 1c69ccc6cd v1.3.12 2022-08-13 00:58:49 +02:00
ed 84b5bbd3b6 u2cli: bail from recursive symlinks + verbose errors 2022-08-13 00:28:08 +02:00
ed 92ed4ba3f8 parallelize python hashing too 2022-08-10 23:12:01 +02:00
ed 4735fb1ebb u2cli: better msg on bad tls certs 2022-08-09 00:11:34 +02:00
ed 680f8ae814 add xdev/xvol indexing guards 2022-08-03 22:20:28 +02:00
ed 10430b347f fix dumb prisonparty bug 2022-07-22 20:49:35 +02:00
ed 48b957f1d5 add -e2v (file integrity checker) 2022-07-13 00:48:39 +02:00
ed 095bd17d10 mtp/vidchk: grab some frames at the start too 2022-07-09 13:10:00 +02:00
ed 204bfac3fa mtp/vidchk: write ffprobe metadata to file 2022-07-09 04:33:19 +02:00
ed ac49b0ca93 mtp: add rclone uploader 2022-07-08 23:47:27 +02:00
ed c5b04f6fef mtp daisychaining 2022-07-08 22:29:05 +02:00
ed 0862860961 misc cleanup 2022-07-06 00:00:56 +02:00
ed 622358b172 flag to control mtp timeout kill behavior 2022-07-05 23:38:49 +02:00
ed 87bb49da36 new mtp: video integrity checker 2022-07-03 01:50:38 +02:00
ed 9ad3af1ef6 misc tweaks 2022-06-19 16:05:48 +02:00
ed 728dc62d0b optimize nonsparse uploads (fat32, exfat, hpfs) 2022-06-16 17:51:42 +02:00
ed 438384425a add types, isort, errorhandling 2022-06-16 01:07:15 +02:00
ed fe73f2d579 cleanup 2022-06-07 23:08:43 +02:00
ed 4c4b3790c7 fix read-spin on d/c during json post + errorhandling 2022-06-07 19:02:52 +02:00
ed 715401ca8e fix timezone in search, zipfiles, fuse 2022-05-15 13:51:44 +02:00
ed 49e962a7dc dbtool: faster, add examples, 
match on hashes rather than paths by default,
add no-clobber option to keep existing tags
 2022-05-14 12:44:05 +02:00
ed c9c4aac6cf v1.2.1 2022-03-03 01:26:29 +01:00
ed c032cd08b3 prisonparty: clean exit on sigterm/int 2022-02-27 20:07:28 +01:00
ed ee9585264e deal with github api change + build vamp if necessary 2022-01-17 00:27:23 +00:00
ed c9ffead7bf prisonparty: support running from src 2022-01-17 00:24:40 +00:00
ed fc9dd5d743 meadup changes 2022-01-03 01:16:27 +01:00
ed 9e6d5dd2b9 vbi: add onscreen qrcode 2021-12-28 20:57:11 +01:00
ed bdad197e2c make it even worse 2021-12-27 00:04:38 +01:00
ed 7e139288a6 add very bad idea 2021-12-26 23:32:46 +01:00
ed b2de1459b6 quick backports to the alternative fuse client 2021-12-10 01:59:45 +01:00
ed f6c7e49eb8 u2cli: better error messages 2021-11-28 03:38:57 +01:00
ed 184bbdd23d legalese rephrasing 2021-11-20 17:58:37 +01:00
ed c3c3d81db1 add mtp plugin for exif stripping 2021-11-20 17:45:56 +01:00
ed 49b7231283 fix mojibake support in misc mtp plugins 2021-11-20 17:33:24 +01:00
ed d01bb69a9c u2cli: option to ignore inaccessible files 2021-11-16 21:53:00 +01:00
ed c26fb43ced more cleanup 2021-11-12 22:30:23 +01:00
ed b5b8a2c9d5 why are there https warnings when https checking is disabled 2021-10-31 03:37:31 +01:00
ed cd5c56e601 u2cli: orz 2021-10-29 01:49:40 +02:00
ed 1faff2a37e u2cli: aggressive flushing on windows 2021-10-18 20:35:50 +02:00
ed 82fe45ac56 u2cli: add -z / yolo 2021-10-13 00:03:49 +02:00
ed bcb7fcda6b u2cli: rsync-like source semantics 2021-10-12 22:46:33 +02:00
ed 734e9d3874 v1.0.8 2021-10-04 22:50:06 +02:00
ed ff32ec515e add mtp plugin cksum.py 2021-10-03 19:35:20 +02:00
ed 63eb08ba9f u2cli: nobody asked for python2.6 support so here you go w 2021-10-02 00:36:41 +02:00
ed e5b67d2b3a u2cli: add eta, errorhandling, better windows support 2021-10-01 22:31:24 +02:00
ed 9a16639a61 u2cli: add webm 2021-10-01 02:25:22 +02:00
ed 57953c68c6 u2cli: add vt100 status panel 2021-10-01 02:10:03 +02:00
ed 088d08963f u2cli: add multithreading 2021-10-01 00:33:45 +02:00