Commit graph

263 commits

Author SHA1 Message Date
ed cacec9c1f3 support copying files/folders; closes #115
behaves according to the target volume's deduplication config;
will create symlinks / hardlinks instead if dedup is enabled
2024-11-07 21:41:53 +00:00
ed 119e88d87b bubble OS-filesystem errors to client
send a 500 or 404 if a folder is inaccessible or does not exist

previously it would return an empty directory listing instead
2024-11-02 17:38:17 +00:00
ed 833c6cf2ec partyfuse: bump dircache size
dircache size should exceed max dir depth, because the OS
may periodically listdir all parents of current folder
2024-10-26 18:25:21 +00:00
ed a9b4436cdc up2k: improve upload retry/timeout
* `js:` make handshake retries more aggressive
* `u2c:` reduce chunks timeout + ^
* `main:` reduce tcp timeout to 128sec (js is 42s)
* `httpcli:` less confusing log messages
2024-10-18 16:24:31 +00:00
ed fc8298c468 up2k: avoid cloudflare upload size-limit
previously, the biggest file that could be uploaded through
cloudflare was 383 GiB, due to max num chunks being 4096

`--u2sz`, which takes three ints (min-size, target, max-size)
can now be used to enforce a max chunksize; chunks larger
than max-size gets split into smaller subchunks / chunklets

subchunks cannot be stitched/joined, and subchunks of the
same chunk must be uploaded sequentially, one at a time

if a subchunk fails due to bitflips or connection-loss,
then the entire chunk must (and will) be reuploaded
2024-10-16 19:29:08 +00:00
ed b2770a2087 u2c: support more crazy filenames
newlines, invalid utf8, and worst of all... %20 (whitespace)

due to up2k protocol limitations,
filenames are normalized when they hit the server,
but folders get to keep their intended jank
2024-10-15 23:01:07 +00:00
ed cba1878bb2 u2c: don't get stuck at fifos and such 2024-10-15 22:53:55 +00:00
ed a2e037d6af u2c: fix chunksize calculation
files which were exactly 128 GiB large would fail
(you can't make this shit up)
2024-10-15 22:39:48 +00:00
ed 65a2b6a223 u2c: fix excessive FDs
it would open separate FDs for all chunks to be uploaded...

open and close files as they are needed during upload instead
2024-10-15 22:30:15 +00:00
ed 414de88925 u2c v2.2 2024-10-13 22:07:41 +00:00
ed 53ffd245dd u2c: fix progress indicator for resumed uploads 2024-10-13 22:07:07 +00:00
ed cf1b756206 u2c: option to list chunk hashes 2024-10-13 22:06:02 +00:00
ed 1ff14b4e05 optimizations, failsafes, formatting 2024-10-02 21:59:53 +00:00
ed c81e898435 partyfuse: also support mounting nginx, iis
these additional parsers are not included in the sfx-embedded
copy of partyfuse.py; grab it from github when necessary
2024-10-01 22:37:07 +00:00
ed f1646b96ca dist: strip some pointless code 2024-10-01 18:35:36 +00:00
ed 847a2bdc85 partyfuse: bump datacache chunksize
previous approach:
* cache 64K on first read
* cache 1M on subsequent intersecting reads

new approach:
* cache 64K on first read
* cache 1M on the next intersecting read
* cache 8M on subsequent intersecting reads
* cache 4M on standalone reads at offsets >1M

improves performance by 50% on windows
and should help on high-latency connections
2024-10-01 17:15:35 +00:00
ed 03f0f99469 partyfuse: fix extremely slow dircache lookups
the cache was a list of files instead of a dict... dude

also adds a max-num dircache limit
in addition to the expiration time
2024-10-01 17:07:28 +00:00
ed 3900e66158 partyfuse: modernize html parser (just in case) 2024-10-01 17:00:17 +00:00
ed 3dff6cda40 partyfuse: normalize naming in parsers 2024-10-01 16:55:00 +00:00
ed 73d05095b5 partyfuse: misc correctness;
* support more unix envs with granular fuse config
* generated URLs were OK but technically incorrect
2024-10-01 16:49:39 +00:00
ed 8b942ea237 partyfuse: cleanup logging and exceptions
windows runs 50% faster with recentlog on infos too...
2024-09-29 23:19:33 +00:00
ed b9c5c7bbde u2c: early exclude + fix py2 perf
* don't traverse into excluded folders
* 3.5x faster on python2.7
2024-09-23 17:20:04 +00:00
ed 9daeed923f u2c: remove all deps to become 3x faster on small files
reduces performance on python 2.7, but that is ok

also fixes `unknown encoding: idna` due to cpython race
2024-09-22 18:07:36 +00:00
ed 08848be784 u2c: add hashgen mode + fix shutdown lag 2024-09-06 00:31:25 +00:00
ed f356faa278 u2c: support multiple exclusion patterns 2024-08-22 20:03:25 +00:00
ed c8f4aeaefa hook/reloc: fix up2k jank
* wark landed in the wrong registry when moved to another volume
   (harmless; upload would succeed on the next handshake)

* dedup did not apply correctly when moved into another volume,
   since all the checks were done based on the previous vol;
   fix this by recursing the whole thing

also update the reloc example after some real-world experience

Reported-by: @daniiooo
2024-08-15 19:26:06 +00:00
ed f709140571 hook/reloc: helptext mentioned jank that doesn't exist anymore 2024-08-11 15:07:21 +00:00
ed ef1c4b7a20 this guy didn't make it in 2024-08-11 14:55:51 +00:00
ed 6c94a63f1c add hook side-effects; closes #86
hooks can now interrupt or redirect actions, and initiate
related actions, by printing json on stdout with commands

mainly to mitigate limitations such as sharex/sharex#3992

xbr/xau can redirect uploads to other destinations with `reloc`
and most hooks can initiate indexing or deletion of additional
files by giving a list of vpaths in json-keys `idx` or `del`

there are limitations;
* xbu/xau effects don't apply to ftp, tftp, smb
* xau will intentionally fail if a reloc destination exists
* xau effects do not apply to up2k

also provides more details for hooks:
* xbu/xau: basic-uploader vpath with filename
* xbr/xar: add client ip
2024-08-11 14:52:32 +00:00
ed 0da719f4c2 up2k: shrink request headers
v1.13.5 made some proxies angry with its massive chunklists

when stitching chunks, only list the first chunk hash in full,
and include a truncated hash for the consecutive chunks

should be enough for logfiles to make sense
and to smoketest that clients are behaving
2024-08-08 18:24:18 +00:00
ed 8916bce306 u2c fixes:
* `--sz` was num.chunks, not the intended MiB
* crash on exit with `-z` and no modified files
* summary upload elapsed-time could exceed wallclock
2024-07-26 19:28:47 +00:00
ed 99edba4fd9 change xm examples to reject users without write-access; #68 2024-07-25 19:23:08 +00:00
ed 987bce2182 u2c fixes:
* don't stitch across deduplicated blocks
* print speed/time for hash/upload
* more compact json in handshakes
2024-07-22 20:55:32 +00:00
ed 132a83501e add chunk stitching; twice as fast long-distance uploads:
rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East,  `1.6x` from US-West to Finland )

the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body

a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value

this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies

the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes

a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case

on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
2024-07-21 23:35:37 +00:00
ed d26a944d95 hooks: add cache-warmer 2024-07-17 21:00:59 +00:00
ed 803e156509 hooks: improve torrent downloader 2024-07-14 17:57:36 +00:00
ed bd3b3863ae hooks: add bittorrent downloader 2024-07-13 01:37:17 +02:00
ed ffd4f9c8b9 hooks: describe examples better 2024-07-13 01:32:26 +02:00
ed bef9617638 u2c.exe: explain that https is disabled 2024-06-01 22:26:47 +00:00
ed 87c60a1ec9 ensure OS signals hit main-thread as intended;
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads

also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
2024-05-09 22:28:16 +00:00
ed 439cb7f85b u2c: add --ow (previously part of --dr) 2024-04-20 16:36:10 +00:00
ed f1e947f37d rehost deps from a flaky server 2024-04-12 21:49:01 +00:00
ed 32553e4520 fix building mtp deps on python 3.12 2024-03-16 13:59:08 +00:00
ed fd552842d4 fix other possible division-by-zeros;
u2c: also fix exe detection
2024-02-17 23:19:11 +00:00
ed b2e233403d u2c: apply exclude-filter to deletion too
if a file gets synced and you later add an exclude-filter for it,
delete the file from the server as if it doesn't exist locally
2024-01-27 18:49:25 +00:00
ed 55eb692134 up2k: add option to touch existing files to match local 2024-01-24 20:36:41 +00:00
ed 86419b8f47 suboptimizations and some future safeguards 2024-01-10 23:20:42 +01:00
ed b4031e8d43 forgot to bump this... oh well, at least the exe is correct 2023-12-08 02:16:40 +00:00
ed 2077dca66f u2c: when deleting from server, heed request size limit 2023-12-08 00:54:57 +00:00
ed 8c52b88767 make linters happier 2023-11-30 17:33:07 +00:00