rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East, `1.6x` from US-West to Finland )
the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body
a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value
this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies
the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes
a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case
on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads
also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
* when bind-mounting, resolve any symlinks ($v/) and read target inode;
for example merged /bin and /usr/bin
* add failsafe in case this test should break in new exciting ways;
inspect `mount` for any instances of the jailed path
(not /proc/mounts since that has funny space encoding)
* unmount in a while-loop because xargs freaks out if one of them fail
* and systemd doesn't give us a /dev/stderr to write to anyways
not even the deprecationwarning that got silently generated burning
20~30% of all CPU-time without actually displaying it anywhere, nice
python 3.12.0 is now only 5% slower than 3.11.6
also fixes some other, less-performance-fatal deprecations
* mention cloudflare-specific nginx config
versus.md:
* seafile has a size limit on zip downloads
* seafile and nextcloud are slow at uploading many small files
u2c: improve error message in funky environments
these are all the protocols that are currently supported by wget,
so this has no practical effect aside from making sure we won't
suddenly get file:// support or something (which would be bad)
mainly motivated by u2cli's folder syncing in turbo mode
which would un-turbo on most dupes due to wrong lastmod
disabled by default for regular http listings
(to avoid confusion in most regular usecases),
enable per-request with urlparam lt
enabled by default for single-level webdav listings
(because rclone hits the same issue as u2cli),
can be disabled with arg --dav-rt or volflag davrt
impossible to enable for recursive webdav listings
