socket.accept() can fail silently --
this would crash the worker-pool and also produce
a confusing useless error-message while doing so
reported by someone on a mac with Little Snitch:
uv python install cpython-3.13.3-macos-aarch64-none
uv python pin cpython-3.13.3-macos-aarch64-none
uv sync
uv run copyparty
...but was also observed on x86_64 linux with
python 2.7 in 2018 (no longer reproduces)
fix this to log what's going on and also don't crash
in case someone writes a plugin which
expects certain params to be sanitized
note that because mojibake filenames are supported,
URLs and filepaths can still be absolutely bonkers
this fixes one known issue:
invalid rss-feed xml if ?pw contains special chars
...and somehow things now run 2% faster, idgi
previously, the biggest file that could be uploaded through
cloudflare was 383 GiB, due to max num chunks being 4096
`--u2sz`, which takes three ints (min-size, target, max-size)
can now be used to enforce a max chunksize; chunks larger
than max-size gets split into smaller subchunks / chunklets
subchunks cannot be stitched/joined, and subchunks of the
same chunk must be uploaded sequentially, one at a time
if a subchunk fails due to bitflips or connection-loss,
then the entire chunk must (and will) be reuploaded
* pyz: yeet the resource tar which is now pointless thanks to pkgres
* cache impresource stuff because pyz lookups are Extremely slow
* prefer tx_file when possible for slightly better performance
* use hardcoded list of expected resources instead of dynamic
discovery at runtime; much simpler and probably safer
* fix some forgotten resources (copying.txt, insecure.pem)
* fix loading jinja templates on windows
add support for reading webdeps and jinja-templates using either
importlib_resources or pkg_resources, which removes the need for
extracting these to a temporary folder on the filesystem
* util: add helper functions to abstract embedded resource access
* http*: serve embedded resources through resource abstraction
* main: check webdeps through resource abstraction
* httpconn: remove unused method `respath(name)`
* use __package__ to find package resources
* util: use importlib_resources backport if available
* pass E.pkg as module object for importlib_resources compatibility
* util: add pkg_resources compatibility to resource abstraction
* support x-forwarded-for
* option to specify socket permissions and group
* in containers, avoid collision during restart
* add --help-bind with examples
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads
also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
some reverse-proxies expect plaintext replies, and
we don't have a brotli decompressor to satisfy this
additionally, because brotli is https-gated (thx google),
it was already an impractical mess anyways
the sfx is now 7 KiB larger
as each chunk is written to the file, httpcli calls
up2k.confirm_chunk to register the chunk as completed, and the reply
indicates whether that was the final outstanding chunk, in which case
httpcli closes the file descriptors since there's nothing more to write
the issue is that the final chunk is registered as completed before the
file descriptors are closed, meaning there could be writes that haven't
finished flushing to disk yet
if the client decides to issue another handshake during this window,
up2k sees that all chunks are complete and calls up2k.finish_upload
even as some threads might still be flushing the final writes to disk
so the conditions to hit this bug were as follows (all must be true):
* multiprocessing is disabled
* there is a reverse-proxy
* a client has several idle connections and reuses one of those
* the server's filesystem is EXTREMELY slow, to the point where
closing a file takes over 30 seconds
the fix is to stop handshakes from being processed while a file is
being closed, which is unfortunately a small bottleneck in that it
prohibits initiating another upload while one is being finalized, but
the required complexity to handle this better is probably not worth it
(a separate mutex for each upload session or something like that)
this issue is mostly harmless, partially because it is super tricky to
hit (only aware of it happening synthetically), and because there is
usually no harmful consequences; the worst-case is if this were to
happen exactly as the server OS decides to crash, which would make the
file appear to be fully uploaded even though it's missing some data
(all extremely unlikely, but not impossible)
there is no performance impact; if anything it should now accept
new tcp connections slightly faster thanks to more granular locking
* start banning malicious clients according to --ban-422
* reply with a blank 500 to stop firefox from retrying like 20 times
* allow Cc's in a few specific URL params (filenames, dirnames)
* cpp_uptime is now a gauge
* cpp_bans is now cpp_active_bans (and also a gauge)
and other related fixes:
* stop emitting invalid cpp_disk_size/free for offline volumes
* support overriding the spec-mandatory mimetype with ?mime=foo
* --ban-url: URLs which 404 and also match --sus-urls (bot-scan)
* --ban-403: trying to access volumes that dont exist or require auth
* --ban-422: invalid POST messages, fuzzing and such
* --nonsus-urls: regex of 404s which shouldn't trigger --ban-404
in may situations it makes sense to handle this logic inside copyparty,
since stuff like cloudflare and running copyparty on another physical
box than the nginx frontend is on becomes fairly clunky
it was possible to set cookie values which contained newlines,
thus terminating the http header and bleeding into the body.
We now disallow control-characters in queries,
but still allow them in paths, as copyparty supports
filenames containing newlines and other mojibake.
The changes in `set_k304` are not necessary in fixing the vulnerability,
but makes the behavior more correct.
* error-message which explains how to run on py2 / older py3
when trying to run from source
* check compatibility between jinja2 and cpython on startup
* verify that webdeps are present on startup
* verify that webdeps are present when building sfx
* make-sfx.sh grabs the strip-hints dependency
prevents running out of FDs thanks to thousands of sqlite3 sessions
and neatly sidesteps what could possibly be a race in python's
sqlite3 bindings where it sometimes forgets to close the fd
