* hasher thread could die if a client would rapidly
upload and delete files (so very unlikely)
* two unprotected calls to register_vpath which was
almost-definitely safe because the volumes
already existed in the registry
the default (256 KiB) appears optimal in the most popular scenario
(linux host with storage on local physical disk, usually NVMe)
was previously a mix of 64 and 512 KiB;
now the same value is enforced everywhere
download-as-tar is now 20% faster with the default value
to abort an upload, refresh the page and access the unpost tab,
which now includes unfinished uploads (sorted before completed ones)
can be configured through u2abort (global or volflag);
by default it requires both the IP and account to match
https://a.ocv.me/pub/g/nerd-stuff/2024-0310-stoltzekleiven.jpg
this improves performance on s3-backed volumes
noktuas reported on discord that the upload performance was
unexpectedly poor when writing to an s3 bucket through a JuiceFS
fuse-mount, only getting 1.5 MiB/s with copyparty, meanwhile a
regular filecopy averaged 30 MiB/s plus
the issue was that s3 does not support sparse files, so copyparty
would fall back to sequential uploading, and also disable fpool,
causing JuiceFS to repeatedly commit the same 5 MiB range to
the storage provider as each chunk arrived from the client
by forcing use of sparse files, s3 adapters such as JuiceFS and
geesefs will "only" write the entire file to s3 *twice*, initially
it writes the full filesize of zerobytes (depending on adapter,
hopefully using gzip compression to reduce the bandwidth necessary)
and then the actual file data in an adapter-specific chunksize
with this volflag, copyparty appears to reach the full expected speed
nothing dangerous, just confusing log messages if an
admin hammers the reload button 100+ times per second,
or another linux process rapidly sends SIGUSR1
some cifs servers cause sqlite to fail in interesting ways; any attempt
to create a table can instantly throw an exception, which results in a
zerobyte database being created. During the next startup, the db would
be determined to be corrupted, and up2k would invoke _backup_db before
deleting and recreating it -- except that sqlite's connection.backup()
will hang indefinitely and deadlock up2k
add a watchdog which fires if it takes longer than 1 minute to open the
database, printing a big warning that the filesystem probably does not
support locking or is otherwise sqlite-incompatible, then writing a
stacktrace of all threads to a textfile in the config directory
(in case this deadlock is due to something completely different),
before finally crashing spectacularly
additionally, delete the database if the creation fails, which should
prevents the deadlock on the next startup, so combine that with a
message hinting at the filesystem incompatibility
the 1-minute limit may sound excessively gracious, but considering what
some of the copyparty instances out there is running on, really isn't
this was reported when connecting to a cifs server running alpine
thx to abex on discord for the detailed bug report!
features which should be good to go:
* user groups
* assigning permissions by group
* dynamically created volumes based on username/groupname
* rebuild vfs when new users/groups appear
but several important features still pending;
* detect dangerous configurations
* dynamic vol below readable path
* remember volumes created during previous runs
* helps prevent unintended access
* correct filesystem-scan on startup
some clients (clonezilla-webdav) rapidly create and delete files;
this fails if copyparty is still hashing the file (usually the case)
and the same thing can probably happen due to antivirus etc
add global-option --rm-retry (volflag rm_retry) specifying
for how long (and how quickly) to keep retrying the deletion
default: retry for 5sec on windows, 0sec (disabled) on everything else
because this is only a problem on windows
when a file was reindexed (due to a change in size or last-modified
timestamp) the uploader-IP would get removed, but the upload timestamp
was ported over. This was intentional so there was probably a reason...
new behavior is to keep both uploader-IP and upload timestamp if the
file contents are unchanged (determined by comparing warks), and to
discard both uploader-IP and upload timestamp if that is not the case
due to all upload APIs invoking up2k.hash_file to index uploads,
the uploads could block during a rescan for a crazy long time
(past most gateway timeouts); now this is mostly fire-and-forget
"mostly" because this also adds a conditional slowdown to
help the hasher churn through if the queue gets too big
worst case, if the server is restarted before it catches up, this
would rely on filesystem reindexing to eventually index the files
after a restart or on a schedule, meaning uploader info would be
lost on shutdown, but this is usually fine anyways (and this was
also the case until now)
* make gen_tree 0.1% faster
* improve filekey warning message
* fix oversight in 0c50ea1757
* support `--xdev` on windows (the python docs mention that os.scandir
doesn't assign st_ino, st_dev and st_nlink on win but i can't read)
* permission `.` grants dotfile visibility if user has `r` too
* `-ed` will grant dotfiles to all `r` accounts (same as before)
* volflag `dots` likewise
also drops compatibility for pre-0.12.0 `-v` syntax
(`-v .::red` will no longer translate to `-v .::r,ed`)
when moving/deleting a file, all symlinked dupes are verified to ensure
this action does not break any symlinks, however it did this by checking
the realpath of each link. This was not good enough, since the deleted
file may be a part of a series of nested symlinks
this situation occurs because the deduper tries to keep relative
symlinks as close as possible, only traversing into parent/sibling
folders as required, which can lead to several levels of nested links
webdav clients tend to upload and then immediately delete
files to test for write-access and available disk space,
so don't crash and burn when that happens
* cpp_uptime is now a gauge
* cpp_bans is now cpp_active_bans (and also a gauge)
and other related fixes:
* stop emitting invalid cpp_disk_size/free for offline volumes
* support overriding the spec-mandatory mimetype with ?mime=foo
on upload, dupes are by default handled by symlinking to the existing
copy on disk, writing the uploader's local mtime into the symlink mtime,
which is also what gets indexed in the db
this worked as intended, however during an -e2dsa rescan on startup the
symlink destination timestamps would be used instead, causing a reindex
and the resulting loss of uploader metadata (ip, timestamp)
will now always use the symlink's mtime;
worst-case 1% slower startup (no dhash)
this change will cause a reindex of incorrectly indexed files, however
as this has already happened at least once due to the bug being fixed,
there will be no additional loss of metadata
* js: use .call instead of .bind when possible
* when running without e2d, the message on startup regarding
unfinished uploads didn't show the correct filesystem path
unposting could collide with most other database-related activities,
causing one or the other to fail.
luckily the unprotected query performed by the unpost API happens to be
very cheap, so also the most likely to fail, and would succeed upon a
manual reattempt from the UI.
even in the worst case scenario, there would be no unrecoverable damage
as the next rescan would auto-repair any resulting inconsistencies.
if someone with admin rights refreshes the homepage exactly as the
directory indexer decides to `_drop_caches`, the indexer thread would
die and the up2k instance would become inoperable...
luckily the probability of hitting this by chance is absolutely minimal,
and the worst case scenario is having to restart copyparty if this
happens immediately after startup; there is no risk of database damage
* when deleting files, do not cascade upwards through empty folders
* when moving folders, also move any empty folders inside
the only remaining action which autoremoves empty folders is
files getting deleted as they expire volume lifetimes
also prevents accidentally moving parent folders into subfolders
(even though that actually worked surprisingly well)
