togglebutton in the ui switches between 2 (off/default) and
1 (on/quick) confirmations; global-option `--qdel` sets the default
setting `--qdel=0` changes the togglebutton to switch
between 1 (off/default) confirmations and 0 (on)
in other words, when the ui-button is enabled, it
always reduces the number of confirmations by one
the unix-permissions of new files/folders can now be changed
* global-option --chmod-f, volflag chmod_f for files
* global-option --chmod-d, volflag chmod_d for directories
the expected value is a standard three-digit octal value
(User/Group/Other) such as 755, 750, 644, 640, etc
If a file has no known extension the content type gets set to
application/octet-stream causing the browser try and download the file
when viewed directly.
This quickly becomes annoying as many of the files I interact with often
have no extension. I.e., config files, log files, LICENSE files and
other random text files.
This patch uses libmagic to detect the file type and set the
content-type header. It also does this for the RSS feed and webdav for
sake of completeness.
This patch does not touch the front end at all so these files still have a 'txt'
button and a type of '%' in the web UI. But when clicked on, the browser
will display the files correctly.
This feature is enabled with the existing "magic" option. I thought this
fit as the existing functionality also uses libmagic and gives file
extensions to files on upload. Tell me if it should be its own option
instead.
The code base was very confusing, this patch works but I have no idea if
it's the way you'd like this implemented. Hopefully its acceptable as
is.
file hashing became drastically slower in recent chrome versions;
* 748 MiB/s in 131.0.6778.86
* 747 MiB/s in 132.0.6834.160
* 485 MiB/s in 133.0.6943.60
* 319 MiB/s in 134.0.6998.36
the silver lining: it looks like chrome-bug 1352210 is improving
(crypto.subtle, the native hasher, now scales with multiple cores)
* 133.0.6943.60: speed peaked at 2 threads; 341 MiB/s, 485 MiB/s
* 134.0.6998.36: peak at 7; 193, 383, 383, 408, 421, 431, 438, 438
* 137.0.7151.41: peak at 8; 210, 382, 445, 513, 573, 573, 585, 598
MiB/s when hashing with 1, 2, ..., 7, 8 webworkers respectively
on a ryzen7-5800x with 2x16g 2133mhz ram
characteristics of versions between v134 and v137 are unknown
(cannot find old official builds to test), but v137 is a good
cutoff for minimizing risk of hitting chrome-bugs
meanwhile, hash-wasm scales linearly up to 8 cores;
0=328 1=377 2=738 3=947 4=1090 5=1190 6=1380 7=1530 8=1810
(0 = wasm on mainthread, no webworkers)
but it looks like chrome-bug 383568268 is making a return,
so keep the limit of max 4 threads if machine has more than
4 cores (and numCores-1 otherwise)
* do not take lock on shares-db / sessions-db when running with
`--ah-gen` or `--ah-cli` (allows a 2nd instance for that purpose)
* add options to print effective salt for ah/fk/dk; useful for nixos
and other usecases where config is derived or otherwise opaque
* mention potential hdd-bottleneck from big values
* most browsers enforce a max-value of 6 (c354a38b)
* chunk-stitching (132a8350) made this less important;
still beneficial, but only to a point
the up2k databases are, by default, stored in a `.hist` subfolder
inside each volume, next to thumbnails and transcoded audio
add a new option for storing the databases in a separate location,
making it possible to tune the underlying filesystem for optimal
performance characteristics
the `--hist` global-option and `hist` volflag still behave like
before, but `--dbpath` and volflag `dbpath` will override the
histpath for the up2k-db and up2k-snap exclusivey
`--md-hist` / volflag `md_hist` specifies where to put old
versions of markdown files when edited using the web-ui;
* `s` = create `.hist` subfolder next to the markdown file
(the default, both previously and now)
* `v` = use the volume's hist-path, either according to
`--hist` or the `hist` volflag. NOTE: old versions
will not be retrievable through the web-ui
* `n` = nope / disabled; overwrite without backup
specifically google, but also some others, have started ignoring
rel="nofollow" while also understanding just enough javascript to
try viewing binary files as text
download-as-tar-gz becomes 2.4x faster in docker
segfaults on windows, so don't use it there
does not affect fedora or gentoo,
since zlib-ng is already system-default on those
also adds a global-option to write list of successful
binds to a textfile, for automation / smoketest purposes
too restrictive, blocking editing through webdav and ftp
but since logues and readmes can be used as helptext for users
with write-only access, it makes sense to block logue/readme
uploads from write-only users
users with write-only access can still upload any file as before,
but the filename prefix `_wo_` is added onto files named either
README.md | PREADME.md | .prologue.html | .epilogue.html
the new option `--wo-up-readme` restores previous behavior, and
will not add the filename-prefix for readmes/logues
previously, the native python-error was printed when reading
the contents of a textfile using the wrong character encoding
while technically correct, it could be confusing for end-users
add a helper to produce a more helpful errormessage when
someone (for example) tries to load a latin-1 config file
adds a third possible value for the `replace` property in handshakes:
* absent or False: never overwrite an existing file on the server,
and instead generate a new filename to avoid collision
* True: always overwrite existing files on the server
* "mt": only overwrite if client's last-modified is more recent
(this is the new option)
the new UI button toggles between all three options,
defaulting to never-overwrite
1. warn about unrecognized volflags
previously, when specifying an unknown volflag, it would
be silently ignored, giving the impression that it applied
2. also allow uppercase, kebab-case
(previously, only snake_case was accepted)
3. mention every volflag in --help-flags
(some volflags were missing)
some clients, including KDE Dolphin (kioworker/6.10) keeps
sending requests without the basic-auth header, expecting
the server to respond with a 401 before it does
most clients only do this for the initial request, which is
usually a PROPFIND, which makes this nice and simple -- but
turns out we need to consider this for GET as well...
this is tricky because a graphical webbrowser must never
receive a 401 lest it becomes near-impossible to deauth,
and that's exactly what Dolphin pretends to be in its UA
man （ ´_ゝ`）
note: `KIO/` hits konqueror so don't
new global-option / volflag `zip_who` specifies
who gets to use the download-as-zip/tar function;
* 0: nobody, same as --no-zip
* 1: admins
* 2: authorized users with read-access
* 3: anyone with read-access
support for "owa", audio-only webm, was introduced in iOS 17.5
owa is a more compliant alternative to opus-caf from iOS 11,
which was technically limited to CBR opus, a limitation which
we ignored since it worked mostly fine for regular opus too
being the new officially-recommended way to do things,
we'll default to owa for iOS 18 and later, even though
iOS still has some bugs affecting our use specifically:
if a weba file is preloaded into a 2nd audio object,
safari will throw a spurious exception as playback is
initiated, even as the file is playing just fine
the `.ld` stuff is an attempt at catching and ignoring this
spurious error without eating any actual network exceptions
previously, when IdP was enabled, the password-based login would be
entirely disabled. This was a semi-conscious decision, based on the
assumption that you would always want to use IdP after enabling it.
it makes more sense to keep password-based login working as usual,
conditionally disengaging it for requests which contains a valid
IdP username header. This makes it possible to define fallback
users, or API-only users, and all similar escape hatches.
a better alternative to using `--no-idx` for this purpose since
this also excludes recent uploads, not just during fs-indexing,
and it doesn't prevent deduplication
also speeds up searches by a tiny amount due to building the
sanchecks into the exclude-filter while parsing the config,
instead of during each search query
* if free ram on startup is less than 2 GiB,
use smaller chunks for parallel file hashing
* if --th-max-ram is lower than 0.25 (256 MiB),
print a warning that thumbnails will not work
* make thumbnail cleaner immediately do a sweep on startup,
forgetting any failed conversions so they can be retried
in case the memory limit was increased since last run
an extremely brutish workaround for issues such as #110 where
browsers receive an HTTP 304 and misinterpret as HTTP 200
option `--no304=1` adds the button `no304` to the controlpanel
which can be enabled to force-disable caching in that browser
the button is default-disabled; by specifying `--no304=2`
instead of `--no304=1` the button becomes default-enabled
can also always be enabled by accessing `/?setck=no304=y`
previously, the biggest file that could be uploaded through
cloudflare was 383 GiB, due to max num chunks being 4096
`--u2sz`, which takes three ints (min-size, target, max-size)
can now be used to enforce a max chunksize; chunks larger
than max-size gets split into smaller subchunks / chunklets
subchunks cannot be stitched/joined, and subchunks of the
same chunk must be uploaded sequentially, one at a time
if a subchunk fails due to bitflips or connection-loss,
then the entire chunk must (and will) be reuploaded
global-option `--no-clone` / volflag `noclone` entirely disables
serverside deduplication; clients will then fully upload dupe files
can be useful when `--safe-dedup=1` is not an option due to other
software tampering with the on-disk files, and your filesystem has
prohibitively slow or expensive reads
* do not absreal paths unless necessary
* do not determine username if no users configured
* impacket 0.12 fixed the foldersize limit, but now
you get extremely poor performance in large folders
so the previous workaround is still default-enabled
* pyz: yeet the resource tar which is now pointless thanks to pkgres
* cache impresource stuff because pyz lookups are Extremely slow
* prefer tx_file when possible for slightly better performance
* use hardcoded list of expected resources instead of dynamic
discovery at runtime; much simpler and probably safer
* fix some forgotten resources (copying.txt, insecure.pem)
* fix loading jinja templates on windows
add support for reading webdeps and jinja-templates using either
importlib_resources or pkg_resources, which removes the need for
extracting these to a temporary folder on the filesystem
* util: add helper functions to abstract embedded resource access
* http*: serve embedded resources through resource abstraction
* main: check webdeps through resource abstraction
* httpconn: remove unused method `respath(name)`
* use __package__ to find package resources
* util: use importlib_resources backport if available
* pass E.pkg as module object for importlib_resources compatibility
* util: add pkg_resources compatibility to resource abstraction
dedup is still encouraged and fully supported, but
being default-enabled has caused too many surprises
enabling `--dedup` restores the previous default behavior
also renames `--never-symlink` to `--hardlink-only`
previously, the assumption was made that the database and filesystem
would not desync, and that an upload could safely be substituted with
a symlink to an existing copy on-disk, assuming said copy still
existed on-disk at all
this is fine if copyparty is the only software that makes changes to
the filesystem, but that is a shitty assumption to make in hindsight
add `--safe-dedup` which takes a "safety level", and by default (50)
it will no longer blindly expect that the filesystem has not been
altered through other means; the file contents will now be hashed
and compared to the database
deduplication can be much slower as a result, but definitely worth it
as this avoids some potentially very unpleasant surprises
the previous behavior can be restored with `--safe-dedup 1`
timezone can be changed with `export TZ=Europe/Oslo` before launch
using naive timestamps like this appears to be safe as of 3.13-rc1,
no deprecation warnings, just a tiny bit slower than assuming UTC
