Commit graph

2670 commits

Author SHA1 Message Date
ed 6a8d5e1731 ui: batch-rename: remember last regex + format 2024-11-02 18:06:39 +00:00
ed d9962f65b3 ui: folder loading indicator stole focus
show a spinning halfcircle around the +/- instead of
moving the focus to the selected folder in the sidebar,
since that could mess with keyboard scrolling
2024-11-02 17:58:30 +00:00
ed 119e88d87b bubble OS-filesystem errors to client
send a 500 or 404 if a folder is inaccessible or does not exist

previously it would return an empty directory listing instead
2024-11-02 17:38:17 +00:00
ed 71d9e010d9 ui: make hotkey-help less eager to show itself
would appear when typing `?` into textboxes
2024-10-30 19:40:48 +00:00
ed 5718caa957 ui: url-options to set grid/thumbs on/off 2024-10-30 19:24:00 +00:00
ed efd8a32ed6 ui: show switch-to-https on 403s too 2024-10-28 03:38:15 +00:00
ed ccdacea0c4 v1.15.10 2024-10-27 07:51:11 +00:00
ed 4bdcbc1cb5 shares: allow upload, unpost
* files can be uploaded into writeable shares

* add "write-only" button to the create-share ui

* unpost is possible while viewing the relevant share
2024-10-26 21:36:07 +00:00
ed dd6dbdd90a http 304: client-option to force-disable cache
an extremely brutish workaround for issues such as #110 where
browsers receive an HTTP 304 and misinterpret as HTTP 200

option `--no304=1` adds the button `no304` to the controlpanel
which can be enabled to force-disable caching in that browser

the button is default-disabled; by specifying `--no304=2`
instead of `--no304=1` the button becomes default-enabled

can also always be enabled by accessing `/?setck=no304=y`
2024-10-26 17:56:54 +00:00
ed 63013cc565 http 304: k304 obsoleted for ie11 by Vary
the Vary header killed caching in all versions of internet explorer
so there's no point conditionally enabling k304 for trident anymore
2024-10-25 22:32:58 +00:00
ed 912402364a http 304: strip Content-Length and Content-Type
these response headers are usually not included in 304 replies,
and their presence are suspected to confuse some clients (#110)

also strip `out_headerlist` (primarily cookie assignments)
2024-10-25 22:24:40 +00:00
ed 159f51b12b http 304: if-range, backdating
add support for the `If-Range` header which is generally used to
prevent resuming a partial download after the source file on the
server has been modified, by returning HTTP 200 instead of a 206

also simplifies `If-Modified-Since` and `If-Range` handling;
previously this was a spec-compliant lexical comparison,
now it's a basic string-comparison instead. The server will now
reply 200 also when the server mtime is older than the client's.
This is technically not according to spec, but should be safer,
as it allows backdating timestamps without purging client cache
2024-10-25 22:05:59 +00:00
ed 7678a91b0e debug: --ohead (log response headers) 2024-10-25 20:00:19 +00:00
ed b13899c63d make --u2sz more intuitive
previously, it only accepted the 3-tuple `min,default,max`

if given a single integer (or any other unexpected value),
the up2k js would enter an infinite loop, eat all the ram
and crash the browser (nice)

fix this by accepting a single integer (for example 96)
and translating it to `1,96,96`
2024-10-22 21:37:51 +00:00
ed 3a0d882c5e fix NetMap -j0 compat
would crash on startup if `-j0` was
combined with `--ipa` or `--ipu`
2024-10-22 20:53:19 +00:00
ed cecef88d6b v1.15.9 2024-10-18 23:42:20 +00:00
ed 7ffd805a03 add RSS feed output; closes #109 2024-10-18 23:24:12 +00:00
ed a7e2a0c981 up2k: fix chinese-specific js crash; closes #108
the client-side ETA, included as metadata in POSTs,
would crash the js with the initial "Starting..." text
2024-10-18 19:04:22 +00:00
ed 2a570bb4ca fix --df for webdav; closes #107
PUT uploads, as used by webdav, would stat the absolute
path of the file to be created, which would throw ENOENT

strip components until the path is an existing directory

and also try to enforce disk space / volume size limits
even when the incoming file is of unknown size
2024-10-18 18:14:35 +00:00
ed 5ca8f0706d up2k.js: detect broken webworkers;
the first time a file is to be hashed after a website refresh,
a set of webworkers are launched for efficient parallelization

in the unlikely event of a network outage exactly at this point,
the workers will fail to start, and the hashing would never begin

add a ping/pong sequence to smoketest the workers, and
fallback to hashing on the main-thread when necessary
2024-10-18 16:50:15 +00:00
ed a9b4436cdc up2k: improve upload retry/timeout
* `js:` make handshake retries more aggressive
* `u2c:` reduce chunks timeout + ^
* `main:` reduce tcp timeout to 128sec (js is 42s)
* `httpcli:` less confusing log messages
2024-10-18 16:24:31 +00:00
ed 9f000beeaf v1.15.8 2024-10-16 21:53:23 +00:00
ed ff0a71f212 gallery: play m4v videos 2024-10-16 21:36:11 +00:00
ed 22dfc6ec24 ui-toast: hide countdown if infinite 2024-10-16 21:32:47 +00:00
ed 48147c079e subchunks: fix eta, cfg-ui 2024-10-16 21:17:00 +00:00
ed d715479ef6 add chickenbit to force hashwasm 2024-10-16 20:23:02 +00:00
ed fc8298c468 up2k: avoid cloudflare upload size-limit
previously, the biggest file that could be uploaded through
cloudflare was 383 GiB, due to max num chunks being 4096

`--u2sz`, which takes three ints (min-size, target, max-size)
can now be used to enforce a max chunksize; chunks larger
than max-size gets split into smaller subchunks / chunklets

subchunks cannot be stitched/joined, and subchunks of the
same chunk must be uploaded sequentially, one at a time

if a subchunk fails due to bitflips or connection-loss,
then the entire chunk must (and will) be reuploaded
2024-10-16 19:29:08 +00:00
ed e94ca5dc91 up2k: improve logging 2024-10-16 15:41:19 +00:00
ed 114b71b751 up2k: fix filesystem toctou
previously and currently, as an upload completes, its "done" flag
is not set until all the data has been flushed to disk

however, the list of missing chunks becomes empty before the flush,
and that list was incorrectly used to determine completion state
in some dedup-related logic

as a result, duplicate uploads could initially fail, and would
succeed after the client automatically retried a handful of times
2024-10-16 15:32:58 +00:00
ed c1c0ecca13 v1.15.7 2024-10-13 22:44:57 +00:00
ed ee62836383 bitflip logging 2024-10-13 22:37:35 +00:00
ed 705f598b1a up2k non-e2d fixes:
* respect noforget when loading snaps
* ...but actually forget deleted files otherwise
* insert empty need/hash as necessary
2024-10-13 22:10:27 +00:00
ed 22b58e31ef unpost: authed users can see anon on same ip 2024-10-13 22:00:15 +00:00
ed b7f9bf5a28 cidr-based autologin 2024-10-13 21:56:26 +00:00
ed fabada95f6 v1.15.6 2024-10-11 22:56:10 +00:00
ed 9ccd8bb3ea support viewing dotfile docs; closes #104 2024-10-11 22:06:43 +00:00
ed 1d68acf8f0 add preadme.md; closes #105 2024-10-11 21:52:44 +00:00
ed 1e7697b551 misc cleanup;
* more typos
* python 3.13 deprecations
2024-10-11 20:46:40 +00:00
ed 4a4ec88d00 up2k: fix hs after bitflips / net-glitch
chunk stitching could cause handshakes to initiate
a new upload rather than resume an ongoing one
2024-10-11 19:48:44 +00:00
ed 6adc778d62 fix a buttload of typos 2024-10-11 18:58:14 +00:00
ed 6b7ebdb7e9 upgrade old snaps to dwrk + fix ptop
ptop would be wrong if a volume was moved on-disk since last run
2024-10-09 06:05:55 +00:00
ed 3d7facd774 add option to entirely disable dedup
global-option `--no-clone` / volflag `noclone` entirely disables
serverside deduplication; clients will then fully upload dupe files

can be useful when `--safe-dedup=1` is not an option due to other
software tampering with the on-disk files, and your filesystem has
prohibitively slow or expensive reads
2024-10-08 21:27:19 +00:00
ed ff012221ae v1.15.5 2024-10-05 18:03:04 +00:00
ed c398553748 pkgres: fix multiprocessing 2024-10-05 17:32:08 +00:00
ed f0abc0ef59 v1.15.4 2024-10-04 23:19:28 +00:00
ed a99fa3375d the impresources.files traversible is not threadsafe 2024-10-04 22:37:29 +00:00
ed 22c7e09b3f small fixes;
* make-sfx: delete failed deps downloads
* tlcheck: detect untranslated strings
2024-10-04 20:56:16 +00:00
ed 0dfe1d5b35 toast countdown bar 2024-10-04 19:29:54 +00:00
ed a99a3bc6d7 audio-player: fix compact-mode rendering glitch on narrow screens 2024-10-04 18:15:18 +00:00
ed 9804f25de3 add option for natural sorting; thx @oshiteku 2024-10-04 00:30:04 +00:00
ed ae98200660 og: support filekeys 2024-10-03 23:52:11 +00:00
ed e45420646f share folders as qr-codes 2024-10-03 23:14:06 +00:00
ed 21be82ef8b fix #101 (show logues even if dotfiles are hidden) 2024-10-03 22:19:32 +02:00
ed 001afe00cb i18n: time plurals 2024-10-03 07:38:33 +00:00
ed 19a5985f29 allow uploading logues; closes #100 2024-10-02 23:16:59 +00:00
ed 2715ee6c61 fix confusing toast on F2 with nothing selected (#100) 2024-10-02 23:11:29 +00:00
ed dc157fa28f webdav: support explicit <allprop/> (WinSCP) 2024-10-02 22:28:23 +00:00
ed 1ff14b4e05 optimizations, failsafes, formatting 2024-10-02 21:59:53 +00:00
ed 480ac254ab webdav: show toplevel volumes when root is unmapped
previously, only real folders could be listed by a webdav client;
a server which does not have any filesystem paths mapped to `/`
would cause clients to panic when trying to list the server root

now, assuming volumes `/foo` and `/bar/qux` exist, when accessing `/`
the user will see `/foo` but not `/bar` due to limitations in `walk`,
and `qux` will only appear when viewing `/bar`

a future rework of the recursion logic should further improve this
2024-10-02 21:12:58 +00:00
ed f1646b96ca dist: strip some pointless code 2024-10-01 18:35:36 +00:00
ed 44f2b63e43 partyfuse: embed fuse.py into sfx 2024-10-01 18:27:42 +00:00
ed 3900e66158 partyfuse: modernize html parser (just in case) 2024-10-01 17:00:17 +00:00
ed fcdc1728eb #102: make UI translation easier in docker 2024-10-01 00:04:07 +00:00
ed 88a1c5ca5d optimize non-e2d ram usage down to 10% or so
drop chunk-hashes in the up2k snap, plus other insignificant attribs
to reduce both the snapfile size and the ram usage by about 90%

reduces startup/shutdown time by a lot since there's less to serdes
(does not affect -e2d which was already optimal)

other changes:

* improve incoming-eta accuracy when the initial handshake
   was made a long time before the upload actually started

* move the list of incoming files in the controlpanel to the top
2024-09-27 21:11:10 +00:00
ed 047176b297 py2 fix 2024-09-27 21:06:01 +00:00
ed dc4d0d8e71 smb: upto 2x faster; but still very buggy:
* do not absreal paths unless necessary
* do not determine username if no users configured
* impacket 0.12 fixed the foldersize limit, but now
   you get extremely poor performance in large folders
   so the previous workaround is still default-enabled
2024-09-27 17:09:48 +00:00
ed 66b260cea9 pkgres: fix tiny leak in template loader 2024-09-20 22:25:36 +00:00
ed 58cf01c2ad fix linter warnings 2024-09-20 22:24:39 +00:00
ed d866841c19 pkgres:
* pyz: yeet the resource tar which is now pointless thanks to pkgres
* cache impresource stuff because pyz lookups are Extremely slow
* prefer tx_file when possible for slightly better performance
* use hardcoded list of expected resources instead of dynamic
   discovery at runtime; much simpler and probably safer
* fix some forgotten resources (copying.txt, insecure.pem)
* fix loading jinja templates on windows
2024-09-19 22:04:49 +00:00
Shiz a462a644fb
Python 3.7 package resources support (#98)
add support for reading webdeps and jinja-templates using either
importlib_resources or pkg_resources, which removes the need for
extracting these to a temporary folder on the filesystem

* util: add helper functions to abstract embedded resource access
* http*: serve embedded resources through resource abstraction
* main: check webdeps through resource abstraction
* httpconn: remove unused method `respath(name)`
* use __package__ to find package resources
* util: use importlib_resources backport if available
* pass E.pkg as module object for importlib_resources compatibility
* util: add pkg_resources compatibility to resource abstraction
2024-09-19 09:00:34 +00:00
ed 678675a9a6 fix prometheus metrics; broke in 609c5921 2024-09-16 21:04:58 +00:00
ed c0c0a1a83a v1.15.3 2024-09-16 01:07:50 +00:00
ed b90e1200d7 v1.15.2 2024-09-16 00:20:20 +00:00
ed 4493a0a804 misc mojibake filename support 2024-09-16 00:12:49 +00:00
ed 58835b2b42 ux bugfixes:
* show media tags in shares
* html hydrator assumed a folder named `foo.txt` was a doc
* due to sessions, use `pwd` as password placeholder on services
2024-09-15 23:37:24 +00:00
ed 427597b603 show total directory size in listings
sizes are computed during `-e2ds` indexing, and new uploads
are counted, but a rescan is necessary after a move or delete
2024-09-15 23:01:18 +00:00
ed 7d64879ba8 more optimizations,
* 5% less cpu load from clients fetching thumbnails
* and slight improvement to up2k stuff
2024-09-15 17:46:43 +00:00
ed bb715704b7 ren_open was too fancy 2024-09-15 14:39:35 +00:00
ed d67e9cc507 sqlite and misc optimizations:
* exponentially slow upload handshakes caused by lack of rd+fn
   sqlite index; became apparent after a volume hit 200k files
* listing big folders 5% faster due to `_quotep3b`
* optimize `unquote`, 20% faster but only used rarely
* reindex on startup 150x faster in some rare cases
   (same filename in MANY folders)

the database is now around 10% larger (likely worst-case)
2024-09-15 13:18:43 +00:00
ed 2927bbb2d6 strip dev-only asserts at build stage 2024-09-14 22:17:35 +00:00
ed 0527b59180 cosmetic: only print hostname warning once 2024-09-14 20:37:56 +00:00
ed 4e75534ef8 optimize BrokerThr, 7x faster:
reduce the overhead of function-calls from the client thread
to the svchub singletons (up2k, thumbs, metrics) down to 14%

and optimize up2k chunk-receiver to spend 5x less time bookkeeping
which restores up2k performance to before introducing incoming-ETA
2024-09-11 20:37:10 +00:00
ultwcz 7a573cafd1
fix: translation: Check the newly added Chinese translation (#97) 2024-09-11 19:03:53 +00:00
ed 844194ee29 incoming-ETA: improve accuracy 2024-09-11 06:56:12 +00:00
ed 609c5921d4 list incoming files + ETA in controlpanel 2024-09-10 21:24:05 +00:00
ed e9d962f273 v1.15.1 2024-09-09 23:43:43 +00:00
ed b5405174ec add login sessions 2024-09-09 23:39:20 +00:00
ed 6eee601521 fix u2c --ow (overwrite/replace)
the u2c flag to overwrite files on the server became no-op in v1.13.8
2024-09-09 19:40:38 +00:00
ed c140eeee6b v1.15.0 2024-09-08 19:25:46 +00:00
ed c5988a04f9 up2k.js: bump handshake timeout for safededup 2024-09-08 18:06:37 +00:00
ed a2e0f98693 disable upload deduplication by default;
dedup is still encouraged and fully supported, but
being default-enabled has caused too many surprises

enabling `--dedup` restores the previous default behavior

also renames `--never-symlink` to `--hardlink-only`
2024-09-08 17:09:14 +00:00
ed e5a836cb7d og: fix links to textfiles 2024-09-08 12:12:34 +00:00
ed b0de84cbc5 db-verify: support newlines in filenames + flag 404s 2024-09-08 00:44:22 +00:00
ed cbb718e10d css fixes:
* improve hotdog-stand theme
* fix up2k tabs glow (went poof in a syntax error)
2024-09-07 19:29:40 +00:00
ed b5ad9369fe confine xlink behavior behind its volflag
symlinks between volumes will only be created if xlink is
enabled, so such symlinks should be ignored if xlink is
disabled, as they might originate from other software

this prevents accidental rewriting of non-dedup symlinks
2024-09-07 19:17:32 +00:00
ed 4401de0413 fix mv with --no-dedup in volumes with dupes;
if --no-dedup was enabled in a volume which already contained
symlinked duplicate files, renaming/moving folders could fail

this is due to folder contents being moved one file at a time
(which is how symlink breakage is prevented) except the links
are moved assuming the final directory layout, meaning they
may be intermittently broken during the movie

with no-dedup, the symlinks are converted into full files as
each symlink is encountered, but a temporarily broken symlink
would crash the procedure

fix this by giving `_symlink` a new parameter `fsrc`
which is a known valid inode for data copying purposes
2024-09-07 00:47:12 +00:00
ed 6e671c5245 verify on-disk contents before dedup;
previously, the assumption was made that the database and filesystem
would not desync, and that an upload could safely be substituted with
a symlink to an existing copy on-disk, assuming said copy still
existed on-disk at all

this is fine if copyparty is the only software that makes changes to
the filesystem, but that is a shitty assumption to make in hindsight

add `--safe-dedup` which takes a "safety level", and by default (50)
it will no longer blindly expect that the filesystem has not been
altered through other means; the file contents will now be hashed
and compared to the database

deduplication can be much slower as a result, but definitely worth it
as this avoids some potentially very unpleasant surprises

the previous behavior can be restored with `--safe-dedup 1`
2024-09-06 19:08:14 +00:00
ed b599fbae97 use local timezone in log messages; closes #96
timezone can be changed with `export TZ=Europe/Oslo` before launch

using naive timestamps like this appears to be safe as of 3.13-rc1,
no deprecation warnings, just a tiny bit slower than assuming UTC
2024-09-05 19:31:33 +00:00
ed a8dabc99f6 add more translations 2024-09-04 23:46:32 +00:00
ed f1130db131 fix confusing message when uploading dupes
due to deduplication, it is intentionally impossible to
upload several identical copies of a file in parallel

by default, the up2k client will upload files sorted by
size, which usually leads to dupes being grouped together,
and it will try to do just that

this is by design, as it improves performance on average,
but it also shows the confusing (but technically-correct)
message "resume the partial upload into the original path"

fix this with a more appropriate message

note that this approach was selected in favor of pausing
handshakes while the initial copy finishes uploading,
because that could severely reduce upload performance
by preventing optimal use of multiple connections
2024-09-04 22:03:26 +00:00