update pkgs to 1.19.2

previous behavior can be restored with --th-spec-p 2

thumbnails cache (.hist/th/) must be deleted to take effect

.github/ISSUE_TEMPLATE/bug_report.md

@@ -8,33 +8,42 @@ assignees: '9001'
 ---
 
 NOTE:
+**please use english, or include an english translation.** aside from that,  
 all of the below are optional, consider them as inspiration, delete and rewrite at will, thx md
 
 
-**Describe the bug**
+### Describe the bug
 a description of what the bug is
 
-**To Reproduce**
+### To Reproduce
 List of steps to reproduce the issue, or, if it's hard to reproduce, then at least a detailed explanation of what you did to run into it
 
-**Expected behavior**
+### Expected behavior
 a description of what you expected to happen
 
-**Screenshots**
+### Screenshots
 if applicable, add screenshots to help explain your problem, such as the kickass crashpage :^)
 
-**Server details**
-if the issue is possibly on the server-side, then mention some of the following:
-* server OS / version: 
-* python version: 
-* copyparty arguments: 
-* filesystem (`lsblk -f` on linux): 
+### Server details (if you are using docker/podman)
+remove the ones that are not relevant:
+* **server OS / version:** 
+* **how you're running copyparty:** (docker/podman/something-else)
+* **docker image:** (variant, version, and arch if you know)
+* **copyparty arguments and/or config-file:** 
 
-**Client details**
+### Server details (if you're NOT using docker/podman)
+remove the ones that are not relevant:
+* **server OS / version:** 
+* **what copyparty did you grab:** (sfx/exe/pip/arch/...)
+* **how you're running it:** (in a terminal, as a systemd-service, ...)
+* run copyparty with `--version` and grab the last 3 lines (they start with `copyparty`, `CPython`, `sqlite`) and paste them below this line:
+* **copyparty arguments and/or config-file:** 
+
+### Client details
 if the issue is possibly on the client-side, then mention some of the following:
 * the device type and model: 
 * OS version: 
 * browser version: 
 
-**Additional context**
+### Additional context
 any other context about the problem here

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,6 +7,8 @@ assignees: '9001'
 
 ---
 
+NOTE:
+**please use english, or include an english translation.** aside from that,  
 all of the below are optional, consider them as inspiration, delete and rewrite at will
 
 **is your feature request related to a problem? Please describe.**

.gitignore

@@ -43,3 +43,6 @@ scripts/docker/*.err
 
 # nix build output link
 result
+
+# IDEA config
+.idea/

CONTRIBUTING.md

@@ -1,8 +1,21 @@
-* do something cool
+* **found a bug?** [create an issue!](https://github.com/9001/copyparty/issues) or let me know in the [discord](https://discord.gg/25J8CdTT6G) :>
+* **fixed a bug?** create a PR or post a patch! big thx in advance :>
+* **have a cool idea?** let's discuss it! anywhere's fine, you choose.
 
-really tho, send a PR or an issue or whatever, all appreciated, anything goes, just behave aight 👍👍
+but please:
+
+
+
+# do not use AI / LMM when writing code
+
+copyparty is 100% organic, free-range, human-written software!
+
+> ⚠ you are now entering a no-copilot zone
+
+the *only* place where LMM/AI *may* be accepted is for [localization](https://github.com/9001/copyparty/tree/hovudstraum/docs/rice#translations) if you are fluent and have confirmed that the translation is accurate.
+
+sorry for the harsh tone, but this is important to me 🙏
 
-but to be more specific,
 
 
 # contribution ideas
@@ -28,6 +41,8 @@ aside from documentation and ideas, some other things that would be cool to have
 
 * **translations** -- the copyparty web-UI has translations for english and norwegian at the top of [browser.js](https://github.com/9001/copyparty/blob/hovudstraum/copyparty/web/browser.js); if you'd like to add a translation for another language then that'd be welcome! and if that language has a grammar that doesn't fit into the way the strings are assembled, then we'll fix that as we go :>
 
+  * but please note that support for [RTL (Right-to-Left) languages](https://en.wikipedia.org/wiki/Right-to-left_script) is currently not planned, since the javascript is a bit too jank for that
+
 * **UI ideas** -- at some point I was thinking of rewriting the UI in react/preact/something-not-vanilla-javascript, but I'll admit the comfiness of not having any build stage combined with raw performance has kinda convinced me otherwise :p but I'd be very open to ideas on how the UI could be improved, or be more intuitive.
 
 * **docker improvements** -- I don't really know what I'm doing when it comes to containers, so I'm sure there's a *huge* room for improvement here, mainly regarding how you're supposed to use the container with kubernetes / docker-compose / any of the other popular ways to do things. At some point I swear I'll start learning about docker so I can pick up clach04's [docker-compose draft](https://github.com/9001/copyparty/issues/38) and learn how that stuff ticks, unless someone beats me to it!

bin/README.md

@@ -78,3 +78,6 @@ cd /mnt/nas/music/.hist
 # [`prisonparty.sh`](prisonparty.sh)
 * run copyparty in a chroot, preventing any accidental file access
 * creates bindmounts for /bin, /lib, and so on, see `sysdirs=`
+
+# [`bubbleparty.sh`](bubbleparty.sh)
+* run copyparty in an isolated process, preventing any accidental file access and more

bin/bubbleparty.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+# usage: ./bubbleparty.sh ./copyparty-sfx.py ....
+bwrap \
+  --unshare-all \
+  --ro-bind /usr /usr \
+  --ro-bind /bin /bin \
+  --ro-bind /lib /lib \
+  --ro-bind /etc/resolv.conf /etc/resolv.conf \
+  --dev-bind /dev /dev \
+  --dir /tmp \
+  --dir /var \
+  --bind $(pwd) $(pwd) \
+  --share-net \
+  --die-with-parent \
+  --file 11 /etc/passwd \
+  --file 12 /etc/group \
+  "$@" \
+  11< <(getent passwd $(id -u) 65534) \
+  12< <(getent group $(id -g) 65534) 

bin/handlers/README.md

@@ -20,6 +20,8 @@ each plugin must define a `main()` which takes 3 arguments;
 
 ## on404
 
+* [redirect.py](redirect.py) sends an HTTP 301 or 302, redirecting the client to another page/file
+* [randpic.py](randpic.py) redirects `/foo/bar/randpic.jpg` to a random pic in `/foo/bar/`
 * [sorry.py](answer.py) replies with a custom message instead of the usual 404
 * [nooo.py](nooo.py) replies with an endless noooooooooooooo
 * [never404.py](never404.py) 100% guarantee that 404 will never be a thing again as it automatically creates dummy files whenever necessary

bin/handlers/randpic.py

@@ -0,0 +1,35 @@
+import os
+import random
+from urllib.parse import quote
+
+
+# assuming /foo/bar/ is a valid URL but /foo/bar/randpic.png does not exist,
+# hijack the 404 with a redirect to a random pic in that folder
+#
+# thx to lia & kipu for the idea
+
+
+def main(cli, vn, rem):
+    req_fn = rem.split("/")[-1]
+    if not cli.can_read or not req_fn.startswith("randpic"):
+        return
+
+    req_abspath = vn.canonical(rem)
+    req_ap_dir = os.path.dirname(req_abspath)
+    files_in_dir = os.listdir(req_ap_dir)
+
+    if "." in req_fn:
+        file_ext = "." + req_fn.split(".")[-1]
+        files_in_dir = [x for x in files_in_dir if x.lower().endswith(file_ext)]
+
+    if not files_in_dir:
+        return
+
+    selected_file = random.choice(files_in_dir)
+
+    req_url = "/".join([vn.vpath, rem]).strip("/")
+    req_dir = req_url.rsplit("/", 1)[0]
+    new_url = "/".join([req_dir, quote(selected_file)]).strip("/")
+
+    cli.reply(b"redirecting...", 302, headers={"Location": "/" + new_url})
+    return "true"

bin/handlers/redirect.py

@@ -0,0 +1,52 @@
+# if someone hits a 404, redirect them to another location
+
+
+def send_http_302_temporary_redirect(cli, new_path):
+    """
+    replies with an HTTP 302, which is a temporary redirect;
+    "new_path" can be any of the following:
+      - "http://a.com/" would redirect to another website,
+      - "/foo/bar" would redirect to /foo/bar on the same server;
+          note the leading '/' in the location which is important
+    """
+    cli.reply(b"redirecting...", 302, headers={"Location": new_path})
+
+
+def send_http_301_permanent_redirect(cli, new_path):
+    """
+    replies with an HTTP 301, which is a permanent redirect;
+    otherwise identical to send_http_302_temporary_redirect
+    """
+    cli.reply(b"redirecting...", 301, headers={"Location": new_path})
+
+
+def send_errorpage_with_redirect_link(cli, new_path):
+    """
+    replies with a website explaining that the page has moved;
+    "new_path" must be an absolute location on the same server
+    but without a leading '/', so for example "foo/bar"
+    would redirect to "/foo/bar"
+    """
+    cli.redirect(new_path, click=False, msg="this page has moved")
+
+
+def main(cli, vn, rem):
+    """
+    this is the function that gets called by copyparty;
+    note that vn.vpath and cli.vpath does not have a leading '/'
+    so we're adding the slash in the debug messages below
+    """
+    print(f"this client just hit a 404: {cli.ip}")
+    print(f"they were accessing this volume: /{vn.vpath}")
+    print(f"and the original request-path (straight from the URL) was /{cli.vpath}")
+    print(f"...which resolves to the following filesystem path: {vn.canonical(rem)}")
+
+    new_path = "/foo/bar/"
+    print(f"will now redirect the client to {new_path}")
+
+    # uncomment one of these:
+    send_http_302_temporary_redirect(cli, new_path)
+    #send_http_301_permanent_redirect(cli, new_path)
+    #send_errorpage_with_redirect_link(cli, new_path)
+
+    return "true"

bin/hooks/README.md

@@ -2,7 +2,7 @@ standalone programs which are executed by copyparty when an event happens (uploa
 
 these programs either take zero arguments, or a filepath (the affected file), or a json message with filepath + additional info
 
-run copyparty with `--help-hooks` for usage details / hook type explanations (xm/xbu/xau/xiu/xbr/xar/xbd/xad/xban)
+run copyparty with `--help-hooks` for usage details / hook type explanations (xm/xbu/xau/xiu/xbc/xac/xbr/xar/xbd/xad/xban)
 
 > **note:** in addition to event hooks (the stuff described here), copyparty has another api to run your programs/scripts while providing way more information such as audio tags / video codecs / etc and optionally daisychaining data between scripts in a processing pipeline; if that's what you want then see [mtp plugins](../mtag/) instead
 
@@ -14,6 +14,8 @@ run copyparty with `--help-hooks` for usage details / hook type explanations (xm
 * [discord-announce.py](discord-announce.py) announces new uploads on discord using webhooks ([example](https://user-images.githubusercontent.com/241032/215304439-1c1cb3c8-ec6f-4c17-9f27-81f969b1811a.png))
 * [reject-mimetype.py](reject-mimetype.py) rejects uploads unless the mimetype is acceptable
 * [into-the-cache-it-goes.py](into-the-cache-it-goes.py) avoids bugs in caching proxies by immediately downloading each file that is uploaded
+* [podcast-normalizer.py](podcast-normalizer.py) creates a second file with dynamic-range-compression whenever an audio file is uploaded
+  * good example of the `idx` [hook effect](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#hook-effects) to tell copyparty about additional files to scan/index
 
 
 # upload batches
@@ -25,9 +27,11 @@ these are `--xiu` hooks; unlike `xbu` and `xau` (which get executed on every sin
 # before upload
 * [reject-extension.py](reject-extension.py) rejects uploads if they match a list of file extensions
 * [reloc-by-ext.py](reloc-by-ext.py) redirects an upload to another destination based on the file extension
+  * good example of the `reloc` [hook effect](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#hook-effects)
 
 
 # on message
 * [wget.py](wget.py) lets you download files by POSTing URLs to copyparty
 * [qbittorrent-magnet.py](qbittorrent-magnet.py) starts downloading a torrent if you post a magnet url
+* [usb-eject.py](usb-eject.py) adds web-UI buttons to safe-remove usb flashdrives shared through copyparty
 * [msg-log.py](msg-log.py) is a guestbook; logs messages to a doc in the same folder

bin/hooks/podcast-normalizer.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+
+import json
+import os
+import sys
+import subprocess as sp
+
+
+_ = r"""
+sends all uploaded audio files through an aggressive
+dynamic-range-compressor to even out the volume levels
+
+dependencies:
+    ffmpeg
+
+being an xau hook, this gets eXecuted After Upload completion
+    but before copyparty has started hashing/indexing the file, so
+    we'll create a second normalized copy in a subfolder and tell
+    copyparty to hash/index that additional file as well
+
+example usage as global config:
+    -e2d -e2t --xau j,c1,bin/hooks/podcast-normalizer.py
+
+parameters explained,
+    e2d/e2t = enable database and metadata indexing
+    xau = execute after upload
+    j   = this hook needs upload information as json (not just the filename)
+    c1  = this hook returns json on stdout, so tell copyparty to read that
+
+example usage as a volflag (per-volume config):
+    -v srv/inc/pods:inc/pods:r:rw,ed:c,xau=j,c1,bin/hooks/podcast-normalizer.py
+                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    (share fs-path srv/inc/pods at URL /inc/pods,
+     readable by all, read-write for user ed,
+     running this xau (exec-after-upload) plugin for all uploaded files)
+
+example usage as a volflag in a copyparty config file:
+    [/inc/pods]
+      srv/inc/pods
+      accs:
+        r: *
+        rw: ed
+      flags:
+        e2d  # enables file indexing
+        e2t  # metadata tags too
+        xau: j,c1,bin/hooks/podcast-normalizer.py
+
+"""
+
+########################################################################
+### CONFIG
+
+# filetypes to process; ignores everything else
+EXTS = "mp3 flac ogg oga opus m4a aac wav wma"
+
+# the name of the subdir to put the normalized files in
+SUBDIR = "normalized"
+
+########################################################################
+
+
+# try to enable support for crazy filenames
+try:
+    from copyparty.util import fsenc
+except:
+
+    def fsenc(p):
+        return p.encode("utf-8")
+
+
+def main():
+    # read info from copyparty
+    inf = json.loads(sys.argv[1])
+    vpath = inf["vp"]
+    abspath = inf["ap"]
+
+    # check if the file-extension is on the to-be-processed list
+    ext = abspath.lower().split(".")[-1]
+    if ext not in EXTS.split():
+        return
+
+    # jump into the folder where the file was uploaded
+    # and create the subfolder to place the normalized copy inside
+    dirpath, filename = os.path.split(abspath)
+    os.chdir(fsenc(dirpath))
+    os.makedirs(SUBDIR, exist_ok=True)
+
+    # the input and output filenames to give ffmpeg
+    fname_in = fsenc(f"./{filename}")
+    fname_out = fsenc(f"{SUBDIR}/{filename}.opus")
+
+    # fmt: off
+    # create and run the ffmpeg command
+    cmd = [
+        b"ffmpeg",
+        b"-nostdin",
+        b"-hide_banner",
+        b"-i", fname_in,
+        b"-af", b"dynaudnorm=f=100:g=9",  # the normalizer config
+        b"-c:a", b"libopus",
+        b"-b:a", b"128k",
+        fname_out,
+    ]
+    # fmt: on
+    sp.check_output(cmd)
+
+    # and finally, tell copyparty about the new file
+    # so it appears in the database and rss-feed:
+    vpath = f"{SUBDIR}/{filename}.opus"
+    print(json.dumps({"idx": {"vp": [vpath]}}))
+
+    # (it's fine to give it a relative path like that; it gets
+    #  resolved relative to the folder the file was uploaded into)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except Exception as ex:
+        print("podcast-normalizer failed; %r" % (ex,))

bin/hooks/reloc-by-ext.py

@@ -71,6 +71,9 @@ def main():
     ## selecting it inside the print at the end:
     ##
 
+    # move all uploads to one specific folder
+    into_junk = {"vp": "/junk"}
+
     # create a subfolder named after the filetype and move it into there
     into_subfolder = {"vp": ext}
 
@@ -92,8 +95,8 @@ def main():
         by_category = {}  # no action
 
     # now choose the default effect to apply; can be any of these:
-    # into_subfolder  into_toplevel  into_sibling  by_category
-    effect = {"vp": "/junk"}
+    # into_junk  into_subfolder  into_toplevel  into_sibling  by_category
+    effect = into_sibling
 
     ##
     ## but we can keep going, adding more speicifc rules

bin/hooks/usb-eject.js

@@ -0,0 +1,62 @@
+// see usb-eject.py for usage
+
+function usbclick() {
+    var o = QS('#treeul a[dst="/usb/"]') || QS('#treepar a[dst="/usb/"]');
+    if (o)
+        o.click();
+}
+
+function eject_cb() {
+    var t = ('' + this.responseText).trim();
+    if (t.indexOf('can be safely unplugged') < 0 && t.indexOf('Device can be removed') < 0)
+        return toast.err(30, 'usb eject failed:\n\n' + t);
+
+    toast.ok(5, esc(t.replace(/ - /g, '\n\n')).trim());
+    usbclick(); setTimeout(usbclick, 10);
+};
+
+function add_eject_2(a) {
+    var aw = a.getAttribute('href').split(/\//g);
+    if (aw.length != 4 || aw[3])
+        return;
+
+    var v = aw[2],
+        k = 'umount_' + v;
+
+    for (var b = 0; b < 9; b++) {
+        var o = ebi(k);
+        if (!o)
+            break;
+        o.parentNode.removeChild(o);
+    }
+
+    a.appendChild(mknod('span', k, '⏏'), a);
+    o = ebi(k);
+    o.style.cssText = 'position:absolute; right:1em; margin-top:-.2em; font-size:1.3em';
+    o.onclick = function (e) {
+        ev(e);
+        var xhr = new XHR();
+        xhr.open('POST', get_evpath(), true);
+        xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded;charset=UTF-8');
+        xhr.send('msg=' + uricom_enc(':usb-eject:' + v + ':'));
+        xhr.onload = xhr.onerror = eject_cb;
+        toast.inf(10, "ejecting " + v + "...");
+    };
+};
+
+function add_eject() {
+    var o = QSA('#treeul a[href^="/usb/"]') || QSA('#treepar a[href^="/usb/"]');
+    for (var a = o.length - 1; a > 0; a--)
+        add_eject_2(o[a]);
+};
+
+(function() {
+    var f0 = treectl.rendertree;
+    treectl.rendertree = function (res, ts, top0, dst, rst) {
+        var ret = f0(res, ts, top0, dst, rst);
+        add_eject();
+        return ret;
+    };
+})();
+
+setTimeout(add_eject, 50);

bin/hooks/usb-eject.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+
+import os
+import stat
+import subprocess as sp
+import sys
+from urllib.parse import unquote_to_bytes as unquote
+
+
+"""
+if you've found yourself using copyparty to serve flashdrives on a LAN
+and your only wish is that the web-UI had a button to unmount / safely
+remove those flashdrives, then boy howdy are you in the right place :D
+
+put usb-eject.js in the webroot (or somewhere else http-accessible)
+then run copyparty with these args:
+
+   -v /run/media/egon:/usb:A:c,hist=/tmp/junk
+   --xm=c1,bin/hooks/usb-eject.py
+   --js-browser=/usb-eject.js
+
+which does the following respectively,
+
+  * share all of /run/media/egon as /usb with admin for everyone
+     and put the histpath somewhere it won't cause trouble
+  * run the usb-eject hook with stdout redirect to the web-ui
+  * add the complementary usb-eject.js to the browser
+
+"""
+
+
+MOUNT_BASE = b"/run/media/egon/"
+
+
+def main():
+    try:
+        label = sys.argv[1].split(":usb-eject:")[1].split(":")[0]
+        mp = MOUNT_BASE + unquote(label)
+        # print("ejecting [%s]... " % (mp,), end="")
+        mp = os.path.abspath(os.path.realpath(mp))
+        st = os.lstat(mp)
+        if not stat.S_ISDIR(st.st_mode) or not mp.startswith(MOUNT_BASE):
+            raise Exception("not a regular directory")
+
+        # if you're running copyparty as root (thx for the faith)
+        # you'll need something like this to make dbus talkative
+        cmd = b"sudo -u egon DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus gio mount -e"
+
+        # but if copyparty and the ui-session is running
+        # as the same user (good) then this is plenty
+        cmd = b"gio mount -e"
+
+        cmd = cmd.split(b" ") + [mp]
+        ret = sp.check_output(cmd).decode("utf-8", "replace")
+        print(ret.strip() or (label + " can be safely unplugged"))
+
+    except Exception as ex:
+        print("unmount failed: %r" % (ex,))
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/README.md

@@ -31,6 +31,9 @@ plugins in this section should only be used with appropriate precautions:
 * [very-bad-idea.py](./very-bad-idea.py) combined with [meadup.js](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/meadup.js) converts copyparty into a janky yet extremely flexible chromecast clone
   * also adds a virtual keyboard by @steinuil to the basic-upload tab for comfy couch crowd control
   * anything uploaded through the [android app](https://github.com/9001/party-up) (files or links) are executed on the server, meaning anyone can infect your PC with malware... so protect this with a password and keep it on a LAN!
+  * [kamelåså](https://github.com/steinuil/kameloso) is a much better (and MUCH safer) alternative to this plugin
+    * powered by [chicken-curry-banana-pineapple-peanut pizza](https://a.ocv.me/pub/g/i/2025/01/298437ce-8351-4c8c-861c-fa131d217999.jpg?cache) so you know it's good
+    * and, unlike this plugin, kamelåså even has windows support (nice)
 
 
 # dependencies

bin/mtag/cksum.py

@@ -2,11 +2,15 @@
 
 import sys
 import json
-import zlib
 import struct
 import base64
 import hashlib
 
+try:
+    from zlib_ng import zlib_ng as zlib
+except:
+    import zlib
+
 try:
     from copyparty.util import fsenc
 except:

bin/mtag/install-deps.sh

@@ -22,6 +22,8 @@ set -e
 #   modifies the keyfinder python lib to load the .so in ~/pe
 
 
+export FORCE_COLOR=1
+
 linux=1
 
 win=
@@ -186,12 +188,15 @@ install_keyfinder() {
 		echo "so not found at $sop"
 		exit 1
 	}
-	
+
+	x=${-//[^x]/}; set -x; cat /etc/alpine-release
 	# rm -rf /Users/ed/Library/Python/3.9/lib/python/site-packages/*keyfinder*
 	CFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
+	CXXFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
 	LDFLAGS="-L$h/pe/keyfinder/lib -L$h/pe/keyfinder/lib64 -L/opt/local/lib" \
-	PKG_CONFIG_PATH=/c/msys64/mingw64/lib/pkgconfig \
+	PKG_CONFIG_PATH="/c/msys64/mingw64/lib/pkgconfig:$h/pe/keyfinder/lib/pkgconfig" \
 	$pybin -m pip install --user keyfinder
+	[ "$x" ] || set +x
 
 	pypath="$($pybin -c 'import keyfinder; print(keyfinder.__file__)')"
 	for pyso in "${pypath%/*}"/*.so; do

bin/mtag/very-bad-idea.py

@@ -6,6 +6,11 @@ WARNING -- DANGEROUS PLUGIN --
   running this plugin, they can execute malware on your machine
   so please keep this on a LAN and protect it with a password
 
+here is a MUCH BETTER ALTERNATIVE (which also works on Windows):
+  https://github.com/steinuil/kameloso
+
+----------------------------------------------------------------------
+
 use copyparty as a chromecast replacement:
   * post a URL and it will open in the default browser
   * upload a file and it will open in the default application

bin/partyfuse.py

@@ -393,7 +393,8 @@ class Gateway(object):
         if r.status != 200:
             self.closeconn()
             info("http error %s reading dir %r", r.status, web_path)
-            raise FuseOSError(errno.ENOENT)
+            err = errno.ENOENT if r.status == 404 else errno.EIO
+            raise FuseOSError(err)
 
         ctype = r.getheader("Content-Type", "")
         if ctype == "application/json":
@@ -1128,7 +1129,7 @@ def main():
 
     # dircache is always a boost,
     #   only want to disable it for tests etc,
-    cdn = 9  # max num dirs; 0=disable
+    cdn = 24  # max num dirs; keep larger than max dir depth; 0=disable
     cds = 1  # numsec until an entry goes stale
 
     where = "local directory"

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "2.1"
-S_BUILD_DT = "2024-09-23"
+S_VERSION = "2.11"
+S_BUILD_DT = "2025-05-18"
 
 """
 u2c.py: upload to copyparty
@@ -52,6 +52,7 @@ if PY2:
 
     sys.dont_write_bytecode = True
     bytes = str
+    files_decoder = lambda s: unicode(s, "utf8")
 else:
     from urllib.parse import quote_from_bytes as quote
     from urllib.parse import unquote_to_bytes as unquote
@@ -61,6 +62,10 @@ else:
     from queue import Queue
 
     unicode = str
+    files_decoder = unicode
+
+
+WTF8 = "replace" if PY2 else "surrogateescape"
 
 VT100 = platform.system() != "Windows"
 
@@ -151,6 +156,7 @@ class HCli(object):
         self.tls = tls
         self.verify = ar.te or not ar.td
         self.conns = []
+        self.hconns = []
         if tls:
             import ssl
 
@@ -170,7 +176,7 @@ class HCli(object):
             "User-Agent": "u2c/%s" % (S_VERSION,),
         }
 
-    def _connect(self):
+    def _connect(self, timeout):
         args = {}
         if PY37:
             args["blocksize"] = 1048576
@@ -182,9 +188,11 @@ class HCli(object):
             if self.ctx:
                 args = {"context": self.ctx}
 
-        return C(self.addr, self.port, timeout=999, **args)
+        return C(self.addr, self.port, timeout=timeout, **args)
 
     def req(self, meth, vpath, hdrs, body=None, ctype=None):
+        now = time.time()
+
         hdrs.update(self.base_hdrs)
         if self.ar.a:
             hdrs["PW"] = self.ar.a
@@ -195,7 +203,11 @@ class HCli(object):
                 0 if not body else body.len if hasattr(body, "len") else len(body)
             )
 
-        c = self.conns.pop() if self.conns else self._connect()
+        # large timeout for handshakes (safededup)
+        conns = self.hconns if ctype == MJ else self.conns
+        while conns and self.ar.cxp < now - conns[0][0]:
+            conns.pop(0)[1].close()
+        c = conns.pop()[1] if conns else self._connect(999 if ctype == MJ else 128)
         try:
             c.request(meth, vpath, body, hdrs)
             if PY27:
@@ -204,8 +216,15 @@ class HCli(object):
                 rsp = c.getresponse()
 
             data = rsp.read()
-            self.conns.append(c)
+            conns.append((time.time(), c))
             return rsp.status, data.decode("utf-8")
+        except http_client.BadStatusLine:
+            if self.ar.cxp > 4:
+                t = "\nWARNING: --cxp probably too high; reducing from %d to 4"
+                print(t % (self.ar.cxp,))
+                self.ar.cxp = 4
+            c.close()
+            raise
         except:
             c.close()
             raise
@@ -217,6 +236,10 @@ CLEN = "Content-Length"
 
 web = None  # type: HCli
 
+links = []  # type: list[str]
+linkmtx = threading.Lock()
+linkfile = None
+
 
 class File(object):
     """an up2k upload task; represents a single file"""
@@ -228,7 +251,7 @@ class File(object):
         self.lmod = lmod  # type: float
 
         self.abs = os.path.join(top, rel)  # type: bytes
-        self.name = self.rel.split(b"/")[-1].decode("utf-8", "replace")  # type: str
+        self.name = self.rel.split(b"/")[-1].decode("utf-8", WTF8)  # type: str
 
         # set by get_hashlist
         self.cids = []  # type: list[tuple[str, int, int]]  # [ hash, ofs, sz ]
@@ -267,10 +290,41 @@ class FileSlice(object):
                 raise Exception(9)
             tlen += clen
 
-        self.len = tlen
+        self.len = self.tlen = tlen
         self.cdr = self.car + self.len
         self.ofs = 0  # type: int
-        self.f = open(file.abs, "rb", 512 * 1024)
+
+        self.f = None
+        self.seek = self._seek0
+        self.read = self._read0
+
+    def subchunk(self, maxsz, nth):
+        if self.tlen <= maxsz:
+            return -1
+
+        if not nth:
+            self.car0 = self.car
+            self.cdr0 = self.cdr
+
+        self.car = self.car0 + maxsz * nth
+        if self.car >= self.cdr0:
+            return -2
+
+        self.cdr = self.car + min(self.cdr0 - self.car, maxsz)
+        self.len = self.cdr - self.car
+        self.seek(0)
+        return nth
+
+    def unsub(self):
+        self.car = self.car0
+        self.cdr = self.cdr0
+        self.len = self.tlen
+
+    def _open(self):
+        self.seek = self._seek
+        self.read = self._read
+
+        self.f = open(self.file.abs, "rb", 512 * 1024)
         self.f.seek(self.car)
 
         # https://stackoverflow.com/questions/4359495/what-is-exactly-a-file-like-object-in-python
@@ -282,10 +336,15 @@ class FileSlice(object):
         except:
             pass  # py27 probably
 
+    def close(self, *a, **ka):
+        return  # until _open
+
     def tell(self):
         return self.ofs
 
-    def seek(self, ofs, wh=0):
+    def _seek(self, ofs, wh=0):
+        assert self.f  # !rm
+
         if wh == 1:
             ofs = self.ofs + ofs
         elif wh == 2:
@@ -299,12 +358,22 @@ class FileSlice(object):
         self.ofs = ofs
         self.f.seek(self.car + ofs)
 
-    def read(self, sz):
+    def _read(self, sz):
+        assert self.f  # !rm
+
         sz = min(sz, self.len - self.ofs)
         ret = self.f.read(sz)
         self.ofs += len(ret)
         return ret
 
+    def _seek0(self, ofs, wh=0):
+        self._open()
+        return self.seek(ofs, wh)
+
+    def _read0(self, sz):
+        self._open()
+        return self.read(sz)
+
 
 class MTHash(object):
     def __init__(self, cores):
@@ -557,13 +626,17 @@ def walkdir(err, top, excl, seen):
     for ap, inf in sorted(statdir(err, top)):
         if excl.match(ap):
             continue
-        yield ap, inf
         if stat.S_ISDIR(inf.st_mode):
+            yield ap, inf
             try:
                 for x in walkdir(err, ap, excl, seen):
                     yield x
             except Exception as ex:
                 err.append((ap, str(ex)))
+        elif stat.S_ISREG(inf.st_mode):
+            yield ap, inf
+        else:
+            err.append((ap, "irregular filetype 0%o" % (inf.st_mode,)))
 
 
 def walkdirs(err, tops, excl):
@@ -609,11 +682,12 @@ def walkdirs(err, tops, excl):
 
 # mostly from copyparty/util.py
 def quotep(btxt):
+    # type: (bytes) -> bytes
     quot1 = quote(btxt, safe=b"/")
     if not PY2:
         quot1 = quot1.encode("ascii")
 
-    return quot1.replace(b" ", b"+")  # type: ignore
+    return quot1.replace(b" ", b"%20")  # type: ignore
 
 
 # from copyparty/util.py
@@ -641,7 +715,7 @@ def up2k_chunksize(filesize):
     while True:
         for mul in [1, 2]:
             nchunks = math.ceil(filesize * 1.0 / chunksize)
-            if nchunks <= 256 or (chunksize >= 32 * 1024 * 1024 and nchunks < 4096):
+            if nchunks <= 256 or (chunksize >= 32 * 1024 * 1024 and nchunks <= 4096):
                 return chunksize
 
             chunksize += stepsize
@@ -693,6 +767,29 @@ def get_hashlist(file, pcb, mth):
             file.kchunks[k] = [v1, v2]
 
 
+def printlink(ar, purl, name, fk):
+    if not name:
+        url = purl  # srch
+    else:
+        name = quotep(name.encode("utf-8", WTF8)).decode("utf-8")
+        if fk:
+            url = "%s%s?k=%s" % (purl, name, fk)
+        else:
+            url = "%s%s" % (purl, name)
+
+    url = "%s/%s" % (ar.burl, url.lstrip("/"))
+
+    with linkmtx:
+        if ar.u:
+            links.append(url)
+        if ar.ud:
+            print(url)
+        if linkfile:
+            zs = "%s\n" % (url,)
+            zb = zs.encode("utf-8", "replace")
+            linkfile.write(zb)
+
+
 def handshake(ar, file, search):
     # type: (argparse.Namespace, File, bool) -> tuple[list[str], bool]
     """
@@ -712,7 +809,9 @@ def handshake(ar, file, search):
     else:
         if ar.touch:
             req["umod"] = True
-        if ar.ow:
+        if ar.owo:
+            req["replace"] = "mt"
+        elif ar.ow:
             req["replace"] = True
 
     file.recheck = False
@@ -720,7 +819,7 @@ def handshake(ar, file, search):
         url = file.url
     else:
         if b"/" in file.rel:
-            url = quotep(file.rel.rsplit(b"/", 1)[0]).decode("utf-8", "replace")
+            url = quotep(file.rel.rsplit(b"/", 1)[0]).decode("utf-8")
         else:
             url = ""
         url = ar.vtop + url
@@ -728,6 +827,7 @@ def handshake(ar, file, search):
     while True:
         sc = 600
         txt = ""
+        t0 = time.time()
         try:
             zs = json.dumps(req, separators=(",\n", ": "))
             sc, txt = web.req("POST", url, {}, zs.encode("utf-8"), MJ)
@@ -752,7 +852,9 @@ def handshake(ar, file, search):
                 print("\nERROR: login required, or wrong password:\n%s" % (txt,))
                 raise BadAuth()
 
-            eprint("handshake failed, retrying: %s\n  %s\n\n" % (file.name, em))
+            t = "handshake failed, retrying: %s\n  t0=%.3f t1=%.3f td=%.3f\n  %s\n\n"
+            now = time.time()
+            eprint(t % (file.name, t0, now, now - t0, em))
             time.sleep(ar.cd)
 
     try:
@@ -761,17 +863,22 @@ def handshake(ar, file, search):
         raise Exception(txt)
 
     if search:
+        if ar.uon and r["hits"]:
+            printlink(ar, r["hits"][0]["rp"], "", "")
         return r["hits"], False
 
-    file.url = r["purl"]
+    file.url = quotep(r["purl"].encode("utf-8", WTF8)).decode("utf-8")
     file.name = r["name"]
     file.wark = r["wark"]
 
+    if ar.uon and not r["hash"]:
+        printlink(ar, file.url, r["name"], r.get("fk"))
+
     return r["hash"], r["sprs"]
 
 
-def upload(fsl, stats):
-    # type: (FileSlice, str) -> None
+def upload(fsl, stats, maxsz):
+    # type: (FileSlice, str, int) -> None
     """upload a range of file data, defined by one or more `cid` (chunk-hash)"""
 
     ctxt = fsl.cids[0]
@@ -789,21 +896,34 @@ def upload(fsl, stats):
     if stats:
         headers["X-Up2k-Stat"] = stats
 
+    nsub = 0
     try:
-        sc, txt = web.req("POST", fsl.file.url, headers, fsl, MO)
+        while nsub != -1:
+            nsub = fsl.subchunk(maxsz, nsub)
+            if nsub == -2:
+                return
+            if nsub >= 0:
+                headers["X-Up2k-Subc"] = str(maxsz * nsub)
+                headers.pop(CLEN, None)
+                nsub += 1
 
-        if sc == 400:
-            if (
-                "already being written" in txt
-                or "already got that" in txt
-                or "only sibling chunks" in txt
-            ):
-                fsl.file.nojoin = 1
+            sc, txt = web.req("POST", fsl.file.url, headers, fsl, MO)
 
-        if sc >= 400:
-            raise Exception("http %s: %s" % (sc, txt))
+            if sc == 400:
+                if (
+                    "already being written" in txt
+                    or "already got that" in txt
+                    or "only sibling chunks" in txt
+                ):
+                    fsl.file.nojoin = 1
+
+            if sc >= 400:
+                raise Exception("http %s: %s" % (sc, txt))
     finally:
-        fsl.f.close()
+        if fsl.f:
+            fsl.f.close()
+            if nsub != -1:
+                fsl.unsub()
 
 
 class Ctl(object):
@@ -869,8 +989,8 @@ class Ctl(object):
             self.hash_b = 0
             self.up_f = 0
             self.up_c = 0
-            self.up_b = 0
-            self.up_br = 0
+            self.up_b = 0  # num bytes handled
+            self.up_br = 0  # num bytes actually transferred
             self.uploader_busy = 0
             self.serialized = False
 
@@ -935,7 +1055,7 @@ class Ctl(object):
                     print("  %d up %s" % (ncs - nc, cid))
                     stats = "%d/0/0/%d" % (nf, self.nfiles - nf)
                     fslice = FileSlice(file, [cid])
-                    upload(fslice, stats)
+                    upload(fslice, stats, self.ar.szm)
 
             print("  ok!")
             if file.recheck:
@@ -949,8 +1069,8 @@ class Ctl(object):
             handshake(self.ar, file, False)
 
     def _fancy(self):
+        atexit.register(self.cleanup_vt100)
         if VT100 and not self.ar.ns:
-            atexit.register(self.cleanup_vt100)
             ss.scroll_region(3)
 
         Daemon(self.hasher)
@@ -958,6 +1078,7 @@ class Ctl(object):
             Daemon(self.handshaker)
             Daemon(self.uploader)
 
+        last_sp = -1
         while True:
             with self.exit_cond:
                 self.exit_cond.wait(0.07)
@@ -996,6 +1117,12 @@ class Ctl(object):
             else:
                 txt = " "
 
+            if not VT100:  # OSC9;4 (taskbar-progress)
+                sp = int(self.up_b * 100 / self.nbytes) or 1
+                if last_sp != sp:
+                    last_sp = sp
+                    txt += "\033]9;4;1;%d\033\\" % (sp,)
+
             if not self.up_br:
                 spd = self.hash_b / ((time.time() - self.t0) or 1)
                 eta = (self.nbytes - self.hash_b) / (spd or 1)
@@ -1006,18 +1133,25 @@ class Ctl(object):
 
             spd = humansize(spd)
             self.eta = str(datetime.timedelta(seconds=int(eta)))
+            if eta > 2591999:
+                self.eta = self.eta.split(",")[0]  # truncate HH:MM:SS
             sleft = humansize(self.nbytes - self.up_b)
             nleft = self.nfiles - self.up_f
             tail = "\033[K\033[u" if VT100 and not self.ar.ns else "\r"
 
-            t = "%s eta @ %s/s, %s, %d# left\033[K" % (self.eta, spd, sleft, nleft)
+            t = "%s eta @ %s/s, %s, %d# left" % (self.eta, spd, sleft, nleft)
+            if not self.hash_b:
+                t = " now hashing..."
             eprint(txt + "\033]0;{0}\033\\\r{0}{1}".format(t, tail))
 
+        if self.ar.wlist:
+            self.at_hash = time.time() - self.t0
+
         if self.hash_b and self.at_hash:
             spd = humansize(self.hash_b / self.at_hash)
             eprint("\nhasher: %.2f sec, %s/s\n" % (self.at_hash, spd))
-        if self.up_b and self.at_up:
-            spd = humansize(self.up_b / self.at_up)
+        if self.up_br and self.at_up:
+            spd = humansize(self.up_br / self.at_up)
             eprint("upload: %.2f sec, %s/s\n" % (self.at_up, spd))
 
         if not self.recheck:
@@ -1028,7 +1162,10 @@ class Ctl(object):
             handshake(self.ar, file, False)
 
     def cleanup_vt100(self):
-        ss.scroll_region(None)
+        if VT100:
+            ss.scroll_region(None)
+        else:
+            eprint("\033]9;4;0\033\\")
         eprint("\033[J\033]0;\033\\")
 
     def cb_hasher(self, file, ofs):
@@ -1043,7 +1180,9 @@ class Ctl(object):
             isdir = stat.S_ISDIR(inf.st_mode)
             if self.ar.z or self.ar.drd:
                 rd = rel if isdir else os.path.dirname(rel)
-                srd = rd.decode("utf-8", "replace").replace("\\", "/")
+                srd = rd.decode("utf-8", "replace").replace("\\", "/").rstrip("/")
+                if srd:
+                    srd += "/"
                 if prd != rd:
                     prd = rd
                     ls = {}
@@ -1051,7 +1190,7 @@ class Ctl(object):
                         print("      ls ~{0}".format(srd))
                         zt = (
                             self.ar.vtop,
-                            quotep(rd.replace(b"\\", b"/")).decode("utf-8", "replace"),
+                            quotep(rd.replace(b"\\", b"/")).decode("utf-8"),
                         )
                         sc, txt = web.req("GET", "%s%s?ls&lt&dots" % zt, {})
                         if sc >= 400:
@@ -1060,13 +1199,16 @@ class Ctl(object):
                         j = json.loads(txt)
                         for f in j["dirs"] + j["files"]:
                             rfn = f["href"].split("?")[0].rstrip("/")
-                            ls[unquote(rfn.encode("utf-8", "replace"))] = f
+                            ls[unquote(rfn.encode("utf-8", WTF8))] = f
                     except Exception as ex:
                         print("   mkdir ~{0}  ({1})".format(srd, ex))
 
                     if self.ar.drd:
                         dp = os.path.join(top, rd)
-                        lnodes = set(os.listdir(dp))
+                        try:
+                            lnodes = set(os.listdir(dp))
+                        except:
+                            lnodes = list(ls)  # fs eio; don't delete
                         if ptn:
                             zs = dp.replace(sep, b"/").rstrip(b"/") + b"/"
                             zls = [zs + x for x in lnodes]
@@ -1074,12 +1216,12 @@ class Ctl(object):
                             lnodes = [x.split(b"/")[-1] for x in zls]
                         bnames = [x for x in ls if x not in lnodes and x != b".hist"]
                         vpath = self.ar.url.split("://")[-1].split("/", 1)[-1]
-                        names = [x.decode("utf-8", "replace") for x in bnames]
-                        locs = [vpath + srd + "/" + x for x in names]
+                        names = [x.decode("utf-8", WTF8) for x in bnames]
+                        locs = [vpath + srd + x for x in names]
                         while locs:
                             req = locs
                             while req:
-                                print("DELETING ~%s/#%s" % (srd, len(req)))
+                                print("DELETING ~%s#%s" % (srd, len(req)))
                                 body = json.dumps(req).encode("utf-8")
                                 sc, txt = web.req(
                                     "POST", self.ar.url + "?delete", {}, body, MJ
@@ -1136,14 +1278,20 @@ class Ctl(object):
                     self.up_b = self.hash_b
 
             if self.ar.wlist:
-                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                vp = file.rel.decode("utf-8")
+                if self.ar.chs:
+                    zsl = [
+                        "%s %d %d" % (zsii[0], n, zsii[1])
+                        for n, zsii in enumerate(file.cids)
+                    ]
+                    print("chs: %s\n%s" % (vp, "\n".join(zsl)))
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.cids]
                 zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
                 wark = ub64enc(zb).decode("utf-8")
-                vp = file.rel.decode("utf-8")
                 if self.ar.jw:
                     print("%s  %s" % (wark, vp))
                 else:
-                    zd = datetime.datetime.fromtimestamp(file.lmod, UTC)
+                    zd = datetime.datetime.fromtimestamp(max(0, file.lmod), UTC)
                     dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
                         zd.year,
                         zd.month,
@@ -1177,6 +1325,7 @@ class Ctl(object):
                 self.q_upload.put(None)
                 return
 
+            chunksz = up2k_chunksize(file.size)
             upath = file.abs.decode("utf-8", "replace")
             if not VT100:
                 upath = upath.lstrip("\\?")
@@ -1236,9 +1385,14 @@ class Ctl(object):
                     file.up_c -= len(hs)
                     for cid in hs:
                         sz = file.kchunks[cid][1]
+                        self.up_br -= sz
                         self.up_b -= sz
                         file.up_b -= sz
 
+                if hs and not file.up_b:
+                    # first hs of this file; is this an upload resume?
+                    file.up_b = chunksz * max(0, len(file.kchunks) - len(hs))
+
                 file.ucids = hs
 
             if not hs:
@@ -1252,7 +1406,7 @@ class Ctl(object):
                         c1 = c2 = ""
 
                     spd_h = humansize(file.size / file.t_hash, True)
-                    if file.up_b:
+                    if file.up_c:
                         t_up = file.t1_up - file.t0_up
                         spd_u = humansize(file.size / t_up, True)
 
@@ -1262,14 +1416,13 @@ class Ctl(object):
                         t = "   found %s %s(%.2fs,%s/s)%s"
                         print(t % (upath, c1, file.t_hash, spd_h, c2))
                 else:
-                    kw = "uploaded" if file.up_b else "   found"
+                    kw = "uploaded" if file.up_c else "   found"
                     print("{0} {1}".format(kw, upath))
 
                 self._check_if_done()
                 continue
 
-            chunksz = up2k_chunksize(file.size)
-            njoin = (self.ar.sz * 1024 * 1024) // chunksz
+            njoin = self.ar.sz // chunksz
             cs = hs[:]
             while cs:
                 fsl = FileSlice(file, cs[:1])
@@ -1321,7 +1474,7 @@ class Ctl(object):
             )
 
             try:
-                upload(fsl, stats)
+                upload(fsl, stats, self.ar.szm)
             except Exception as ex:
                 t = "upload failed, retrying: %s #%s+%d (%s)\n"
                 eprint(t % (file.name, cids[0][:8], len(cids) - 1, ex))
@@ -1355,7 +1508,7 @@ class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFor
 
 
 def main():
-    global web
+    global web, linkfile
 
     time.strptime("19970815", "%Y%m%d")  # python#7980
     "".encode("idna")  # python#29288
@@ -1365,7 +1518,7 @@ def main():
     cores = (os.cpu_count() if hasattr(os, "cpu_count") else 0) or 2
     hcores = min(cores, 3)  # 4% faster than 4+ on py3.9 @ r5-4500U
 
-    ver = "{0}  v{1}  https://youtu.be/BIcOO6TLKaY".format(S_BUILD_DT, S_VERSION)
+    ver = "{0}, v{1}".format(S_BUILD_DT, S_VERSION)
     if "--version" in sys.argv:
         print(ver)
         return
@@ -1381,7 +1534,7 @@ source file/folder selection uses rsync syntax, meaning that:
 """)
 
     ap.add_argument("url", type=unicode, help="server url, including destination folder")
-    ap.add_argument("files", type=unicode, nargs="+", help="files and/or folders to process")
+    ap.add_argument("files", type=files_decoder, nargs="+", help="files and/or folders to process")
     ap.add_argument("-v", action="store_true", help="verbose")
     ap.add_argument("-a", metavar="PASSWD", help="password or $filepath")
     ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
@@ -1389,9 +1542,15 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
     ap.add_argument("--touch", action="store_true", help="if last-modified timestamps differ, push local to server (need write+delete perms)")
     ap.add_argument("--ow", action="store_true", help="overwrite existing files instead of autorenaming")
+    ap.add_argument("--owo", action="store_true", help="overwrite existing files if server-file is older")
     ap.add_argument("--spd", action="store_true", help="print speeds for each file")
     ap.add_argument("--version", action="store_true", help="show version and exit")
 
+    ap = app.add_argument_group("print links")
+    ap.add_argument("-u", action="store_true", help="print list of download-links after all uploads finished")
+    ap.add_argument("-ud", action="store_true", help="print download-link after each upload finishes")
+    ap.add_argument("-uf", type=unicode, metavar="PATH", help="print list of download-links to file")
+
     ap = app.add_argument_group("compatibility")
     ap.add_argument("--cls", action="store_true", help="clear screen before start")
     ap.add_argument("--rh", type=int, metavar="TRIES", default=0, help="resolve server hostname before upload (good for buggy networks, but TLS certs will break)")
@@ -1403,14 +1562,17 @@ source file/folder selection uses rsync syntax, meaning that:
 
     ap = app.add_argument_group("file-ID calculator; enable with url '-' to list warks (file identifiers) instead of upload/search")
     ap.add_argument("--wsalt", type=unicode, metavar="S", default="hunter2", help="salt to use when creating warks; must match server config")
+    ap.add_argument("--chs", action="store_true", help="verbose (print the hash/offset of each chunk in each file)")
     ap.add_argument("--jw", action="store_true", help="just identifier+filepath, not mtime/size too")
 
     ap = app.add_argument_group("performance tweaks")
     ap.add_argument("-j", type=int, metavar="CONNS", default=2, help="parallel connections")
     ap.add_argument("-J", type=int, metavar="CORES", default=hcores, help="num cpu-cores to use for hashing; set 0 or 1 for single-core hashing")
     ap.add_argument("--sz", type=int, metavar="MiB", default=64, help="try to make each POST this big")
+    ap.add_argument("--szm", type=int, metavar="MiB", default=96, help="max size of each POST (default is cloudflare max)")
     ap.add_argument("-nh", action="store_true", help="disable hashing while uploading")
     ap.add_argument("-ns", action="store_true", help="no status panel (for slow consoles and macos)")
+    ap.add_argument("--cxp", type=float, metavar="SEC", default=57, help="assume http connections expired after SEConds")
     ap.add_argument("--cd", type=float, metavar="SEC", default=5, help="delay before reattempting a failed handshake/upload")
     ap.add_argument("--safe", action="store_true", help="use simple fallback approach")
     ap.add_argument("-z", action="store_true", help="ZOOMIN' (skip uploading files if they exist at the destination with the ~same last-modified timestamp, so same as yolo / turbo with date-chk but even faster)")
@@ -1430,15 +1592,54 @@ source file/folder selection uses rsync syntax, meaning that:
             except:
                 pass
 
+    # msys2 doesn't uncygpath absolute paths with whitespace
+    if not VT100:
+        zsl = []
+        for fn in ar.files:
+            if re.search("^/[a-z]/", fn):
+                fn = r"%s:\%s" % (fn[1:2], fn[3:])
+            zsl.append(fn.replace("/", "\\"))
+        ar.files = zsl
+
+    fok = []
+    fng = []
+    for fn in ar.files:
+        if os.path.exists(fn):
+            fok.append(fn)
+        elif VT100:
+            fng.append(fn)
+        else:
+            # windows leaves glob-expansion to the invoked process... okayyy let's get to work
+            from glob import glob
+
+            fns = glob(fn)
+            if fns:
+                fok.extend(fns)
+            else:
+                fng.append(fn)
+
+    if fng:
+        t = "some files/folders were not found:\n  %s"
+        raise Exception(t % ("\n  ".join(fng),))
+
+    ar.files = fok
+
     if ar.drd:
         ar.dr = True
 
     if ar.dr:
         ar.ow = True
 
+    ar.sz *= 1024 * 1024
+    ar.szm *= 1024 * 1024
+
     ar.x = "|".join(ar.x or [])
 
     setattr(ar, "wlist", ar.url == "-")
+    setattr(ar, "uon", ar.u or ar.ud or ar.uf)
+
+    if ar.uf:
+        linkfile = open(ar.uf, "wb")
 
     for k in "dl dr drd wlist".split():
         errs = []
@@ -1501,6 +1702,12 @@ source file/folder selection uses rsync syntax, meaning that:
         ar.z = True
         ctl = Ctl(ar, ctl.stats)
 
+    if links:
+        print()
+        print("\n".join(links))
+    if linkfile:
+        linkfile.close()
+
     if ctl.errs:
         print("WARNING: %d errors" % (ctl.errs))
 

bin/zmq-recv.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+
+import sys
+import zmq
+
+"""
+zmq-recv.py: demo zmq receiver
+2025-01-22, v1.0, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py
+
+basic zmq-server to receive events from copyparty; try one of
+the below and then "send a message to serverlog" in the web-ui:
+
+1) dumb fire-and-forget to any and all listeners;
+run this script with "sub" and run copyparty with this:
+  --xm zmq:pub:tcp://*:5556
+
+2) one lucky listener gets the message, blocks if no listeners:
+run this script with "pull" and run copyparty with this:
+  --xm t3,zmq:push:tcp://*:5557
+
+3) blocking syn/ack mode, client must ack each message;
+run this script with "rep" and run copyparty with this:
+  --xm t3,zmq:req:tcp://localhost:5555
+
+note: to conditionally block uploads based on message contents,
+use rep_server to answer with "return 1" and run copyparty with
+  --xau t3,c,zmq:req:tcp://localhost:5555
+"""
+
+
+ctx = zmq.Context()
+
+
+def sub_server():
+    # PUB/SUB allows any number of servers/clients, and
+    # messages are fire-and-forget
+    sck = ctx.socket(zmq.SUB)
+    sck.connect("tcp://localhost:5556")
+    sck.setsockopt_string(zmq.SUBSCRIBE, "")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def pull_server():
+    # PUSH/PULL allows any number of servers/clients, and
+    # each message is sent to a exactly one PULL client
+    sck = ctx.socket(zmq.PULL)
+    sck.connect("tcp://localhost:5557")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def rep_server():
+    # REP/REQ is a server/client pair where each message must be
+    # acked by the other before another message can be sent, so
+    # copyparty will do a blocking-wait for the ack
+    sck = ctx.socket(zmq.REP)
+    sck.bind("tcp://*:5555")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+        reply = b"thx"
+        # reply = b"return 1"  # non-zero to block an upload
+        sck.send(reply)
+
+
+mode = sys.argv[1].lower() if len(sys.argv) > 1 else ""
+
+if mode == "sub":
+    sub_server()
+elif mode == "pull":
+    pull_server()
+elif mode == "rep":
+    rep_server()
+else:
+    print("specify mode as first argument:  SUB | PULL | REP")

contrib/README.md

@@ -12,14 +12,19 @@
 * assumes the webserver and copyparty is running on the same server/IP
 * modify `10.13.1.1` as necessary if you wish to support browsers without javascript
 
-### [`sharex.sxcu`](sharex.sxcu)
-* sharex config file to upload screenshots and grab the URL
+### [`sharex.sxcu`](sharex.sxcu) - Windows screenshot uploader
+* [sharex](https://getsharex.com/) config file to upload screenshots and grab the URL
 * `RequestURL`: full URL to the target folder
 * `pw`: password (remove the `pw` line if anon-write)
 * the `act:bput` thing is optional since copyparty v1.9.29
 * using an older sharex version, maybe sharex v12.1.1 for example? dw fam i got your back 👉😎👉 [`sharex12.sxcu`](sharex12.sxcu)
 
-### [`flameshot.sh`](flameshot.sh)
+### [`ishare.iscu`](ishare.iscu) - MacOS screenshot uploader
+* [ishare](https://isharemac.app/) config file to upload screenshots and grab the URL
+* `RequestURL`: full URL to the target folder
+* `pw`: password (remove the `pw` line if anon-write)
+
+### [`flameshot.sh`](flameshot.sh) - Linux screenshot uploader
 * takes a screenshot with [flameshot](https://flameshot.org/) on Linux, uploads it, and writes the URL to clipboard
 
 ### [`send-to-cpp.contextlet.json`](send-to-cpp.contextlet.json)
@@ -45,6 +50,9 @@
 * give a 3rd argument to install it to your copyparty config
 * systemd service at [`systemd/cfssl.service`](systemd/cfssl.service)
 
+### [`zfs-tune.py`](zfs-tune.py)
+* optimizes databases for optimal performance when stored on a zfs filesystem; also see [openzfs docs](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#database-workloads) and specifically the SQLite subsection
+
 # OS integration
 init-scripts to start copyparty as a service
 * [`systemd/copyparty.service`](systemd/copyparty.service) runs the sfx normally
@@ -53,5 +61,10 @@ init-scripts to start copyparty as a service
 * [`openrc/copyparty`](openrc/copyparty)
 
 # Reverse-proxy
-copyparty has basic support for running behind another webserver
-* [`nginx/copyparty.conf`](nginx/copyparty.conf)
+copyparty supports running behind another webserver
+* [`apache/copyparty.conf`](apache/copyparty.conf)
+* [`haproxy/copyparty.conf`](haproxy/copyparty.conf)
+* [`lighttpd/subdomain.conf`](lighttpd/subdomain.conf)
+* [`lighttpd/subpath.conf`](lighttpd/subpath.conf)
+* [`nginx/copyparty.conf`](nginx/copyparty.conf) -- recommended
+* [`traefik/copyparty.yaml`](traefik/copyparty.yaml)

contrib/apache/copyparty.conf

@@ -1,14 +1,29 @@
-# when running copyparty behind a reverse proxy,
-# the following arguments are recommended:
+# if you would like to use unix-sockets (recommended),
+# you must run copyparty with one of the following:
 #
-#   -i 127.0.0.1    only accept connections from nginx
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
 #
 # if you are doing location-based proxying (such as `/stuff` below)
 # you must run copyparty with --rp-loc=stuff
 #
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
+
 LoadModule proxy_module modules/mod_proxy.so
-ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"
-# do not specify ProxyPassReverse
+
 RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+# NOTE: do not specify ProxyPassReverse
+
+
+##
+## then, enable one of the below:
+
+# use subdomain proxying to unix-socket (best)
+ProxyPass "/" "unix:///dev/shm/party.sock|http://whatever/"
+
+# use subdomain proxying to 127.0.0.1 (slower)
+#ProxyPass "/" "http://127.0.0.1:3923/"
+
+# use subpath proxying to 127.0.0.1 (slow and maybe buggy)
+#ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"

contrib/haproxy/copyparty.conf

@@ -0,0 +1,24 @@
+# this config is essentially two separate examples;
+#
+#   foo1 connects to copyparty using tcp, and
+#   foo2 uses unix-sockets for 27% higher performance
+#
+# to use foo2 you must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+
+defaults
+  mode http
+  option forwardfor
+  timeout connect 1s
+  timeout client 610s
+  timeout server 610s
+
+listen foo1
+  bind *:8081
+  server srv1 127.0.0.1:3923 maxconn 512
+
+listen foo2
+  bind *:8082
+  server srv1 /dev/shm/party.sock maxconn 512

contrib/ishare.iscu

@@ -0,0 +1,10 @@
+{
+  "Name": "copyparty",
+  "RequestURL": "http://127.0.0.1:3923/screenshots/",
+  "Headers": {
+    "pw": "PUT_YOUR_PASSWORD_HERE_MY_DUDE",
+    "accept": "json"
+  },
+  "FileFormName": "f",
+  "ResponseURL": "{{fileurl}}"
+}

contrib/lighttpd/subdomain.conf

@@ -0,0 +1,24 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subdomain.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+proxy.forwarded = ( "for" => 1, "proto" => 1 )
+proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+# if you really need to use tcp instead of unix-sockets, do this instead:
+#proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )

contrib/lighttpd/subpath.conf

@@ -0,0 +1,31 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subpath.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# also since this example proxies a subpath instead of the
+# recommended subdomain-proxying, we must also specify this:
+#
+#   --rp-loc files
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+$HTTP["url"] =~ "^/files" {
+    proxy.forwarded = ( "for" => 1, "proto" => 1 )
+    proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+    # if you really need to use tcp instead of unix-sockets, do this instead:
+    #proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )
+}

contrib/nginx/copyparty.conf

@@ -2,19 +2,38 @@
 # not accept more consecutive clients than what copyparty is able to;
 # nginx default is 512  (worker_processes 1, worker_connections 512)
 #
+# ======================================================================
+#
+# to reverse-proxy a specific path/subpath/location below a domain
+# (rather than a complete subdomain), for example "/qw/er", you must
+# run copyparty with --rp-loc /qw/as and also change the following:
+# 	location / {
+# 		proxy_pass http://cpp_tcp;
+# to this:
+# 	location /qw/er/ {
+# 		proxy_pass http://cpp_tcp/qw/er/;
+#
+# ======================================================================
+#
 # rarely, in some extreme usecases, it can be good to add -j0
 # (40'000 requests per second, or 20gbps upload/download in parallel)
 # but this is usually counterproductive and slightly buggy
 #
+# ======================================================================
+#
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 #
-# if you are behind cloudflare (or another protection service),
+# ======================================================================
+#
+# if you are behind cloudflare (or another CDN/WAF/protection service),
 # remember to reject all connections which are not coming from your
 # protection service -- for cloudflare in particular, you can
 # generate the list of permitted IP ranges like so:
 #   (curl -s https://www.cloudflare.com/ips-v{4,6} | sed 's/^/allow /; s/$/;/'; echo; echo "deny all;") > /etc/nginx/cloudflare-only.conf
 #
 # and then enable it below by uncomenting the cloudflare-only.conf line
+#
+# ======================================================================
 
 
 upstream cpp_tcp {
@@ -36,9 +55,9 @@ upstream cpp_uds {
 	# but there must be at least one unix-group which both
 	# nginx and copyparty is a member of; if that group is
 	# "www" then run copyparty with the following args:
-	# -i unix:770:www:/tmp/party.sock
+	# -i unix:770:www:/dev/shm/party.sock
 
-	server unix:/tmp/party.sock fail_timeout=1s;
+	server unix:/dev/shm/party.sock fail_timeout=1s;
 	keepalive 1;
 }
 
@@ -61,14 +80,18 @@ server {
 		client_max_body_size 0;
 		proxy_buffering off;
 		proxy_request_buffering off;
+		# improve download speed from 600 to 1500 MiB/s
+		proxy_buffers 32 8k;
+		proxy_buffer_size 16k;
+		proxy_busy_buffers_size 24k;
 
+		proxy_set_header   Connection        "Keep-Alive";
 		proxy_set_header   Host              $host;
 		proxy_set_header   X-Real-IP         $remote_addr;
-		proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
-		# NOTE: with cloudflare you want this instead:
-		#proxy_set_header   X-Forwarded-For   $http_cf_connecting_ip;
 		proxy_set_header   X-Forwarded-Proto $scheme;
-		proxy_set_header   Connection        "Keep-Alive";
+		proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
+		# NOTE: with cloudflare you want this X-Forwarded-For instead:
+		#proxy_set_header   X-Forwarded-For   $http_cf_connecting_ip;
 	}
 }
 

contrib/nixos/modules/copyparty.nix

@@ -1,23 +1,28 @@
-{ config, pkgs, lib, ... }:
-
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
 with lib;
-
 let
-  mkKeyValue = key: value:
+  mkKeyValue =
+    key: value:
     if value == true then
-    # sets with a true boolean value are coerced to just the key name
+      # sets with a true boolean value are coerced to just the key name
       key
     else if value == false then
-    # or omitted completely when false
+      # or omitted completely when false
       ""
     else
       (generators.mkKeyValueDefault { inherit mkValueString; } ": " key value);
 
   mkAttrsString = value: (generators.toKeyValue { inherit mkKeyValue; } value);
 
-  mkValueString = value:
+  mkValueString =
+    value:
     if isList value then
-      (concatStringsSep ", " (map mkValueString value))
+      (concatStringsSep "," (map mkValueString value))
     else if isAttrs value then
       "\n" + (mkAttrsString value)
     else
@@ -49,13 +54,14 @@ let
     ${concatStringsSep "\n" (mapAttrsToList mkVolume cfg.volumes)}
   '';
 
-  name = "copyparty";
   cfg = config.services.copyparty;
-  configFile = pkgs.writeText "${name}.conf" configStr;
-  runtimeConfigPath = "/run/${name}/${name}.conf";
-  home = "/var/lib/${name}";
-  defaultShareDir = "${home}/data";
-in {
+  configFile = pkgs.writeText "copyparty.conf" configStr;
+  runtimeConfigPath = "/run/copyparty/copyparty.conf";
+  externalCacheDir = "/var/cache/copyparty";
+  externalStateDir = "/var/lib/copyparty";
+  defaultShareDir = "${externalStateDir}/data";
+in
+{
   options.services.copyparty = {
     enable = mkEnableOption "web-based file manager";
 
@@ -68,6 +74,35 @@ in {
       '';
     };
 
+    mkHashWrapper = mkOption {
+      type = types.bool;
+      default = true;
+      description = ''
+        Make a shell script wrapper called 'copyparty-hash' with all options set here,
+        that launches the hashing cli.
+      '';
+    };
+
+    user = mkOption {
+      type = types.str;
+      default = "copyparty";
+      description = ''
+        The user that copyparty will run under.
+
+        If changed from default, you are responsible for making sure the user exists.
+      '';
+    };
+
+    group = mkOption {
+      type = types.str;
+      default = "copyparty";
+      description = ''
+        The group that copyparty will run under.
+
+        If changed from default, you are responsible for making sure the user exists.
+      '';
+    };
+
     openFilesLimit = mkOption {
       default = 4096;
       type = types.either types.int types.str;
@@ -79,33 +114,41 @@ in {
       description = ''
         Global settings to apply.
         Directly maps to values in the [global] section of the copyparty config.
+        Cannot set "c" or "hist", those are set by this module.
         See `${getExe cfg.package} --help` for more details.
       '';
       default = {
         i = "127.0.0.1";
         no-reload = true;
+        hist = externalCacheDir;
       };
       example = literalExpression ''
         {
           i = "0.0.0.0";
           no-reload = true;
+          hist = ${externalCacheDir};
         }
       '';
     };
 
     accounts = mkOption {
-      type = types.attrsOf (types.submodule ({ ... }: {
-        options = {
-          passwordFile = mkOption {
-            type = types.str;
-            description = ''
-              Runtime file path to a file containing the user password.
-              Must be readable by the copyparty user.
-            '';
-            example = "/run/keys/copyparty/ed";
-          };
-        };
-      }));
+      type = types.attrsOf (
+        types.submodule (
+          { ... }:
+          {
+            options = {
+              passwordFile = mkOption {
+                type = types.str;
+                description = ''
+                  Runtime file path to a file containing the user password.
+                  Must be readable by the copyparty user.
+                '';
+                example = "/run/keys/copyparty/ed";
+              };
+            };
+          }
+        )
+      );
       description = ''
         A set of copyparty accounts to create.
       '';
@@ -118,74 +161,81 @@ in {
     };
 
     volumes = mkOption {
-      type = types.attrsOf (types.submodule ({ ... }: {
-        options = {
-          path = mkOption {
-            type = types.str;
-            description = ''
-              Path of a directory to share.
-            '';
-          };
-          access = mkOption {
-            type = types.attrs;
-            description = ''
-              Attribute list of permissions and the users to apply them to.
-
-              The key must be a string containing any combination of allowed permission:
-                "r" (read):   list folder contents, download files
-                "w" (write):  upload files; need "r" to see the uploads
-                "m" (move):   move files and folders; need "w" at destination
-                "d" (delete): permanently delete files and folders
-                "g" (get):    download files, but cannot see folder contents
-                "G" (upget):  "get", but can see filekeys of their own uploads
-                "h" (html):   "get", but folders return their index.html
-                "a" (admin):  can see uploader IPs, config-reload
-
-              For example: "rwmd"
-
-              The value must be one of:
-                an account name, defined in `accounts`
-                a list of account names
-                "*", which means "any account"
-            '';
-            example = literalExpression ''
-              {
-                # wG = write-upget = see your own uploads only
-                wG = "*";
-                # read-write-modify-delete for users "ed" and "k"
-                rwmd = ["ed" "k"];
+      type = types.attrsOf (
+        types.submodule (
+          { ... }:
+          {
+            options = {
+              path = mkOption {
+                type = types.path;
+                description = ''
+                  Path of a directory to share.
+                '';
               };
-            '';
-          };
-          flags = mkOption {
-            type = types.attrs;
-            description = ''
-              Attribute list of volume flags to apply.
-              See `${getExe cfg.package} --help-flags` for more details.
-            '';
-            example = literalExpression ''
-              {
-                # "fk" enables filekeys (necessary for upget permission) (4 chars long)
-                fk = 4;
-                # scan for new files every 60sec
-                scan = 60;
-                # volflag "e2d" enables the uploads database
-                e2d = true;
-                # "d2t" disables multimedia parsers (in case the uploads are malicious)
-                d2t = true;
-                # skips hashing file contents if path matches *.iso
-                nohash = "\.iso$";
+              access = mkOption {
+                type = types.attrs;
+                description = ''
+                  Attribute list of permissions and the users to apply them to.
+
+                  The key must be a string containing any combination of allowed permission:
+                    "r" (read):   list folder contents, download files
+                    "w" (write):  upload files; need "r" to see the uploads
+                    "m" (move):   move files and folders; need "w" at destination
+                    "d" (delete): permanently delete files and folders
+                    "g" (get):    download files, but cannot see folder contents
+                    "G" (upget):  "get", but can see filekeys of their own uploads
+                    "h" (html):   "get", but folders return their index.html
+                    "a" (admin):  can see uploader IPs, config-reload
+
+                  For example: "rwmd"
+
+                  The value must be one of:
+                    an account name, defined in `accounts`
+                    a list of account names
+                    "*", which means "any account"
+                '';
+                example = literalExpression ''
+                  {
+                    # wG = write-upget = see your own uploads only
+                    wG = "*";
+                    # read-write-modify-delete for users "ed" and "k"
+                    rwmd = ["ed" "k"];
+                  };
+                '';
               };
-            '';
-            default = { };
-          };
-        };
-      }));
+              flags = mkOption {
+                type = types.attrs;
+                description = ''
+                  Attribute list of volume flags to apply.
+                  See `${getExe cfg.package} --help-flags` for more details.
+                '';
+                example = literalExpression ''
+                  {
+                    # "fk" enables filekeys (necessary for upget permission) (4 chars long)
+                    fk = 4;
+                    # scan for new files every 60sec
+                    scan = 60;
+                    # volflag "e2d" enables the uploads database
+                    e2d = true;
+                    # "d2t" disables multimedia parsers (in case the uploads are malicious)
+                    d2t = true;
+                    # skips hashing file contents if path matches *.iso
+                    nohash = "\.iso$";
+                  };
+                '';
+                default = { };
+              };
+            };
+          }
+        )
+      );
       description = "A set of copyparty volumes to create";
       default = {
         "/" = {
           path = defaultShareDir;
-          access = { r = "*"; };
+          access = {
+            r = "*";
+          };
         };
       };
       example = literalExpression ''
@@ -204,80 +254,122 @@ in {
     };
   };
 
-  config = mkIf cfg.enable {
-    systemd.services.copyparty = {
-      description = "http file sharing hub";
-      wantedBy = [ "multi-user.target" ];
+  config = mkIf cfg.enable (
+    let
+      command = "${getExe cfg.package} -c ${runtimeConfigPath}";
+    in
+    {
+      systemd.services.copyparty = {
+        description = "http file sharing hub";
+        wantedBy = [ "multi-user.target" ];
 
-      environment = {
-        PYTHONUNBUFFERED = "true";
-        XDG_CONFIG_HOME = "${home}/.config";
+        environment = {
+          PYTHONUNBUFFERED = "true";
+          XDG_CONFIG_HOME = externalStateDir;
+        };
+
+        preStart =
+          let
+            replaceSecretCommand =
+              name: attrs:
+              "${getExe pkgs.replace-secret} '${passwordPlaceholder name}' '${attrs.passwordFile}' ${runtimeConfigPath}";
+          in
+          ''
+            set -euo pipefail
+            install -m 600 ${configFile} ${runtimeConfigPath}
+            ${concatStringsSep "\n" (mapAttrsToList replaceSecretCommand cfg.accounts)}
+          '';
+
+        serviceConfig = {
+          Type = "simple";
+          ExecStart = command;
+          # Hardening options
+          User = cfg.user;
+          Group = cfg.group;
+          RuntimeDirectory = [ "copyparty" ];
+          RuntimeDirectoryMode = "0700";
+          StateDirectory = [ "copyparty" ];
+          StateDirectoryMode = "0700";
+          CacheDirectory = lib.mkIf (cfg.settings ? hist) [ "copyparty" ];
+          CacheDirectoryMode = lib.mkIf (cfg.settings ? hist) "0700";
+          WorkingDirectory = externalStateDir;
+          BindReadOnlyPaths = [
+            "/nix/store"
+            "-/etc/resolv.conf"
+            "-/etc/nsswitch.conf"
+            "-/etc/group"
+            "-/etc/hosts"
+            "-/etc/localtime"
+          ] ++ (mapAttrsToList (k: v: "-${v.passwordFile}") cfg.accounts);
+          BindPaths =
+            (if cfg.settings ? hist then [ cfg.settings.hist ] else [ ])
+            ++ [ externalStateDir ]
+            ++ (mapAttrsToList (k: v: v.path) cfg.volumes);
+          # ProtectSystem = "strict";
+          # Note that unlike what 'ro' implies,
+          # this actually makes it impossible to read anything in the root FS,
+          # except for things explicitly mounted via `RuntimeDirectory`, `StateDirectory`, `CacheDirectory`, and `BindReadOnlyPaths`.
+          # This is because TemporaryFileSystem creates a *new* *empty* filesystem for the process, so only bindmounts are visible.
+          TemporaryFileSystem = "/:ro";
+          PrivateTmp = true;
+          PrivateDevices = true;
+          ProtectKernelTunables = true;
+          ProtectControlGroups = true;
+          RestrictSUIDSGID = true;
+          PrivateMounts = true;
+          ProtectKernelModules = true;
+          ProtectKernelLogs = true;
+          ProtectHostname = true;
+          ProtectClock = true;
+          ProtectProc = "invisible";
+          ProcSubset = "pid";
+          RestrictNamespaces = true;
+          RemoveIPC = true;
+          UMask = "0077";
+          LimitNOFILE = cfg.openFilesLimit;
+          NoNewPrivileges = true;
+          LockPersonality = true;
+          RestrictRealtime = true;
+          MemoryDenyWriteExecute = true;
+        };
       };
 
-      preStart = let
-        replaceSecretCommand = name: attrs:
-          "${getExe pkgs.replace-secret} '${
-            passwordPlaceholder name
-          }' '${attrs.passwordFile}' ${runtimeConfigPath}";
-      in ''
-        set -euo pipefail
-        install -m 600 ${configFile} ${runtimeConfigPath}
-        ${concatStringsSep "\n"
-        (mapAttrsToList replaceSecretCommand cfg.accounts)}
-      '';
+      # ensure volumes exist:
+      systemd.tmpfiles.settings."copyparty" = (
+        lib.attrsets.mapAttrs' (
+          name: value:
+          lib.attrsets.nameValuePair (value.path) {
+            d = {
+              #: in front of things means it wont change it if the directory already exists.
+              group = ":${cfg.group}";
+              user = ":${cfg.user}";
+              mode = ":755";
+            };
+          }
+        ) cfg.volumes
+      );
 
-      serviceConfig = {
-        Type = "simple";
-        ExecStart = "${getExe cfg.package} -c ${runtimeConfigPath}";
-
-        # Hardening options
-        User = "copyparty";
-        Group = "copyparty";
-        RuntimeDirectory = name;
-        RuntimeDirectoryMode = "0700";
-        StateDirectory = [ name "${name}/data" "${name}/.config" ];
-        StateDirectoryMode = "0700";
-        WorkingDirectory = home;
-        TemporaryFileSystem = "/:ro";
-        BindReadOnlyPaths = [
-          "/nix/store"
-          "-/etc/resolv.conf"
-          "-/etc/nsswitch.conf"
-          "-/etc/hosts"
-          "-/etc/localtime"
-        ] ++ (mapAttrsToList (k: v: "-${v.passwordFile}") cfg.accounts);
-        BindPaths = [ home ] ++ (mapAttrsToList (k: v: v.path) cfg.volumes);
-        # Would re-mount paths ignored by temporary root
-        #ProtectSystem = "strict";
-        ProtectHome = true;
-        PrivateTmp = true;
-        PrivateDevices = true;
-        ProtectKernelTunables = true;
-        ProtectControlGroups = true;
-        RestrictSUIDSGID = true;
-        PrivateMounts = true;
-        ProtectKernelModules = true;
-        ProtectKernelLogs = true;
-        ProtectHostname = true;
-        ProtectClock = true;
-        ProtectProc = "invisible";
-        ProcSubset = "pid";
-        RestrictNamespaces = true;
-        RemoveIPC = true;
-        UMask = "0077";
-        LimitNOFILE = cfg.openFilesLimit;
-        NoNewPrivileges = true;
-        LockPersonality = true;
-        RestrictRealtime = true;
+      users.groups.copyparty = lib.mkIf (cfg.user == "copyparty" && cfg.group == "copyparty") { };
+      users.users.copyparty = lib.mkIf (cfg.user == "copyparty" && cfg.group == "copyparty") {
+        description = "Service user for copyparty";
+        group = "copyparty";
+        home = externalStateDir;
+        isSystemUser = true;
       };
-    };
+      environment.systemPackages = lib.mkIf cfg.mkHashWrapper [
+        (pkgs.writeShellScriptBin "copyparty-hash" ''
+          set -a  # automatically export variables
+          # set same environment variables as the systemd service
+          ${lib.pipe config.systemd.services.copyparty.environment [
+            (lib.filterAttrs (n: v: v != null && n != "PATH"))
+            (lib.mapAttrs (_: v: "${v}"))
+            (lib.toShellVars)
+          ]}
+          PATH=${config.systemd.services.copyparty.environment.PATH}:$PATH
 
-    users.groups.copyparty = { };
-    users.users.copyparty = {
-      description = "Service user for copyparty";
-      group = "copyparty";
-      home = home;
-      isSystemUser = true;
-    };
-  };
+          exec ${command} --ah-cli
+        '')
+      ];
+    }
+  );
 }

contrib/package/arch/PKGBUILD

@@ -1,56 +1,48 @@
 # Maintainer: icxes <dev.null@need.moe>
+# Contributor: Morgan Adamiec <morganamilo@archlinux.org>
+# NOTE: You generally shouldn't use this PKGBUILD on Arch, as it is mainly for testing purposes. Install copyparty using pacman instead.
+
 pkgname=copyparty
-pkgver="1.15.4"
+pkgver="1.19.2"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
 url="https://github.com/9001/${pkgname}"
 license=('MIT')
-depends=("python" "lsof" "python-jinja")
+depends=("bash" "python" "lsof" "python-jinja")
 makedepends=("python-wheel" "python-setuptools" "python-build" "python-installer" "make" "pigz")
 optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tags"
-            "cfssl: generate TLS certificates on startup (pointless when reverse-proxied)"
-            "python-mutagen: music tags (alternative)" 
-            "python-pillow: thumbnails for images" 
-            "python-pyvips: thumbnails for images (higher quality, faster, uses more ram)" 
-            "libkeyfinder-git: detection of musical keys" 
-            "qm-vamp-plugins: BPM detection" 
-            "python-pyopenssl: ftps functionality" 
-            "python-argon2_cffi: hashed passwords in config" 
-            "python-impacket-git: smb support (bad idea)"
+            "cfssl: generate TLS certificates on startup"
+            "python-mutagen: music tags (alternative)"
+            "python-pillow: thumbnails for images"
+            "python-pyvips: thumbnails for images (higher quality, faster, uses more ram)"
+            "libkeyfinder: detection of musical keys"
+            "python-pyopenssl: ftps functionality"
+            "python-pyzmq: send zeromq messages from event-hooks"
+            "python-argon2-cffi: hashed passwords in config"
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
-backup=("etc/${pkgname}.d/init" )
-sha256sums=("e0c91b2344f1cbec8be60f715d076f8ba79eef09c1f9f016f5192f743621800d")
+backup=("etc/${pkgname}/copyparty.conf" )
+sha256sums=("9f0dcd8124f260a0c72676b70d84c82388cfe5b47e7d0556f5190c88208580a2")
 
 build() {
+    cd "${srcdir}/${pkgname}-${pkgver}/copyparty/web"
+    make
+
     cd "${srcdir}/${pkgname}-${pkgver}"
-    
-    pushd copyparty/web
-    make -j$(nproc)
-    rm Makefile
-    popd
-    
-    python3 -m build -wn
+    python -m build --wheel --no-isolation
 }
 
 package() {
     cd "${srcdir}/${pkgname}-${pkgver}"
-    python3 -m installer -d "$pkgdir" dist/*.whl
+    python -m installer --destdir="$pkgdir" dist/*.whl
 
-    install -dm755 "${pkgdir}/etc/${pkgname}.d"
+    install -dm755 "${pkgdir}/etc/${pkgname}"
     install -Dm755 "bin/prisonparty.sh" "${pkgdir}/usr/bin/prisonparty"
-    install -Dm644 "contrib/package/arch/${pkgname}.conf" "${pkgdir}/etc/${pkgname}.d/init"
-    install -Dm644 "contrib/package/arch/${pkgname}.service" "${pkgdir}/usr/lib/systemd/system/${pkgname}.service"
-    install -Dm644 "contrib/package/arch/prisonparty.service" "${pkgdir}/usr/lib/systemd/system/prisonparty.service"
-    install -Dm644 "contrib/package/arch/index.md" "${pkgdir}/var/lib/${pkgname}-jail/README.md"
+    install -Dm644 "contrib/systemd/${pkgname}.conf" "${pkgdir}/etc/${pkgname}/copyparty.conf"
+    install -Dm644 "contrib/systemd/${pkgname}@.service" "${pkgdir}/usr/lib/systemd/system/${pkgname}@.service"
+    install -Dm644 "contrib/systemd/${pkgname}-user.service" "${pkgdir}/usr/lib/systemd/user/${pkgname}.service"
+    install -Dm644 "contrib/systemd/prisonparty@.service" "${pkgdir}/usr/lib/systemd/system/prisonparty@.service"
+    install -Dm644 "contrib/systemd/index.md" "${pkgdir}/var/lib/${pkgname}-jail/README.md"
     install -Dm644 "LICENSE" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
-
-    find /etc/${pkgname}.d -iname '*.conf' 2>/dev/null | grep -qE . && return
-    echo "┏━━━━━━━━━━━━━━━──-"
-    echo "┃ Configure ${pkgname} by adding .conf files into /etc/${pkgname}.d/"
-    echo "┃ and maybe copy+edit one of the following to /etc/systemd/system/:"
-    echo "┣━♦ /usr/lib/systemd/system/${pkgname}.service   (standard)"
-    echo "┣━♦ /usr/lib/systemd/system/prisonparty.service (chroot)"
-    echo "┗━━━━━━━━━━━━━━━──-"
 }

contrib/package/makedeb-mpr/PKGBUILD

@@ -0,0 +1,44 @@
+# Contributor: Beethoven <beethovenisadog@protonmail.com>
+
+
+pkgname=copyparty
+pkgver=1.19.2
+pkgrel=1
+pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
+arch=("any")
+url="https://github.com/9001/${pkgname}"
+license=('MIT')
+depends=("bash" "python3" "lsof" "python3-jinja2")
+makedepends=("python3-wheel" "python3-setuptools" "python3-build" "python3-installer" "make" "pigz")
+optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tags"
+            "golang-cfssl: generate TLS certificates on startup"
+            "python3-mutagen: music tags (alternative)"
+            "python3-pil: thumbnails for images"
+            "python3-openssl: ftps functionality"
+            "python3-zmq: send zeromq messages from event-hooks"
+            "python3-argon2: hashed passwords in config"
+)
+source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
+backup=("/etc/${pkgname}.d/init" )
+sha256sums=("9f0dcd8124f260a0c72676b70d84c82388cfe5b47e7d0556f5190c88208580a2")
+
+build() {
+    cd "${srcdir}/${pkgname}-${pkgver}/copyparty/web"
+    make
+
+    cd "${srcdir}/${pkgname}-${pkgver}"
+    python -m build --wheel --no-isolation
+}
+
+package() {
+    cd "${srcdir}/${pkgname}-${pkgver}"
+    python -m installer --destdir="$pkgdir" dist/*.whl
+
+    install -dm755 "${pkgdir}/etc/${pkgname}.d"
+    install -Dm755 "bin/prisonparty.sh" "${pkgdir}/usr/bin/prisonparty"
+    install -Dm644 "contrib/package/makedeb-mpr/${pkgname}.conf" "${pkgdir}/etc/${pkgname}.d/init"
+    install -Dm644 "contrib/package/makedeb-mpr/${pkgname}.service" "${pkgdir}/usr/lib/systemd/system/${pkgname}.service"
+    install -Dm644 "contrib/package/makedeb-mpr/prisonparty.service" "${pkgdir}/usr/lib/systemd/system/prisonparty.service"
+    install -Dm644 "contrib/package/makedeb-mpr/index.md" "${pkgdir}/var/lib/${pkgname}-jail/README.md"
+    install -Dm644 "LICENSE" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
+}

contrib/package/nix/copyparty/default.nix

@@ -1,63 +1,118 @@
-{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, ffmpeg, mutagen,
+{
+  lib,
+  buildPythonApplication,
+  fetchurl,
+  util-linux,
+  python,
+  setuptools,
+  jinja2,
+  impacket,
+  pyopenssl,
+  cfssl,
+  argon2-cffi,
+  pillow,
+  pyvips,
+  pyzmq,
+  ffmpeg,
+  mutagen,
+  pyftpdlib,
+  magic,
+  partftpy,
+  fusepy, # for partyfuse
 
-# use argon2id-hashed passwords in config files (sha2 is always available)
-withHashedPasswords ? true,
+  # use argon2id-hashed passwords in config files (sha2 is always available)
+  withHashedPasswords ? true,
 
-# generate TLS certificates on startup (pointless when reverse-proxied)
-withCertgen ? false,
+  # generate TLS certificates on startup (pointless when reverse-proxied)
+  withCertgen ? false,
 
-# create thumbnails with Pillow; faster than FFmpeg / MediaProcessing
-withThumbnails ? true,
+  # create thumbnails with Pillow; faster than FFmpeg / MediaProcessing
+  withThumbnails ? true,
 
-# create thumbnails with PyVIPS; even faster, uses more memory
-# -- can be combined with Pillow to support more filetypes
-withFastThumbnails ? false,
+  # create thumbnails with PyVIPS; even faster, uses more memory
+  # -- can be combined with Pillow to support more filetypes
+  withFastThumbnails ? false,
 
-# enable FFmpeg; thumbnails for most filetypes (also video and audio), extract audio metadata, transcode audio to opus
-# -- possibly dangerous if you allow anonymous uploads, since FFmpeg has a huge attack surface
-# -- can be combined with Thumbnails and/or FastThumbnails, since FFmpeg is slower than both
-withMediaProcessing ? true,
+  # enable FFmpeg; thumbnails for most filetypes (also video and audio), extract audio metadata, transcode audio to opus
+  # -- possibly dangerous if you allow anonymous uploads, since FFmpeg has a huge attack surface
+  # -- can be combined with Thumbnails and/or FastThumbnails, since FFmpeg is slower than both
+  withMediaProcessing ? true,
 
-# if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
-withBasicAudioMetadata ? false,
+  # if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
+  withBasicAudioMetadata ? false,
 
-# enable FTPS support in the FTP server
-withFTPS ? false,
+  # send ZeroMQ messages from event-hooks
+  withZeroMQ ? true,
 
-# samba/cifs server; dangerous and buggy, enable if you really need it
-withSMB ? false,
+  # enable FTP server
+  withFTP ? true,
+
+  # enable FTPS support in the FTP server
+  withFTPS ? false,
+
+  # enable TFTP server
+  withTFTP ? false,
+
+  # samba/cifs server; dangerous and buggy, enable if you really need it
+  withSMB ? false,
+
+  # enables filetype detection for nameless uploads
+  withMagic ? false,
+
+  # extra packages to add to the PATH
+  extraPackages ? [ ],
+
+  # function that accepts a python packageset and returns a list of packages to
+  # be added to the python venv. useful for scripts and such that require
+  # additional dependencies
+  extraPythonPackages ? (_p: [ ]),
 
 }:
 
 let
   pinData = lib.importJSON ./pin.json;
-  pyEnv = python.withPackages (ps:
-    with ps; [
+  runtimeDeps = ([ util-linux ] ++ extraPackages ++ lib.optional withMediaProcessing ffmpeg);
+in
+buildPythonApplication {
+  pname = "copyparty";
+  inherit (pinData) version;
+  src = fetchurl {
+    inherit (pinData) url hash;
+  };
+  dependencies =
+    [
       jinja2
+      fusepy
     ]
     ++ lib.optional withSMB impacket
+    ++ lib.optional withFTP pyftpdlib
     ++ lib.optional withFTPS pyopenssl
+    ++ lib.optional withTFTP partftpy
     ++ lib.optional withCertgen cfssl
     ++ lib.optional withThumbnails pillow
     ++ lib.optional withFastThumbnails pyvips
     ++ lib.optional withMediaProcessing ffmpeg
     ++ lib.optional withBasicAudioMetadata mutagen
     ++ lib.optional withHashedPasswords argon2-cffi
-    );
-in stdenv.mkDerivation {
-  pname = "copyparty";
-  version = pinData.version;
-  src = fetchurl {
-    url = pinData.url;
-    hash = pinData.hash;
+    ++ lib.optional withZeroMQ pyzmq
+    ++ lib.optional withMagic magic
+    ++ (extraPythonPackages python.pkgs);
+  makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath runtimeDeps}" ];
+
+  pyproject = true;
+  build-system = [
+    setuptools
+  ];
+  meta = {
+    description = "Turn almost any device into a file server";
+    longDescription = ''
+      Portable file server with accelerated resumable uploads, dedup, WebDAV,
+      FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps
+    '';
+    homepage = "https://github.com/9001/copyparty";
+    changelog = "https://github.com/9001/copyparty/releases/tag/v${pinData.version}";
+    license = lib.licenses.mit;
+    mainProgram = "copyparty";
+    sourceProvenance = [ lib.sourceTypes.fromSource ];
   };
-  buildInputs = [ makeWrapper ];
-  dontUnpack = true;
-  dontBuild = true;
-  installPhase = ''
-    install -Dm755 $src $out/share/copyparty-sfx.py
-    makeWrapper ${pyEnv.interpreter} $out/bin/copyparty \
-      --set PATH '${lib.makeBinPath ([ utillinux ] ++ lib.optional withMediaProcessing ffmpeg)}:$PATH' \
-      --add-flags "$out/share/copyparty-sfx.py"
-  '';
 }

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.15.4/copyparty-sfx.py",
-    "version": "1.15.4",
-    "hash": "sha256-a6zgGg1EOu7wtiOzPLwv4h17yrmq2Iharip9rpI8kt0="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.19.2/copyparty-1.19.2.tar.gz",
+    "version": "1.19.2",
+    "hash": "sha256-nw3NgSTyYKDHJna3DYTII4jP5bR+fQVW9RkMiCCFgKI="
 }

contrib/package/nix/copyparty/update.py

@@ -11,14 +11,14 @@ import base64
 import json
 import hashlib
 import sys
-import re
+import tarfile
 from pathlib import Path
 
 OUTPUT_FILE = Path("pin.json")
-TARGET_ASSET = "copyparty-sfx.py"
+TARGET_ASSET = lambda version: f"copyparty-{version}.tar.gz"
 HASH_TYPE = "sha256"
 LATEST_RELEASE_URL = "https://api.github.com/repos/9001/copyparty/releases/latest"
-DOWNLOAD_URL = lambda version: f"https://github.com/9001/copyparty/releases/download/v{version}/{TARGET_ASSET}"
+DOWNLOAD_URL = lambda version: f"https://github.com/9001/copyparty/releases/download/v{version}/{TARGET_ASSET(version)}"
 
 
 def get_formatted_hash(binary):
@@ -29,11 +29,13 @@ def get_formatted_hash(binary):
     return f"{HASH_TYPE}-{encoded_hash}"
 
 
-def version_from_sfx(binary):
-    result = re.search(b'^VER = "(.*)"$', binary, re.MULTILINE)
-    if result:
-        return result.groups(1)[0].decode("ascii")
+def version_from_tar_gz(path):
+    with tarfile.open(path) as tarball:
+        release_name = tarball.getmembers()[0].name
+        prefix = "copyparty-"
 
+        if release_name.startswith(prefix):
+            return release_name.replace(prefix, "")
     raise ValueError("version not found in provided file")
 
 
@@ -42,7 +44,7 @@ def remote_release_pin():
 
     response = requests.get(LATEST_RELEASE_URL).json()
     version = response["tag_name"].lstrip("v")
-    asset_info = [a for a in response["assets"] if a["name"] == TARGET_ASSET][0]
+    asset_info = [a for a in response["assets"] if a["name"] == TARGET_ASSET(version)][0]
     download_url = asset_info["browser_download_url"]
     asset = requests.get(download_url)
     formatted_hash = get_formatted_hash(asset.content)
@@ -52,10 +54,9 @@ def remote_release_pin():
 
 
 def local_release_pin(path):
-    asset = path.read_bytes()
-    version = version_from_sfx(asset)
+    version = version_from_tar_gz(path)
     download_url = DOWNLOAD_URL(version)
-    formatted_hash = get_formatted_hash(asset)
+    formatted_hash = get_formatted_hash(path.read_bytes())
 
     result = {"url": download_url, "version": version, "hash": formatted_hash}
     return result

contrib/package/nix/partftpy/default.nix

@@ -0,0 +1,30 @@
+{
+  lib,
+  buildPythonPackage,
+  fetchurl,
+  setuptools,
+}:
+let
+  pinData = lib.importJSON ./pin.json;
+in
+
+buildPythonPackage rec {
+  pname = "partftpy";
+  inherit (pinData) version;
+  pyproject = true;
+
+  src = fetchurl {
+    inherit (pinData) url hash;
+  };
+
+  build-system = [ setuptools ];
+
+  pythonImportsCheck = [ "partftpy.TftpServer" ];
+
+  meta = {
+    description = "Pure Python TFTP library  (copyparty edition)";
+    homepage = "https://github.com/9001/partftpy";
+    changelog = "https://github.com/9001/partftpy/releases/tag/${version}";
+    license = lib.licenses.mit;
+  };
+}

contrib/package/nix/partftpy/pin.json

@@ -0,0 +1,5 @@
+{
+    "url": "https://github.com/9001/partftpy/releases/download/v0.4.0/partftpy-0.4.0.tar.gz",
+    "version": "0.4.0",
+    "hash": "sha256-5Q2zyuJ892PGZmb+YXg0ZPW/DK8RDL1uE0j5HPd4We0="
+}

contrib/package/nix/partftpy/update.py

@@ -0,0 +1,50 @@
+#!/usr/bin/env python3
+
+# Update the Nix package pin
+#
+# Usage: ./update.sh
+
+import base64
+import json
+import hashlib
+import sys
+from pathlib import Path
+
+OUTPUT_FILE = Path("pin.json")
+TARGET_ASSET = lambda version: f"partftpy-{version}.tar.gz"
+HASH_TYPE = "sha256"
+LATEST_RELEASE_URL = "https://api.github.com/repos/9001/partftpy/releases/latest"
+
+
+def get_formatted_hash(binary):
+    hasher = hashlib.new("sha256")
+    hasher.update(binary)
+    asset_hash = hasher.digest()
+    encoded_hash = base64.b64encode(asset_hash).decode("ascii")
+    return f"{HASH_TYPE}-{encoded_hash}"
+
+
+def remote_release_pin():
+    import requests
+
+    response = requests.get(LATEST_RELEASE_URL).json()
+    version = response["tag_name"].lstrip("v")
+    asset_info = [a for a in response["assets"] if a["name"] == TARGET_ASSET(version)][0]
+    download_url = asset_info["browser_download_url"]
+    asset = requests.get(download_url)
+    formatted_hash = get_formatted_hash(asset.content)
+
+    result = {"url": download_url, "version": version, "hash": formatted_hash}
+    return result
+
+
+def main():
+    result = remote_release_pin()
+
+    print(result)
+    json_result = json.dumps(result, indent=4)
+    OUTPUT_FILE.write_text(json_result)
+
+
+if __name__ == "__main__":
+    main()

contrib/package/rpm/copyparty.spec

@@ -0,0 +1,62 @@
+Name:           copyparty
+Version:        $pkgver
+Release:        $pkgrel
+License:        MIT
+Group:          Utilities
+URL:            https://github.com/9001/copyparty
+Source0:        copyparty-$pkgver.tar.gz
+Summary:        File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++
+BuildArch:      noarch
+BuildRequires:  python3, python3-devel, pyproject-rpm-macros, python-setuptools, python-wheel, make
+Requires:       python3, (python3-jinja2 or python-jinja2), lsof
+Recommends:     ffmpeg, (golang-github-cloudflare-cfssl or cfssl), python-mutagen, python-pillow, python-pyvips
+Recommends:     qm-vamp-plugins, python-argon2-cffi, (python-pyopenssl or pyopenssl), python-impacket
+
+%description
+Portable file server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps
+
+See release at https://github.com/9001/copyparty/releases
+
+%global debug_package %{nil}
+
+%generate_buildrequires
+%pyproject_buildrequires
+
+%prep
+%setup -q
+
+%build
+cd "copyparty/web"
+make
+cd -
+%pyproject_wheel
+
+%install
+mkdir -p %{buildroot}%{_bindir}
+mkdir -p %{buildroot}%{_libdir}/systemd/{system,user}
+mkdir -p %{buildroot}/etc/%{name}
+mkdir -p %{buildroot}/var/lib/%{name}-jail
+mkdir -p %{buildroot}%{_datadir}/licenses/%{name}
+
+%pyproject_install
+%pyproject_save_files copyparty
+
+install -m 0755 bin/prisonparty.sh                       %{buildroot}%{_bindir}/prisonpary.sh
+install -m 0644 contrib/systemd/%{name}.conf             %{buildroot}/etc/%{name}/%{name}.conf
+install -m 0644 contrib/systemd/%{name}@.service         %{buildroot}%{_libdir}/systemd/system/%{name}@.service
+install -m 0644 contrib/systemd/%{name}-user.service     %{buildroot}%{_libdir}/systemd/user/%{name}.service
+install -m 0644 contrib/systemd/prisonparty@.service     %{buildroot}%{_libdir}/systemd/system/prisonparty@.service
+install -m 0644 contrib/systemd/index.md                 %{buildroot}/var/lib/%{name}-jail/README.md
+install -m 0644 LICENSE                                  %{buildroot}%{_datadir}/licenses/%{name}/LICENSE
+
+%files -n copyparty -f %{pyproject_files}
+%license LICENSE
+%{_bindir}/copyparty
+%{_bindir}/partyfuse
+%{_bindir}/u2c
+%{_bindir}/prisonpary.sh
+/etc/%{name}/%{name}.conf
+%{_libdir}/systemd/system/%{name}@.service
+%{_libdir}/systemd/user/%{name}.service
+%{_libdir}/systemd/system/prisonparty@.service
+/var/lib/%{name}-jail/README.md

contrib/plugins/README.md

@@ -15,6 +15,7 @@ save one of these as `.epilogue.html` inside a folder to customize it:
 point `--js-browser` to one of these by URL:
 
 * [`minimal-up2k.js`](minimal-up2k.js) is similar to the above `minimal-up2k.html` except it applies globally to all write-only folders
+* [`quickmove.js`](quickmove.js) adds a hotkey to move selected files into a subfolder
 * [`up2k-hooks.js`](up2k-hooks.js) lets you specify a ruleset for files to skip uploading
   * [`up2k-hook-ytid.js`](up2k-hook-ytid.js) is a more specific example checking youtube-IDs against some API
 

contrib/plugins/graft-thumbs.js

@@ -0,0 +1,117 @@
+// USAGE:
+//   place this file somewhere in the webroot and then
+//   python3 -m copyparty --js-browser /.res/graft-thumbs.js
+//
+// DESCRIPTION:
+//   this is a gridview plugin which, for each file in a folder,
+//   looks for another file with the same filename (but with a
+//   different file extension)
+//
+//   if one of those files is an image and the other is not,
+//   then this plugin assumes the image is a "sidecar thumbnail"
+//   for the other file, and it will graft the image thumbnail
+//   onto the non-image file (for example an mp3)
+//
+//   optional feature 1, default-enabled:
+//   the image-file is then hidden from the directory listing
+//
+//   optional feature 2, default-enabled:
+//   when clicking the audio file, the image will also open
+
+
+(function() {
+
+	// `graft_thumbs` assumes the gridview has just been rendered;
+	// it looks for sidecars, and transplants those thumbnails onto
+	// the other file with the same basename (filename sans extension)
+
+	var graft_thumbs = function () {
+		if (!thegrid.en)
+			return;  // not in grid mode
+
+		var files = msel.getall(),
+			pairs = {};
+
+		console.log(files);
+
+		for (var a = 0; a < files.length; a++) {
+			var file = files[a],
+				is_pic = /\.(jpe?g|png|gif|webp)$/i.exec(file.vp),
+				is_audio = re_au_all.exec(file.vp),
+				basename = file.vp.replace(/\.[^\.]+$/, ""),
+				entry = pairs[basename];
+
+			if (!entry)
+				// first time seeing this basename; create a new entry in pairs
+				entry = pairs[basename] = {};
+
+			if (is_pic)
+				entry.thumb = file;
+			else if (is_audio)
+				entry.audio = file;
+		}
+
+		var basenames = Object.keys(pairs);
+		for (var a = 0; a < basenames.length; a++)
+			(function(a) {
+				var pair = pairs[basenames[a]];
+
+				if (!pair.thumb || !pair.audio)
+					return;  // not a matching pair of files
+
+				var img_thumb = QS('#ggrid a[ref="' + pair.thumb.id + '"] img[onload]'),
+					img_audio = QS('#ggrid a[ref="' + pair.audio.id + '"] img[onload]');
+
+				if (!img_thumb || !img_audio)
+					return;  // something's wrong... let's bail
+
+				// alright, graft the thumb...
+				img_audio.src = img_thumb.src;
+
+				// ...and hide the sidecar
+				img_thumb.closest('a').style.display = 'none';
+
+				// ...and add another onclick-handler to the audio,
+				// so it also opens the pic while playing the song
+				img_audio.addEventListener('click', function() {
+					img_thumb.click();
+					return false;  // let it bubble to the next listener
+				});
+
+			})(a);
+	};
+
+	// ...and then the trick! near the end of loadgrid,
+	// thegrid.bagit is called to initialize the baguettebox
+	// (image/video gallery); this is the perfect function to
+	// "hook" (hijack) so we can run our code :^)
+
+	// need to grab a backup of the original function first,
+	var orig_func = thegrid.bagit;
+
+	// and then replace it with our own:
+	thegrid.bagit = function (isrc) {
+
+		if (isrc !== '#ggrid')
+			// we only want to modify the grid, so
+			// let the original function handle this one
+			return orig_func(isrc);
+
+		graft_thumbs();
+
+		// when changing directories, the grid is
+		// rendered before msel returns the correct
+		// filenames, so schedule another run:
+		setTimeout(graft_thumbs, 1);
+
+		// and finally, call the original thegrid.bagit function
+		return orig_func(isrc);
+	};
+
+	if (ls0) {
+		// the server included an initial listing json (ls0),
+		// so the grid has already been rendered without our hook
+		graft_thumbs();
+	}
+
+})();

contrib/plugins/minimal-up2k.js

@@ -12,6 +12,23 @@ almost the same as minimal-up2k.html except this one...:
 
  -- looks slightly better
 
+
+========================
+== USAGE INSTRUCTIONS ==
+
+1. create a volume which anyone can read from (if you haven't already)
+2. copy this file into that volume, so anyone can download it
+3. enable the plugin by telling the webbrowser to load this file;
+    assuming the URL to the public volume is /res/, and
+    assuming you're using config-files, then add this to your config:
+
+    [global]
+      js-browser: /res/minimal-up2k.js
+
+alternatively, if you're not using config-files, then
+add the following commandline argument instead:
+  --js-browser=/res/minimal-up2k.js
+
 */
 
 var u2min = `

contrib/plugins/quickmove.js

@@ -0,0 +1,140 @@
+"use strict";
+
+
+// USAGE:
+//   place this file somewhere in the webroot, 
+//   for example in a folder named ".res" to hide it, and then
+//   python3 copyparty-sfx.py -v .::A --js-browser /.res/quickmove.js
+//
+// DESCRIPTION:
+//   the command above launches copyparty with one single volume;
+//   ".::A" = current folder as webroot, and everyone has Admin
+//
+//   the plugin adds hotkey "W" which moves all selected files
+//   into a subfolder named "foobar" inside the current folder
+
+
+(function() {
+
+    var action_to_perform = ask_for_confirmation_and_then_move;
+    // this decides what the new hotkey should do;
+    //  ask_for_confirmation_and_then_move  =  show a yes/no box,
+    //  move_selected_files  =  just move the files immediately
+
+    var move_destination = "foobar";
+    // this is the target folder to move files to;
+    // by default it is a subfolder of the current folder,
+    // but it can also be an absolute path like "/foo/bar"
+
+    // ===
+    // ===   END OF CONFIG
+    // ===
+
+    var main_hotkey_handler,  // copyparty's original hotkey handler
+        plugin_enabler,  // timer to engage this plugin when safe
+        files_to_move;  // list of files to move
+
+    function ask_for_confirmation_and_then_move() {
+        var num_files = msel.getsel().length,
+            msg = "move the selected " + num_files + " files?";
+
+        if (!num_files)
+            return toast.warn(2, 'no files were selected to be moved');
+
+        modal.confirm(msg, move_selected_files, null);
+    }
+
+    function move_selected_files() {
+        var selection = msel.getsel();
+
+        if (!selection.length)
+            return toast.warn(2, 'no files were selected to be moved');
+
+        if (thegrid.bbox) {
+            // close image/video viewer
+            thegrid.bbox = null;
+            baguetteBox.destroy();
+        }
+
+        files_to_move = [];
+        for (var a = 0; a < selection.length; a++)
+            files_to_move.push(selection[a].vp);
+
+        move_next_file();
+    }
+
+    function move_next_file() {
+        var num_files = files_to_move.length,
+            filepath = files_to_move.pop(),
+            filename = vsplit(filepath)[1];
+
+        toast.inf(10, "moving " + num_files + " files...\n\n" + filename);
+
+        var dst = move_destination;
+
+        if (!dst.endsWith('/'))
+            // must have a trailing slash, so add it
+            dst += '/';
+
+        if (!dst.startsWith('/'))
+            // destination is a relative path, so prefix current folder path
+            dst = get_evpath() + dst;
+
+        // and finally append the filename
+        dst += '/' + filename;
+
+        // prepare the move-request to be sent
+        var xhr = new XHR();
+        xhr.onload = xhr.onerror = function() {
+            if (this.status !== 201)
+                return toast.err(30, 'move failed: ' + esc(this.responseText));
+
+            if (files_to_move.length)
+                return move_next_file();  // still more files to go
+
+            toast.ok(1, 'move OK');
+            treectl.goto(); // reload the folder contents
+        };
+        xhr.open('POST', filepath + '?move=' + dst);
+        xhr.send();
+    }
+
+    function our_hotkey_handler(e) {
+        // bail if either ALT, CTRL, or SHIFT is pressed
+        if (e.altKey || e.shiftKey || e.isComposing || ctrl(e))
+            return main_hotkey_handler(e);  // let copyparty handle this keystroke
+
+        var key_name = (e.code || e.key) + '',
+    		ae = document.activeElement,
+		    aet = ae && ae != document.body ? ae.nodeName.toLowerCase() : '';
+
+        // check the current aet (active element type),
+        // only continue if one of the following currently has input focus:
+        //   nothing | link | button | table-row | table-cell | div | text
+        if (aet && !/^(a|button|tr|td|div|pre)$/.test(aet))
+            return main_hotkey_handler(e);  // let copyparty handle this keystroke
+
+        if (key_name == 'KeyW') {
+            // okay, this one's for us... do the thing
+            action_to_perform();
+            return ev(e);
+        }
+
+        return main_hotkey_handler(e);  // let copyparty handle this keystroke
+    }
+
+    function enable_plugin() {
+        if (!window.hotkeys_attached)
+            return console.log('quickmove is waiting for the page to finish loading');
+
+        clearInterval(plugin_enabler);
+        main_hotkey_handler = document.onkeydown;
+        document.onkeydown = our_hotkey_handler;
+        console.log('quickmove is now enabled');
+    }
+
+    // copyparty doesn't enable its hotkeys until the page
+    // has finished loading, so we'll wait for that too
+    plugin_enabler = setInterval(enable_plugin, 100);
+
+})();

contrib/systemd/copyparty-user.service

@@ -0,0 +1,26 @@
+# this will start `/usr/bin/copyparty`
+# and read config from `$HOME/.config/copyparty.conf`
+#
+# unless you add -q to disable logging, you may want to remove the
+#   following line to allow buffering (slightly better performance):
+#   Environment=PYTHONUNBUFFERED=x
+
+[Unit]
+Description=copyparty file server
+
+[Service]
+Type=notify
+SyslogIdentifier=copyparty
+WorkingDirectory=/var/lib/copyparty-jail
+Environment=PYTHONUNBUFFERED=x
+Environment=PRTY_CONFIG=%h/.config/copyparty/copyparty.conf
+ExecReload=/bin/kill -s USR1 $MAINPID
+
+# ensure there is a config
+ExecStartPre=/bin/bash -c 'if [[ ! -f %h/.config/copyparty/copyparty.conf ]]; then mkdir -p %h/.config/copyparty; cp /etc/copyparty/copyparty.conf %h/.config/copyparty/copyparty.conf; fi'
+
+# run copyparty
+ExecStart=/usr/bin/python3 /usr/bin/copyparty
+
+[Install]
+WantedBy=default.target

contrib/systemd/copyparty.conf

@@ -1,42 +1,13 @@
-# not actually YAML but lets pretend:
-# -*- mode: yaml -*-
-# vim: ft=yaml:
-
-
-# put this file in /etc/
-
-
 [global]
-  e2dsa  # enable file indexing and filesystem scanning
-  e2ts   # and enable multimedia indexing
-  ansi   # and colors in log messages
-
-  # disable logging to stdout/journalctl and log to a file instead;
-  # $LOGS_DIRECTORY is usually /var/log/copyparty (comes from systemd)
-  # and copyparty replaces %Y-%m%d with Year-MonthDay, so the
-  # full path will be something like /var/log/copyparty/2023-1130.txt
-  # (note: enable compression by adding .xz at the end)
-  q, lo: $LOGS_DIRECTORY/%Y-%m%d.log
-
-  # p: 80,443,3923   # listen on 80/443 as well (requires CAP_NET_BIND_SERVICE)
-  # i: 127.0.0.1     # only allow connections from localhost (reverse-proxies)
-  # ftp: 3921        # enable ftp server on port 3921
-  # p: 3939          # listen on another port
-  # df: 16           # stop accepting uploads if less than 16 GB free disk space
-  # ver              # show copyparty version in the controlpanel
-  # grid             # show thumbnails/grid-view by default
-  # theme: 2         # monokai
-  # name: datasaver  # change the server-name that's displayed in the browser
-  # stats, nos-dup   # enable the prometheus endpoint, but disable the dupes counter (too slow)
-  # no-robots, force-js  # make it harder for search engines to read your server
-
+  i: 127.0.0.1
 
 [accounts]
-  ed: wark  # username: password
+  user: password
 
-
-[/]            # create a volume at "/" (the webroot), which will
-  /mnt         # share the contents of the "/mnt" folder
+[/]
+  /var/lib/copyparty-jail
   accs:
-    rw: *      # everyone gets read-write access, but
-    rwmda: ed  # the user "ed" gets read-write-move-delete-admin
+    r: *
+    rwdma: user
+  flags:
+    grid

contrib/systemd/copyparty.example.conf

@@ -0,0 +1,42 @@
+# not actually YAML but lets pretend:
+# -*- mode: yaml -*-
+# vim: ft=yaml:
+
+
+# put this file in /etc/
+
+
+[global]
+  e2dsa  # enable file indexing and filesystem scanning
+  e2ts   # and enable multimedia indexing
+  ansi   # and colors in log messages
+
+  # disable logging to stdout/journalctl and log to a file instead;
+  # $LOGS_DIRECTORY is usually /var/log/copyparty (comes from systemd)
+  # and copyparty replaces %Y-%m%d with Year-MonthDay, so the
+  # full path will be something like /var/log/copyparty/2023-1130.txt
+  # (note: enable compression by adding .xz at the end)
+  q, lo: $LOGS_DIRECTORY/%Y-%m%d.log
+
+  # p: 80,443,3923   # listen on 80/443 as well (requires CAP_NET_BIND_SERVICE)
+  # i: 127.0.0.1     # only allow connections from localhost (reverse-proxies)
+  # ftp: 3921        # enable ftp server on port 3921
+  # p: 3939          # listen on another port
+  # df: 16           # stop accepting uploads if less than 16 GB free disk space
+  # ver              # show copyparty version in the controlpanel
+  # grid             # show thumbnails/grid-view by default
+  # theme: 2         # monokai
+  # name: datasaver  # change the server-name that's displayed in the browser
+  # stats, nos-dup   # enable the prometheus endpoint, but disable the dupes counter (too slow)
+  # no-robots, force-js  # make it harder for search engines to read your server
+
+
+[accounts]
+  ed: wark  # username: password
+
+
+[/]            # create a volume at "/" (the webroot), which will
+  /mnt         # share the contents of the "/mnt" folder
+  accs:
+    rw: *      # everyone gets read-write access, but
+    rwmda: ed  # the user "ed" gets read-write-move-delete-admin

contrib/systemd/copyparty@.service

@@ -0,0 +1,30 @@
+# this will start `/usr/bin/copyparty`
+# and read config from `/etc/copyparty/copyparty.conf`
+#
+# the %i refers to whatever you put after the copyparty@
+#   so with copyparty@foo.service, %i == foo
+#
+# unless you add -q to disable logging, you may want to remove the
+#   following line to allow buffering (slightly better performance):
+#   Environment=PYTHONUNBUFFERED=x
+
+[Unit]
+Description=copyparty file server
+
+[Service]
+Type=notify
+SyslogIdentifier=copyparty
+WorkingDirectory=/var/lib/copyparty-jail
+Environment=PYTHONUNBUFFERED=x
+Environment=PRTY_CONFIG=/etc/copyparty/copyparty.conf
+ExecReload=/bin/kill -s USR1 $MAINPID
+
+# user to run as + where the TLS certificate is (if any)
+User=%i
+Environment=XDG_CONFIG_HOME=/home/%i/.config
+
+# run copyparty
+ExecStart=/usr/bin/python3 /usr/bin/copyparty
+
+[Install]
+WantedBy=multi-user.target

contrib/systemd/index.md

@@ -0,0 +1,10 @@
+this is `/var/lib/copyparty-jail`, the fallback webroot when copyparty has not yet been configured
+
+please edit `/etc/copyparty/copyparty.conf` (if running as a system service)
+or `$HOME/.config/copyparty/copyparty.conf` if running as a user service
+
+a basic configuration example is available at https://github.com/9001/copyparty/blob/hovudstraum/contrib/systemd/copyparty.example.conf
+a configuration example that explains most flags is available at https://github.com/9001/copyparty/blob/hovudstraum/docs/chungus.conf
+
+the full list of configuration options can be seen at https://ocv.me/copyparty/helptext.html 
+or by running `copyparty --help`

contrib/systemd/prisonparty@.service

@@ -0,0 +1,38 @@
+# this will start `/usr/bin/copyparty`
+# in a chroot, preventing accidental access elsewhere,
+# and read copyparty config from `/etc/copyparty/copyparty.conf`
+#
+# expose additional filesystem locations to copyparty
+#   by listing them between the last `%i` and `--`
+#
+# `%i %i` = user/group to run copyparty as; can be IDs (1000 1000)
+#   the %i refers to whatever you put after the prisonparty@
+#   so with prisonparty@foo.service, %i == foo
+#
+# unless you add -q to disable logging, you may want to remove the
+#   following line to allow buffering (slightly better performance):
+#   Environment=PYTHONUNBUFFERED=x
+
+[Unit]
+Description=copyparty file server
+
+[Service]
+Type=notify
+SyslogIdentifier=prisonparty
+WorkingDirectory=/var/lib/copyparty-jail
+Environment=PYTHONUNBUFFERED=x
+Environment=PRTY_CONFIG=/etc/copyparty/copyparty.conf
+ExecReload=/bin/kill -s USR1 $MAINPID
+
+# user to run as + where the TLS certificate is (if any)
+User=%i
+Environment=XDG_CONFIG_HOME=/home/%i/.config
+
+# run copyparty
+ExecStart=/bin/bash /usr/bin/prisonparty /var/lib/copyparty-jail %i %i \
+  /etc/copyparty \
+  -- \
+  /usr/bin/python3 /usr/bin/copyparty
+
+[Install]
+WantedBy=multi-user.target

contrib/traefik/copyparty.yaml

@@ -0,0 +1,25 @@
+# ./traefik --configFile=copyparty.yaml
+
+entryPoints:
+  web:
+    address: :8080
+    transport:
+      # don't disconnect during big uploads
+      respondingTimeouts:
+        readTimeout: "0s"
+log:
+  level: DEBUG
+providers:
+  file:
+    # WARNING: must be same filename as current file
+    filename: "copyparty.yaml"
+http:
+  services:
+    service-cpp:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:3923/"
+  routers:
+    my-router:
+      rule: "PathPrefix(`/`)"
+      service: service-cpp

contrib/zfs-tune.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+
+import os
+import sqlite3
+import sys
+import traceback
+
+
+"""
+when the up2k-database is stored on a zfs volume, this may give
+slightly higher performance (actual gains not measured yet)
+
+NOTE: must be applied in combination with the related advice in the openzfs documentation;
+https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#database-workloads
+and see specifically the SQLite subsection
+
+it is assumed that all databases are stored in a single location,
+for example with `--hist /var/store/hists`
+
+three alternatives for running this script:
+
+1. copy it into /var/store/hists and run "python3 zfs-tune.py s"
+    (s = modify all databases below folder containing script)
+
+2. cd into /var/store/hists and run "python3 ~/zfs-tune.py w"
+    (w = modify all databases below current working directory)
+
+3. python3 ~/zfs-tune.py /var/store/hists
+
+if you use docker, run copyparty with `--hist /cfg/hists`, copy this script into /cfg, and run this:
+podman run --rm -it --entrypoint /usr/bin/python3 ghcr.io/9001/copyparty-ac /cfg/zfs-tune.py s
+
+"""
+
+
+PAGESIZE = 65536
+
+
+# borrowed from copyparty; short efficient stacktrace for errors
+def min_ex(max_lines: int = 8, reverse: bool = False) -> str:
+    et, ev, tb = sys.exc_info()
+    stb = traceback.extract_tb(tb) if tb else traceback.extract_stack()[:-1]
+    fmt = "%s:%d <%s>: %s"
+    ex = [fmt % (fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
+    if et or ev or tb:
+        ex.append("[%s] %s" % (et.__name__ if et else "(anonymous)", ev))
+    return "\n".join(ex[-max_lines:][:: -1 if reverse else 1])
+
+
+def set_pagesize(db_path):
+    try:
+        # check current page_size
+        with sqlite3.connect(db_path) as db:
+            v = db.execute("pragma page_size").fetchone()[0]
+            if v == PAGESIZE:
+                print(" `-- OK")
+                return
+
+        # https://www.sqlite.org/pragma.html#pragma_page_size
+        #  `- disable wal; set pagesize; vacuum
+        #      (copyparty will reenable wal if necessary)
+
+        with sqlite3.connect(db_path) as db:
+            db.execute("pragma journal_mode=delete")
+            db.commit()
+
+        with sqlite3.connect(db_path) as db:
+            db.execute(f"pragma page_size = {PAGESIZE}")
+            db.execute("vacuum")
+
+        print(" `-- new pagesize OK")
+
+    except Exception:
+        err = min_ex().replace("\n", "\n -- ")
+        print(f"FAILED: {db_path}\n -- {err}")
+
+
+def main():
+    top = os.path.dirname(os.path.abspath(__file__))
+    cwd = os.path.abspath(os.getcwd())
+    try:
+        x = sys.argv[1]
+    except:
+        print(f"""
+this script takes one mandatory argument:
+specify 's' to start recursing from folder containing this script file ({top})
+specify 'w' to start recursing from the current working directory ({cwd})
+specify a path to start recursing from there
+""")
+        sys.exit(1)
+
+    if x.lower() == "w":
+        top = cwd
+    elif x.lower() != "s":
+        top = x
+
+    for dirpath, dirs, files in os.walk(top):
+        for fname in files:
+            if not fname.endswith(".db"):
+                continue
+            db_path = os.path.join(dirpath, fname)
+            print(db_path)
+            set_pagesize(db_path)
+
+
+if __name__ == "__main__":
+    main()

copyparty/__init__.py

@@ -63,10 +63,6 @@ web/browser.js
 web/browser2.html
 web/cf.html
 web/copyparty.gif
-web/dd/2.png
-web/dd/3.png
-web/dd/4.png
-web/dd/5.png
 web/deps/busy.mp3
 web/deps/easymde.css
 web/deps/easymde.js
@@ -80,6 +76,8 @@ web/deps/prismd.css
 web/deps/scp.woff2
 web/deps/sha512.ac.js
 web/deps/sha512.hw.js
+web/idp.html
+web/iiam.gif
 web/md.css
 web/md.html
 web/md.js
@@ -90,6 +88,9 @@ web/mde.html
 web/mde.js
 web/msg.css
 web/msg.html
+web/rups.css
+web/rups.html
+web/rups.js
 web/shares.css
 web/shares.html
 web/shares.js

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 15, 5)
-CODENAME = "fill the drives"
-BUILD_DT = (2024, 10, 5)
+VERSION = (1, 19, 2)
+CODENAME = "usernames"
+BUILD_DT = (2025, 8, 17)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/bos/bos.py

@@ -9,8 +9,11 @@ from . import path as path
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
-_ = (path,)
-__all__ = ["path"]
+MKD_755 = {"chmod_d": 0o755}
+MKD_700 = {"chmod_d": 0o700}
+
+_ = (path, MKD_755, MKD_700)
+__all__ = ["path", "MKD_755", "MKD_700"]
 
 # grep -hRiE '(^|[^a-zA-Z_\.-])os\.' . | gsed -r 's/ /\n/g;s/\(/(\n/g' | grep -hRiE '(^|[^a-zA-Z_\.-])os\.' | sort | uniq -c
 # printf 'os\.(%s)' "$(grep ^def bos/__init__.py | gsed -r 's/^def //;s/\(.*//' | tr '\n' '|' | gsed -r 's/.$//')"
@@ -20,19 +23,39 @@ def chmod(p: str, mode: int) -> None:
     return os.chmod(fsenc(p), mode)
 
 
+def chown(p: str, uid: int, gid: int) -> None:
+    return os.chown(fsenc(p), uid, gid)
+
+
 def listdir(p: str = ".") -> list[str]:
     return [fsdec(x) for x in os.listdir(fsenc(p))]
 
 
-def makedirs(name: str, mode: int = 0o755, exist_ok: bool = True) -> bool:
+def makedirs(name: str, vf: dict[str, Any] = MKD_755, exist_ok: bool = True) -> bool:
+    # os.makedirs does 777 for all but leaf; this does mode on all
+    todo = []
     bname = fsenc(name)
-    try:
-        os.makedirs(bname, mode)
-        return True
-    except:
-        if not exist_ok or not os.path.isdir(bname):
-            raise
+    while bname:
+        if os.path.isdir(bname):
+            break
+        todo.append(bname)
+        bname = os.path.dirname(bname)
+    if not todo:
+        if not exist_ok:
+            os.mkdir(bname)  # to throw
         return False
+    mode = vf["chmod_d"]
+    chown = "chown" in vf
+    for zb in todo[::-1]:
+        try:
+            os.mkdir(zb, mode)
+            if chown:
+                os.chown(zb, vf["uid"], vf["gid"])
+        except:
+            if os.path.isdir(zb):
+                continue
+            raise
+    return True
 
 
 def mkdir(p: str, mode: int = 0o755) -> None:

copyparty/broker_mp.py

@@ -43,6 +43,9 @@ class BrokerMp(object):
         self.procs = []
         self.mutex = threading.Lock()
 
+        self.retpend: dict[int, Any] = {}
+        self.retpend_mutex = threading.Lock()
+
         self.num_workers = self.args.j or CORES
         self.log("broker", "booting {} subprocesses".format(self.num_workers))
         for n in range(1, self.num_workers + 1):
@@ -54,6 +57,8 @@ class BrokerMp(object):
             self.procs.append(proc)
             proc.start()
 
+        Daemon(self.periodic, "mp-periodic")
+
     def shutdown(self) -> None:
         self.log("broker", "shutting down")
         for n, proc in enumerate(self.procs):
@@ -90,8 +95,10 @@ class BrokerMp(object):
                 self.log(*args)
 
             elif dest == "retq":
-                # response from previous ipc call
-                raise Exception("invalid broker_mp usage")
+                with self.retpend_mutex:
+                    retq = self.retpend.pop(retq_id)
+
+                retq.put(args[0])
 
             else:
                 # new ipc invoking managed service in hub
@@ -109,7 +116,6 @@ class BrokerMp(object):
                     proc.q_pend.put((retq_id, "retq", rv))
 
     def ask(self, dest: str, *args: Any) -> Union[ExceptionalQueue, NotExQueue]:
-
         # new non-ipc invoking managed service in hub
         obj = self.hub
         for node in dest.split("."):
@@ -121,17 +127,30 @@ class BrokerMp(object):
         retq.put(rv)
         return retq
 
+    def wask(self, dest: str, *args: Any) -> list[Union[ExceptionalQueue, NotExQueue]]:
+        # call from hub to workers
+        ret = []
+        for p in self.procs:
+            retq = ExceptionalQueue(1)
+            retq_id = id(retq)
+            with self.retpend_mutex:
+                self.retpend[retq_id] = retq
+
+            p.q_pend.put((retq_id, dest, list(args)))
+            ret.append(retq)
+        return ret
+
     def say(self, dest: str, *args: Any) -> None:
         """
         send message to non-hub component in other process,
         returns a Queue object which eventually contains the response if want_retval
         (not-impl here since nothing uses it yet)
         """
-        if dest == "listen":
+        if dest == "httpsrv.listen":
             for p in self.procs:
                 p.q_pend.put((0, dest, [args[0], len(self.procs)]))
 
-        elif dest == "set_netdevs":
+        elif dest == "httpsrv.set_netdevs":
             for p in self.procs:
                 p.q_pend.put((0, dest, list(args)))
 
@@ -140,3 +159,19 @@ class BrokerMp(object):
 
         else:
             raise Exception("what is " + str(dest))
+
+    def periodic(self) -> None:
+        while True:
+            time.sleep(1)
+
+            tdli = {}
+            tdls = {}
+            qs = self.wask("httpsrv.read_dls")
+            for q in qs:
+                qr = q.get()
+                dli, dls = qr
+                tdli.update(dli)
+                tdls.update(dls)
+            tdl = (tdli, tdls)
+            for p in self.procs:
+                p.q_pend.put((0, "httpsrv.write_dls", tdl))

copyparty/broker_mpw.py

@@ -82,37 +82,38 @@ class MpWorker(BrokerCli):
         while True:
             retq_id, dest, args = self.q_pend.get()
 
-            # self.logw("work: [{}]".format(d[0]))
+            if dest == "retq":
+                # response from previous ipc call
+                with self.retpend_mutex:
+                    retq = self.retpend.pop(retq_id)
+
+                retq.put(args)
+                continue
+
             if dest == "shutdown":
                 self.httpsrv.shutdown()
                 self.logw("ok bye")
                 sys.exit(0)
                 return
 
-            elif dest == "reload":
+            if dest == "reload":
                 self.logw("mpw.asrv reloading")
                 self.asrv.reload()
                 self.logw("mpw.asrv reloaded")
+                continue
 
-            elif dest == "reload_sessions":
+            if dest == "reload_sessions":
                 with self.asrv.mutex:
                     self.asrv.load_sessions()
+                continue
 
-            elif dest == "listen":
-                self.httpsrv.listen(args[0], args[1])
+            obj = self
+            for node in dest.split("."):
+                obj = getattr(obj, node)
 
-            elif dest == "set_netdevs":
-                self.httpsrv.set_netdevs(args[0])
-
-            elif dest == "retq":
-                # response from previous ipc call
-                with self.retpend_mutex:
-                    retq = self.retpend.pop(retq_id)
-
-                retq.put(args)
-
-            else:
-                raise Exception("what is " + str(dest))
+            rv = obj(*args)  # type: ignore
+            if retq_id:
+                self.say("retq", rv, retq_id=retq_id)
 
     def ask(self, dest: str, *args: Any) -> Union[ExceptionalQueue, NotExQueue]:
         retq = ExceptionalQueue(1)
@@ -123,5 +124,5 @@ class MpWorker(BrokerCli):
         self.q_yield.put((retq_id, dest, list(args)))
         return retq
 
-    def say(self, dest: str, *args: Any) -> None:
-        self.q_yield.put((0, dest, list(args)))
+    def say(self, dest: str, *args: Any, retq_id=0) -> None:
+        self.q_yield.put((retq_id, dest, list(args)))

copyparty/broker_thr.py

@@ -53,11 +53,11 @@ class BrokerThr(BrokerCli):
         return NotExQueue(obj(*args))  # type: ignore
 
     def say(self, dest: str, *args: Any) -> None:
-        if dest == "listen":
+        if dest == "httpsrv.listen":
             self.httpsrv.listen(args[0], 1)
             return
 
-        if dest == "set_netdevs":
+        if dest == "httpsrv.set_netdevs":
             self.httpsrv.set_netdevs(args[0])
             return
 

copyparty/cert.py

@@ -1,13 +1,11 @@
 import calendar
 import errno
-import filecmp
 import json
 import os
-import shutil
 import time
 
 from .__init__ import ANYWIN
-from .util import Netdev, load_resource, runcmd, wrename, wunlink
+from .util import Netdev, atomic_move, load_resource, runcmd, wunlink
 
 HAVE_CFSSL = not os.environ.get("PRTY_NO_CFSSL")
 
@@ -122,7 +120,7 @@ def _gen_ca(log: "RootLogger", args):
         wunlink(nlog, bname + ".key", VF)
     except:
         pass
-    wrename(nlog, bname + "-key.pem", bname + ".key", VF)
+    atomic_move(nlog, bname + "-key.pem", bname + ".key", VF)
     wunlink(nlog, bname + ".csr", VF)
 
     log("cert", "new ca OK", 2)
@@ -215,7 +213,7 @@ def _gen_srv(log: "RootLogger", args, netdevs: dict[str, Netdev]):
         wunlink(nlog, bname + ".key", VF)
     except:
         pass
-    wrename(nlog, bname + "-key.pem", bname + ".key", VF)
+    atomic_move(nlog, bname + "-key.pem", bname + ".key", VF)
     wunlink(nlog, bname + ".csr", VF)
 
     with open(os.path.join(args.crt_dir, "ca.pem"), "rb") as f:

copyparty/cfg.py

@@ -5,6 +5,9 @@ from __future__ import print_function, unicode_literals
 zs = "a c e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vp e2vu ed emp i j lo mcr mte mth mtm mtp nb nc nid nih nth nw p q s ss sss v z zv"
 onedash = set(zs.split())
 
+# verify that all volflags are documented here:
+# grep volflag= __main__.py | sed -r 's/.*volflag=//;s/\).*//' | sort | uniq | while IFS= read -r x; do grep -E "\"$x(=[^ \"]+)?\": \"" cfg.py || printf '%s\n' "$x"; done
+
 
 def vf_bmap() -> dict[str, str]:
     """argv-to-volflag: simple bools"""
@@ -13,11 +16,13 @@ def vf_bmap() -> dict[str, str]:
         "dav_rt": "davrt",
         "ed": "dots",
         "hardlink_only": "hardlinkonly",
+        "no_clone": "noclone",
         "no_dirsz": "nodirsz",
         "no_dupe": "nodupe",
         "no_forget": "noforget",
         "no_pipe": "nopipe",
         "no_robots": "norobots",
+        "no_tail": "notail",
         "no_thumb": "dthumb",
         "no_vthumb": "dvthumb",
         "no_athumb": "dathumb",
@@ -39,15 +44,22 @@ def vf_bmap() -> dict[str, str]:
         "gsel",
         "hardlink",
         "magic",
+        "no_db_ip",
         "no_sb_md",
         "no_sb_lg",
+        "nsort",
         "og",
         "og_no_head",
         "og_s_title",
         "rand",
+        "reflink",
+        "rmagic",
+        "rss",
+        "wo_up_readme",
         "xdev",
         "xlink",
         "xvol",
+        "zipmaxu",
     ):
         ret[k] = k
     return ret
@@ -56,6 +68,7 @@ def vf_bmap() -> dict[str, str]:
 def vf_vmap() -> dict[str, str]:
     """argv-to-volflag: simple values"""
     ret = {
+        "ac_convt": "aconvt",
         "no_hash": "nohash",
         "no_idx": "noidx",
         "re_maxage": "scan",
@@ -66,11 +79,20 @@ def vf_vmap() -> dict[str, str]:
         "th_x3": "th3x",
     }
     for k in (
+        "bup_ck",
+        "chmod_d",
+        "chmod_f",
         "dbd",
+        "forget_ip",
+        "hsortn",
         "html_head",
         "lg_sbf",
         "md_sbf",
+        "lg_sba",
+        "md_sba",
+        "md_hist",
         "nrand",
+        "u2ow",
         "og_desc",
         "og_site",
         "og_th",
@@ -80,13 +102,29 @@ def vf_vmap() -> dict[str, str]:
         "og_title_i",
         "og_tpl",
         "og_ua",
+        "put_ck",
+        "put_name",
         "mv_retry",
         "rm_retry",
         "sort",
+        "tail_fd",
+        "tail_rate",
+        "tail_tmax",
+        "tail_who",
         "tcolor",
+        "th_spec_p",
+        "txt_eol",
         "unlist",
         "u2abort",
         "u2ts",
+        "uid",
+        "gid",
+        "unp_who",
+        "ups_who",
+        "zip_who",
+        "zipmaxn",
+        "zipmaxs",
+        "zipmaxt",
     ):
         ret[k] = k
     return ret
@@ -98,13 +136,16 @@ def vf_cmap() -> dict[str, str]:
     for k in (
         "exp_lg",
         "exp_md",
+        "ext_th",
         "mte",
         "mth",
         "mtp",
+        "xac",
         "xad",
         "xar",
         "xau",
         "xban",
+        "xbc",
         "xbd",
         "xbr",
         "xbu",
@@ -134,13 +175,24 @@ flagcats = {
         "dedup": "enable symlink-based file deduplication",
         "hardlink": "enable hardlink-based file deduplication,\nwith fallback on symlinks when that is impossible",
         "hardlinkonly": "dedup with hardlink only, never symlink;\nmake a full copy if hardlink is impossible",
+        "reflink": "enable reflink-based file deduplication,\nwith fallback on full copy when that is impossible",
         "safededup": "verify on-disk data before using it for dedup",
-        "nodupe": "rejects existing files (instead of symlinking them)",
+        "noclone": "take dupe data from clients, even if available on HDD",
+        "nodupe": "rejects existing files (instead of linking/cloning them)",
+        "chmod_d=755": "unix-permission for new dirs/folders",
+        "chmod_f=644": "unix-permission for new files",
+        "uid=573": "change owner of new files/folders to unix-user 573",
+        "gid=999": "change owner of new files/folders to unix-group 999",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
+        "nosparse": "deny use of sparse files, mainly for slow storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
         "magic": "enables filetype detection for nameless uploads",
-        "gz": "allows server-side gzip of uploads with ?gz (also c,xz)",
+        "put_name": "fallback filename for nameless uploads",
+        "put_ck": "default checksum-hasher for PUT/WebDAV uploads",
+        "bup_ck": "default checksum-hasher for bup/basic uploads",
+        "gz": "allows server-side gzip compression of uploads with ?gz",
+        "xz": "allows server-side lzma compression of uploads with ?xz",
         "pk": "forces server-side compression, optional arg: xz,9",
     },
     "upload rules": {
@@ -149,8 +201,10 @@ flagcats = {
         "vmaxb=1g": "total volume size max 1 GiB (suffixes: b, k, m, g, t)",
         "vmaxn=4k": "max 4096 files in volume (suffixes: b, k, m, g, t)",
         "medialinks": "return medialinks for non-up2k uploads (not hotlinks)",
+        "wo_up_readme": "write-only users can upload logues without getting renamed",
         "rand": "force randomized filenames, 9 chars long by default",
         "nrand=N": "randomized filenames are N chars long",
+        "u2ow=N": "overwrite existing files? 0=no 1=if-older 2=always",
         "u2ts=fc": "[f]orce [c]lient-last-modified or [u]pload-time",
         "u2abort=1": "allow aborting unfinished uploads? 0=no 1=strict 2=ip-chk 3=acct-chk",
         "sz=1k-3m": "allow filesizes between 1 KiB and 3MiB",
@@ -167,17 +221,24 @@ flagcats = {
         "e2dsa": "scans all folders for new files on startup; also sets -e2d",
         "e2t": "enable multimedia indexing; makes it possible to search for tags",
         "e2ts": "scan existing files for tags on startup; also sets -e2t",
-        "e2tsa": "delete all metadata from DB (full rescan); also sets -e2ts",
+        "e2tsr": "delete all metadata from DB (full rescan); also sets -e2ts",
         "d2ts": "disables metadata collection for existing files",
+        "e2v": "verify integrity on startup by hashing files and comparing to db",
+        "e2vu": "when e2v fails, update the db (assume on-disk files are good)",
+        "e2vp": "when e2v fails, panic and quit copyparty",
         "d2ds": "disables onboot indexing, overrides -e2ds*",
         "d2t": "disables metadata collection, overrides -e2t*",
         "d2v": "disables file verification, overrides -e2v*",
         "d2d": "disables all database stuff, overrides -e2*",
         "hist=/tmp/cdb": "puts thumbnails and indexes at that location",
+        "dbpath=/tmp/cdb": "puts indexes at that location",
+        "landmark=foo": "disable db if file foo doesn't exist",
         "scan=60": "scan for new files every 60sec, same as --re-maxage",
         "nohash=\\.iso$": "skips hashing file contents if path matches *.iso",
         "noidx=\\.iso$": "fully ignores the contents at paths matching *.iso",
         "noforget": "don't forget files when deleted from disk",
+        "forget_ip=43200": "forget uploader-IP after 30 days (GDPR)",
+        "no_db_ip": "never store uploader-IP in the db; disables unpost",
         "fat32": "avoid excessive reindexing on android sdcardfs",
         "dbd=[acid|swal|wal|yolo]": "database speed-durability tradeoff",
         "xlink": "cross-volume dupe detection / linking (dangerous)",
@@ -185,8 +246,11 @@ flagcats = {
         "xvol": "do not follow symlinks leaving the volume root",
         "dotsrch": "show dotfiles in search results",
         "nodotsrch": "hide dotfiles in search results (default)",
+        "srch_excl": "exclude search results with URL matching this regex",
     },
     'database, audio tags\n"mte", "mth", "mtp", "mtm" all work the same as -mte, -mth, ...': {
+        "mte=artist,title": "media-tags to index/display",
+        "mth=fmt,res,ac": "media-tags to hide by default",
         "mtp=.bpm=f,audio-bpm.py": 'uses the "audio-bpm.py" program to\ngenerate ".bpm" tags from uploads (f = overwrite tags)',
         "mtp=ahash,vhash=media-hash.py": "collects two tags at once",
     },
@@ -199,7 +263,10 @@ flagcats = {
         "thsize": "thumbnail res; WxH",
         "crop": "center-cropping (y/n/fy/fn)",
         "th3x": "3x resolution (y/n/fy/fn)",
-        "convt": "conversion timeout in seconds",
+        "convt": "convert-to-image timeout in seconds",
+        "aconvt": "convert-to-audio timeout in seconds",
+        "th_spec_p=1": "make spectrograms? 0=never 1=fallback 2=always",
+        "ext_th=s=/b.png": "use /b.png as thumbnail for file-extension s",
     },
     "handlers\n(better explained in --help-handlers)": {
         "on404=PY": "handle 404s by executing PY file",
@@ -209,6 +276,8 @@ flagcats = {
         "xbu=CMD": "execute CMD before a file upload starts",
         "xau=CMD": "execute CMD after  a file upload finishes",
         "xiu=CMD": "execute CMD after  all uploads finish and volume is idle",
+        "xbc=CMD": "execute CMD before a file copy",
+        "xac=CMD": "execute CMD after  a file copy",
         "xbr=CMD": "execute CMD before a file rename/move",
         "xar=CMD": "execute CMD after  a file rename/move",
         "xbd=CMD": "execute CMD before a file delete",
@@ -220,22 +289,71 @@ flagcats = {
         "grid": "show grid/thumbnails by default",
         "gsel": "select files in grid by ctrl-click",
         "sort": "default sort order",
+        "nsort": "natural-sort of leading digits in filenames",
+        "hsortn": "number of sort-rules to add to media URLs",
         "unlist": "dont list files matching REGEX",
         "html_head=TXT": "includes TXT in the <head>, or @PATH for file at PATH",
+        "tcolor=#fc0": "theme color (a hint for webbrowsers, discord, etc.)",
+        "nodirsz": "don't show total folder size",
         "robots": "allows indexing by search engines (default)",
         "norobots": "kindly asks search engines to leave",
+        "unlistcr": "don't list read-access in controlpanel",
+        "unlistcw": "don't list write-access in controlpanel",
         "no_sb_md": "disable js sandbox for markdown files",
         "no_sb_lg": "disable js sandbox for prologue/epilogue",
         "sb_md": "enable js sandbox for markdown files (default)",
         "sb_lg": "enable js sandbox for prologue/epilogue (default)",
         "md_sbf": "list of markdown-sandbox safeguards to disable",
         "lg_sbf": "list of *logue-sandbox safeguards to disable",
+        "md_sba": "value of iframe allow-prop for markdown-sandbox",
+        "lg_sba": "value of iframe allow-prop for *logue-sandbox",
         "nohtml": "return html and markdown as text/html",
     },
+    "opengraph (discord embeds)": {
+        "og": "enable OG (disables hotlinking)",
+        "og_site": "sitename; defaults to --name, disable with '-'",
+        "og_desc": "description text for all files; disable with '-'",
+        "og_th=jf": "thumbnail format; j / jf / jf3 / w / w3 / ...",
+        "og_title_a": "audio title format; default: {{ artist }} - {{ title }}",
+        "og_title_v": "video title format; default: {{ title }}",
+        "og_title_i": "image title format; default: {{ title }}",
+        "og_title=foo": "fallback title if there's nothing in the db",
+        "og_s_title": "force default title; do not read from tags",
+        "og_tpl": "custom html; see --og-tpl in --help",
+        "og_no_head": "you want to add tags manually with og_tpl",
+        "og_ua": "if defined: only send OG html if useragent matches this regex",
+    },
+    "textfiles": {
+        "md_hist": "where to put markdown backups; s=subfolder, v=volHist, n=nope",
+        "exp": "enable textfile expansion; see --help-exp",
+        "exp_md": "placeholders to expand in markdown files; see --help",
+        "exp_lg": "placeholders to expand in prologue/epilogue; see --help",
+        "txt_eol=lf": "enable EOL conversion when writing docs (LF or CRLF)",
+    },
+    "tailing": {
+        "notail": "disable ?tail (download a growing file continuously)",
+        "tail_fd=1": "check if file was replaced (new fd) every 1 sec",
+        "tail_rate=0.2": "check for new data every 0.2 sec",
+        "tail_tmax=30": "kill connection after 30 sec",
+        "tail_who=2": "restrict ?tail access (1=admins,2=authed,3=everyone)",
+    },
     "others": {
         "dots": "allow all users with read-access to\nenable the option to show dotfiles in listings",
         "fk=8": 'generates per-file accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
         "fka=8": 'generates slightly weaker per-file accesskeys,\nwhich are then required at the "g" permission;\nnot affected by filesize or inode numbers',
+        "dk=8": 'generates per-directory accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
+        "dks": "per-directory accesskeys allow browsing into subdirs",
+        "dky": 'allow seeing files (not folders) inside a specific folder\nwith "g" perm, and does not require a valid dirkey to do so',
+        "rss": "allow '?rss' URL suffix (experimental)",
+        "rmagic": "expensive analysis for mimetype accuracy",
+        "unp_who=2": "unpost only if same... 1=ip+name, 2=ip, 3=name",
+        "ups_who=2": "restrict viewing the list of recent uploads",
+        "zip_who=2": "restrict access to download-as-zip/tar",
+        "zipmaxn=9k": "reject download-as-zip if more than 9000 files",
+        "zipmaxs=2g": "reject download-as-zip if size over 2 GiB",
+        "zipmaxt=no": "reply with 'no' if download-as-zip exceeds max",
+        "zipmaxu": "zip-size-limit does not apply to authenticated users",
+        "nopipe": "disable race-the-beam (download unfinished uploads)",
         "mv_retry": "ms-windows: timeout for renaming busy files",
         "rm_retry": "ms-windows: timeout for deleting busy files",
         "davauth": "ask webdav clients to login for all folders",
@@ -245,3 +363,10 @@ flagcats = {
 
 
 flagdescs = {k.split("=")[0]: v for tab in flagcats.values() for k, v in tab.items()}
+
+
+if True:  # so it gets removed in release-builds
+    for fun in [vf_bmap, vf_cmap, vf_vmap]:
+        for k in fun().values():
+            if k not in flagdescs:
+                raise Exception("undocumented volflag: " + k)

copyparty/dxml.py

@@ -1,3 +1,6 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
 import importlib
 import sys
 import xml.etree.ElementTree as ET
@@ -8,6 +11,10 @@ if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
 
+class BadXML(Exception):
+    pass
+
+
 def get_ET() -> ET.XMLParser:
     pn = "xml.etree.ElementTree"
     cn = "_elementtree"
@@ -34,7 +41,7 @@ def get_ET() -> ET.XMLParser:
 XMLParser: ET.XMLParser = get_ET()
 
 
-class DXMLParser(XMLParser):  # type: ignore
+class _DXMLParser(XMLParser):  # type: ignore
     def __init__(self) -> None:
         tb = ET.TreeBuilder()
         super(DXMLParser, self).__init__(target=tb)
@@ -49,16 +56,57 @@ class DXMLParser(XMLParser):  # type: ignore
         raise BadXML("{}, {}".format(a, ka))
 
 
-class BadXML(Exception):
-    pass
+class _NG(XMLParser):  # type: ignore
+    def __int__(self) -> None:
+        raise BadXML("dxml selftest failed")
+
+
+DXMLParser = _DXMLParser
 
 
 def parse_xml(txt: str) -> ET.Element:
+    """
+    Parse XML into an xml.etree.ElementTree.Element while defusing some unsafe parts.
+    """
     parser = DXMLParser()
     parser.feed(txt)
     return parser.close()  # type: ignore
 
 
+def selftest() -> bool:
+    qbe = r"""<!DOCTYPE d [
+<!ENTITY a "nice_bakuretsu">
+]>
+<root>&a;&a;&a;</root>"""
+
+    emb = r"""<!DOCTYPE d [
+<!ENTITY a SYSTEM "file:///etc/hostname">
+]>
+<root>&a;</root>"""
+
+    # future-proofing; there's never been any known vulns
+    # regarding DTDs and ET.XMLParser, but might as well
+    # block them since webdav-clients don't use them
+    dtd = r"""<!DOCTYPE d SYSTEM "a.dtd">
+<root>a</root>"""
+
+    for txt in (qbe, emb, dtd):
+        try:
+            parse_xml(txt)
+            t = "WARNING: dxml selftest failed:\n%s\n"
+            print(t % (txt,), file=sys.stderr)
+            return False
+        except BadXML:
+            pass
+
+    return True
+
+
+DXML_OK = selftest()
+if not DXML_OK:
+    DXMLParser = _NG
+
+
 def mktnod(name: str, text: str) -> ET.Element:
     el = ET.Element(name)
     el.text = text

copyparty/fsutil.py

@@ -42,14 +42,14 @@ class Fstab(object):
             self.cache = {}
 
         fs = "ext4"
-        msg = "failed to determine filesystem at [{}]; assuming {}\n{}"
+        msg = "failed to determine filesystem at %r; assuming %s\n%s"
 
         if ANYWIN:
             fs = "vfat"
             try:
                 path = self._winpath(path)
             except:
-                self.log(msg.format(path, fs, min_ex()), 3)
+                self.log(msg % (path, fs, min_ex()), 3)
                 return fs
 
         path = undot(path)
@@ -61,11 +61,11 @@ class Fstab(object):
         try:
             fs = self.get_w32(path) if ANYWIN else self.get_unix(path)
         except:
-            self.log(msg.format(path, fs, min_ex()), 3)
+            self.log(msg % (path, fs, min_ex()), 3)
 
         fs = fs.lower()
         self.cache[path] = fs
-        self.log("found {} at {}".format(fs, path))
+        self.log("found %s at %r" % (fs, path))
         return fs
 
     def _winpath(self, path: str) -> str:
@@ -78,7 +78,7 @@ class Fstab(object):
             return vid
 
     def build_fallback(self) -> None:
-        self.tab = VFS(self.log_func, "idk", "/", AXS(), {})
+        self.tab = VFS(self.log_func, "idk", "/", "/", AXS(), {})
         self.trusted = False
 
     def build_tab(self) -> None:
@@ -111,9 +111,10 @@ class Fstab(object):
 
         tab1.sort(key=lambda x: (len(x[0]), x[0]))
         path1, fs1 = tab1[0]
-        tab = VFS(self.log_func, fs1, path1, AXS(), {})
+        tab = VFS(self.log_func, fs1, path1, path1, AXS(), {})
         for path, fs in tab1[1:]:
-            tab.add(fs, path.lstrip("/"))
+            zs = path.lstrip("/")
+            tab.add(fs, zs, zs)
 
         self.tab = tab
         self.srctab = srctab
@@ -130,9 +131,10 @@ class Fstab(object):
         if not self.trusted:
             # no mtab access; have to build as we go
             if "/" in rem:
-                self.tab.add("idk", os.path.join(vn.vpath, rem.split("/")[0]))
+                zs = os.path.join(vn.vpath, rem.split("/")[0])
+                self.tab.add("idk", zs, zs)
             if rem:
-                self.tab.add(nval, path)
+                self.tab.add(nval, path, path)
             else:
                 vn.realpath = nval
 

copyparty/ftpd.py

@@ -19,6 +19,7 @@ from .__init__ import PY2, TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
 from .util import (
+    FN_EMB,
     VF_CAREFUL,
     Daemon,
     ODict,
@@ -30,6 +31,7 @@ from .util import (
     relchk,
     runhook,
     sanitize_fn,
+    set_fperms,
     vjoin,
     wunlink,
 )
@@ -76,16 +78,29 @@ class FtpAuth(DummyAuthorizer):
             else:
                 raise AuthenticationFailed("banned")
 
+        args = self.hub.args
         asrv = self.hub.asrv
         uname = "*"
         if username != "anonymous":
             uname = ""
-            for zs in (password, username):
+            if args.usernames:
+                alts = ["%s:%s" % (username, password)]
+            else:
+                alts = password, username
+
+            for zs in alts:
                 zs = asrv.iacct.get(asrv.ah.hash(zs), "")
                 if zs:
                     uname = zs
                     break
 
+        if args.ipu and uname == "*":
+            uname = args.ipu_iu[args.ipu_nm.map(ip)]
+        if args.ipr and uname in args.ipr_u:
+            if not args.ipr_u[uname].map(ip):
+                logging.warning("username [%s] rejected by --ipr", uname)
+                uname = "*"
+
         if not uname or not (asrv.vfs.aread.get(uname) or asrv.vfs.awrite.get(uname)):
             g = self.hub.gpwd
             if g.lim:
@@ -166,6 +181,16 @@ class FtpFs(AbstractedFS):
             fn = sanitize_fn(fn or "", "")
             vpath = vjoin(rd, fn)
             vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, r, w, m, d)
+            if (
+                w
+                and fn.lower() in FN_EMB
+                and self.h.uname not in vfs.axs.uread
+                and "wo_up_readme" not in vfs.flags
+            ):
+                fn = "_wo_" + fn
+                vpath = vjoin(rd, fn)
+                vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, r, w, m, d)
+
             if not vfs.realpath:
                 t = "No filesystem mounted at [{}]"
                 raise FSE(t.format(vpath))
@@ -214,7 +239,7 @@ class FtpFs(AbstractedFS):
         r = "r" in mode
         w = "w" in mode or "a" in mode or "+" in mode
 
-        ap = self.rv2a(filename, r, w)[0]
+        ap, vfs, _ = self.rv2a(filename, r, w)
         self.validpath(ap)
         if w:
             try:
@@ -246,7 +271,11 @@ class FtpFs(AbstractedFS):
 
             wunlink(self.log, ap, VF_CAREFUL)
 
-        return open(fsenc(ap), mode, self.args.iobuf)
+        ret = open(fsenc(ap), mode, self.args.iobuf)
+        if w and "fperms" in vfs.flags:
+            set_fperms(ret, vfs.flags)
+
+        return ret
 
     def chdir(self, path: str) -> None:
         nwd = join(self.cwd, path)
@@ -260,9 +289,12 @@ class FtpFs(AbstractedFS):
             # returning 550 is library-default and suitable
             raise FSE("No such file or directory")
 
-        avfs = vfs.chk_ap(ap, st)
-        if not avfs:
-            raise FSE("Permission denied", 1)
+        if vfs.realpath:
+            avfs = vfs.chk_ap(ap, st)
+            if not avfs:
+                raise FSE("Permission denied", 1)
+        else:
+            avfs = vfs
 
         self.cwd = nwd
         (
@@ -277,8 +309,8 @@ class FtpFs(AbstractedFS):
         ) = avfs.can_access("", self.h.uname)
 
     def mkdir(self, path: str) -> None:
-        ap = self.rv2a(path, w=True)[0]
-        bos.makedirs(ap)  # filezilla expects this
+        ap, vfs, _ = self.rv2a(path, w=True)
+        bos.makedirs(ap, vf=vfs.flags)  # filezilla expects this
 
     def listdir(self, path: str) -> list[str]:
         vpath = join(self.cwd, path)
@@ -292,6 +324,7 @@ class FtpFs(AbstractedFS):
                 self.uname,
                 not self.args.no_scandir,
                 [[True, False], [False, True]],
+                throw=True,
             )
             vfs_ls = [x[0] for x in vfs_ls1]
             vfs_ls.extend(vfs_virt.keys())
@@ -376,8 +409,12 @@ class FtpFs(AbstractedFS):
             return st
 
     def utime(self, path: str, timeval: float) -> None:
-        ap = self.rv2a(path, w=True)[0]
-        return bos.utime(ap, (timeval, timeval))
+        try:
+            ap = self.rv2a(path, w=True)[0]
+            return bos.utime(ap, (int(time.time()), int(timeval)))
+        except Exception as ex:
+            logging.error("ftp.utime: %s, %r", ex, ex)
+            raise
 
     def lstat(self, path: str) -> os.stat_result:
         ap = self.rv2a(path)[0]
@@ -466,7 +503,11 @@ class FtpHandler(FTPHandler):
     def ftp_STOR(self, file: str, mode: str = "w") -> Any:
         # Optional[str]
         vp = join(self.fs.cwd, file).lstrip("/")
-        ap, vfs, rem = self.fs.v2a(vp, w=True)
+        try:
+            ap, vfs, rem = self.fs.v2a(vp, w=True)
+        except Exception as ex:
+            self.respond("550 %s" % (ex,), logging.info)
+            return
         self.vfs_map[ap] = vp
         xbu = vfs.flags.get("xbu")
         if xbu and not runhook(
@@ -586,7 +627,7 @@ class Ftpd(object):
         if "::" in ips:
             ips.append("0.0.0.0")
 
-        ips = [x for x in ips if "unix:" not in x]
+        ips = [x for x in ips if not x.startswith(("unix:", "fd:"))]
 
         if self.args.ftp4:
             ips = [x for x in ips if ":" not in x]

copyparty/httpconn.py

@@ -59,6 +59,8 @@ class HttpConn(object):
         self.asrv: AuthSrv = hsrv.asrv  # mypy404
         self.u2fh: Util.FHC = hsrv.u2fh  # mypy404
         self.pipes: Util.CachedDict = hsrv.pipes  # mypy404
+        self.ipu_iu: Optional[dict[str, str]] = hsrv.ipu_iu
+        self.ipu_nm: Optional[NetMap] = hsrv.ipu_nm
         self.ipa_nm: Optional[NetMap] = hsrv.ipa_nm
         self.xff_nm: Optional[NetMap] = hsrv.xff_nm
         self.xff_lan: NetMap = hsrv.xff_lan  # type: ignore
@@ -222,3 +224,6 @@ class HttpConn(object):
             if self.u2idx:
                 self.hsrv.put_u2idx(str(self.addr), self.u2idx)
                 self.u2idx = None
+
+            if self.rproxy:
+                self.set_rproxy()

copyparty/httpsrv.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import hashlib
 import math
 import os
 import re
@@ -69,6 +70,8 @@ from .util import (
     build_netmap,
     has_resource,
     ipnorm,
+    load_ipr,
+    load_ipu,
     load_resource,
     min_ex,
     shut_socket,
@@ -79,6 +82,7 @@ from .util import (
 )
 
 if TYPE_CHECKING:
+    from .authsrv import VFS
     from .broker_util import BrokerCli
     from .ssdp import SSDPr
 
@@ -120,6 +124,7 @@ class HttpSrv(object):
         self.nm = NetMap([], [])
         self.ssdp: Optional["SSDPr"] = None
         self.gpwd = Garda(self.args.ban_pw)
+        self.gpwc = Garda(self.args.ban_pwc)
         self.g404 = Garda(self.args.ban_404)
         self.g403 = Garda(self.args.ban_403)
         self.g422 = Garda(self.args.ban_422, False)
@@ -128,6 +133,12 @@ class HttpSrv(object):
         self.bans: dict[str, int] = {}
         self.aclose: dict[str, int] = {}
 
+        dli: dict[str, tuple[float, int, "VFS", str, str]] = {}  # info
+        dls: dict[str, tuple[float, int]] = {}  # state
+        self.dli = self.tdli = dli
+        self.dls = self.tdls = dls
+        self.iiam = '<img src="%s.cpr/iiam.gif?cache=i" />' % (self.args.SRS,)
+
         self.bound: set[tuple[str, int]] = set()
         self.name = "hsrv" + nsuf
         self.mutex = threading.Lock()
@@ -143,6 +154,7 @@ class HttpSrv(object):
         self.t_periodic: Optional[threading.Thread] = None
 
         self.u2fh = FHC()
+        self.u2sc: dict[str, tuple[int, "hashlib._Hash"]] = {}
         self.pipes = CachedDict(0.2)
         self.metrics = Metrics(self)
         self.nreq = 0
@@ -162,27 +174,35 @@ class HttpSrv(object):
         env = jinja2.Environment()
         env.loader = jinja2.FunctionLoader(lambda f: load_jinja2_resource(self.E, f))
         jn = [
-            "splash",
-            "shares",
-            "svcs",
             "browser",
             "browser2",
-            "msg",
+            "cf",
+            "idp",
             "md",
             "mde",
-            "cf",
+            "msg",
+            "rups",
+            "shares",
+            "splash",
+            "svcs",
         ]
         self.j2 = {x: env.get_template(x + ".html") for x in jn}
         self.prism = has_resource(self.E, "web/deps/prism.js.gz")
 
+        if self.args.ipu:
+            self.ipu_iu, self.ipu_nm = load_ipu(self.log, self.args.ipu)
+        else:
+            self.ipu_iu = self.ipu_nm = None
+
+        if self.args.ipr:
+            self.ipr = load_ipr(self.log, self.args.ipr)
+        else:
+            self.ipr = None
+
         self.ipa_nm = build_netmap(self.args.ipa)
         self.xff_nm = build_netmap(self.args.xff_src)
         self.xff_lan = build_netmap("lan")
 
-        self.ptn_cc = re.compile(r"[\x00-\x1f]")
-        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
-        self.uparam_cc_ok = set("doc move tree".split())
-
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:
             zs = "PROPFIND PROPPATCH LOCK UNLOCK MKCOL COPY MOVE"
@@ -197,6 +217,9 @@ class HttpSrv(object):
             self.start_threads(4)
 
         if nid:
+            self.tdli = {}
+            self.tdls = {}
+
             if self.args.stackmon:
                 start_stackmon(self.args.stackmon, nid)
 
@@ -298,6 +321,8 @@ class HttpSrv(object):
 
         Daemon(self.broker.say, "sig-hsrv-up1", ("cb_httpsrv_up",))
 
+        saddr = ("", 0)  # fwd-decl for `except TypeError as ex:`
+
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="90")
@@ -305,7 +330,8 @@ class HttpSrv(object):
             spins = 0
             while self.ncli >= self.nclimax:
                 if not spins:
-                    self.log(self.name, "at connection limit; waiting", 3)
+                    t = "at connection limit (global-option 'nc'); waiting"
+                    self.log(self.name, t, 3)
 
                 spins += 1
                 time.sleep(0.1)
@@ -379,6 +405,19 @@ class HttpSrv(object):
                 self.log(self.name, "accept({}): {}".format(fno, ex), c=6)
                 time.sleep(0.02)
                 continue
+            except TypeError as ex:
+                # on macOS, accept() may return a None saddr if blocked by LittleSnitch;
+                # unicode(saddr[0]) ==> TypeError: 'NoneType' object is not subscriptable
+                if tcp and not saddr:
+                    t = "accept(%s): failed to accept connection from client due to firewall or network issue"
+                    self.log(self.name, t % (fno,), c=3)
+                    try:
+                        sck.close()  # type: ignore
+                    except:
+                        pass
+                    time.sleep(0.02)
+                    continue
+                raise
 
             if self.args.log_conn:
                 t = "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(
@@ -571,3 +610,32 @@ class HttpSrv(object):
                 ident += "a"
 
             self.u2idx_free[ident] = u2idx
+
+    def read_dls(
+        self,
+    ) -> tuple[
+        dict[str, tuple[float, int, str, str, str]], dict[str, tuple[float, int]]
+    ]:
+        """
+        mp-broker asking for local dl-info + dl-state;
+        reduce overhead by sending just the vfs vpath
+        """
+        dli = {k: (a, b, c.vpath, d, e) for k, (a, b, c, d, e) in self.dli.items()}
+        return (dli, self.dls)
+
+    def write_dls(
+        self,
+        sdli: dict[str, tuple[float, int, str, str, str]],
+        dls: dict[str, tuple[float, int]],
+    ) -> None:
+        """
+        mp-broker pushing total dl-info + dl-state;
+        swap out the vfs vpath with the vfs node
+        """
+        dli: dict[str, tuple[float, int, "VFS", str, str]] = {}
+        for k, (a, b, c, d, e) in sdli.items():
+            vn = self.asrv.vfs.all_nodes[c]
+            dli[k] = (a, b, vn, d, e)
+
+        self.tdli = dli
+        self.tdls = dls

copyparty/ico.py

@@ -94,10 +94,21 @@ class Ico(object):
 <?xml version="1.0" encoding="UTF-8"?>
 <svg version="1.1" viewBox="0 0 100 {}" xmlns="http://www.w3.org/2000/svg"><g>
 <rect width="100%" height="100%" fill="#{}" />
-<text x="50%" y="50%" dominant-baseline="middle" text-anchor="middle" xml:space="preserve"
+<text x="50%" y="{}" dominant-baseline="middle" text-anchor="middle" xml:space="preserve"
   fill="#{}" font-family="monospace" font-size="14px" style="letter-spacing:.5px">{}</text>
 </g></svg>
 """
-        svg = svg.format(h, c[:6], c[6:], html_escape(ext, True))
+
+        txt = html_escape(ext, True)
+        if "\n" in txt:
+            lines = txt.split("\n")
+            n = len(lines)
+            y = "20%" if n == 2 else "10%" if n == 3 else "0"
+            zs = '<tspan x="50%%" dy="1.2em">%s</tspan>'
+            txt = "".join([zs % (x,) for x in lines])
+        else:
+            y = "50%"
+
+        svg = svg.format(h, c[:6], y, c[6:], txt)
 
         return "image/svg+xml", svg.encode("utf-8")

copyparty/mdns.py

@@ -25,6 +25,7 @@ from .stolen.dnslib import (
     DNSHeader,
     DNSQuestion,
     DNSRecord,
+    set_avahi_379,
 )
 from .util import CachedSet, Daemon, Netdev, list_ips, min_ex
 
@@ -72,7 +73,11 @@ class MDNS(MCast):
         self.ngen = ngen
         self.ttl = 300
 
-        zs = self.args.name + ".local."
+        if not self.args.zm_nwa_1:
+            set_avahi_379()
+
+        zs = self.args.zm_fqdn or (self.args.name + ".local")
+        zs = zs.replace("--name", self.args.name).rstrip(".") + "."
         zs = zs.encode("ascii", "replace").decode("ascii", "replace")
         self.hn = "-".join(x for x in zs.split("?") if x) or (
             "vault-{}".format(random.randint(1, 255))
@@ -336,6 +341,9 @@ class MDNS(MCast):
                         self.log("stopped", 2)
                         return
 
+                    if self.args.zm_no_pe:
+                        continue
+
                     t = "{} {} \033[33m|{}| {}\n{}".format(
                         self.srv[sck].name, addr, len(buf), repr(buf)[2:-1], min_ex()
                     )

copyparty/metrics.py

@@ -18,7 +18,7 @@ class Metrics(object):
 
     def tx(self, cli: "HttpCli") -> bool:
         if not cli.avol:
-            raise Pebkac(403, "not allowed for user " + cli.uname)
+            raise Pebkac(403, "'stats' not allowed for user " + cli.uname)
 
         args = cli.args
         if not args.stats:
@@ -72,6 +72,9 @@ class Metrics(object):
         v = "{:.3f}".format(self.hsrv.t0)
         addug("cpp_boot_unixtime", "seconds", v, t)
 
+        t = "number of active downloads"
+        addg("cpp_active_dl", str(len(self.hsrv.tdls)), t)
+
         t = "number of open http(s) client connections"
         addg("cpp_http_conns", str(self.hsrv.ncli), t)
 
@@ -128,7 +131,7 @@ class Metrics(object):
             addbh("cpp_disk_size_bytes", "total HDD size of volume")
             addbh("cpp_disk_free_bytes", "free HDD space in volume")
             for vpath, vol in allvols:
-                free, total = get_df(vol.realpath)
+                free, total, _ = get_df(vol.realpath, False)
                 if free is None or total is None:
                     continue
 

copyparty/mtag.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import argparse
 import json
 import os
+import re
 import shutil
 import subprocess as sp
 import sys
@@ -17,6 +18,7 @@ from .util import (
     REKOBO_LKEY,
     VF_CAREFUL,
     fsenc,
+    gzip,
     min_ex,
     pybin,
     retchk,
@@ -27,7 +29,7 @@ from .util import (
 )
 
 if True:  # pylint: disable=using-constant-test
-    from typing import Any, Optional, Union
+    from typing import IO, Any, Optional, Union
 
     from .util import NamedLogger, RootLogger
 
@@ -62,6 +64,11 @@ def have_ff(scmd: str) -> bool:
 HAVE_FFMPEG = not os.environ.get("PRTY_NO_FFMPEG") and have_ff("ffmpeg")
 HAVE_FFPROBE = not os.environ.get("PRTY_NO_FFPROBE") and have_ff("ffprobe")
 
+CBZ_PICS = set("png jpg jpeg gif bmp tga tif tiff webp avif".split())
+CBZ_01 = re.compile(r"(^|[^0-9v])0+[01]\b")
+
+FMT_AU = set("mp3 ogg flac wav".split())
+
 
 class MParser(object):
     def __init__(self, cmdline: str) -> None:
@@ -126,6 +133,7 @@ def au_unpk(
     log: "NamedLogger", fmt_map: dict[str, str], abspath: str, vn: Optional[VFS] = None
 ) -> str:
     ret = ""
+    maxsz = 1024 * 1024 * 64
     try:
         ext = abspath.split(".")[-1].lower()
         au, pk = fmt_map[ext].split(".")
@@ -133,8 +141,6 @@ def au_unpk(
         fd, ret = tempfile.mkstemp("." + au)
 
         if pk == "gz":
-            import gzip
-
             fi = gzip.GzipFile(abspath, mode="rb")
 
         elif pk == "xz":
@@ -148,24 +154,52 @@ def au_unpk(
             zf = zipfile.ZipFile(abspath, "r")
             zil = zf.infolist()
             zil = [x for x in zil if x.filename.lower().split(".")[-1] == au]
+            if not zil:
+                raise Exception("no audio inside zip")
             fi = zf.open(zil[0])
 
+        elif pk == "cbz":
+            import zipfile
+
+            zf = zipfile.ZipFile(abspath, "r")
+            znil = [(x.filename.lower(), x) for x in zf.infolist()]
+            nf = len(znil)
+            znil = [x for x in znil if x[0].split(".")[-1] in CBZ_PICS]
+            znil = [x for x in znil if "cover" in x[0]] or znil
+            znil = [x for x in znil if CBZ_01.search(x[0])] or znil
+            t = "cbz: %d files, %d hits" % (nf, len(znil))
+            using = sorted(znil)[0][1].filename
+            if znil:
+                t += ", using " + using
+            log(t)
+            if not znil:
+                raise Exception("no images inside cbz")
+            fi = zf.open(using)
+
+        elif pk == "epub":
+            fi = get_cover_from_epub(log, abspath)
+
         else:
             raise Exception("unknown compression %s" % (pk,))
 
+        fsz = 0
         with os.fdopen(fd, "wb") as fo:
             while True:
                 buf = fi.read(32768)
                 if not buf:
                     break
 
+                fsz += len(buf)
+                if fsz > maxsz:
+                    raise Exception("zipbomb defused")
+
                 fo.write(buf)
 
         return ret
 
     except Exception as ex:
         if ret:
-            t = "failed to decompress audio file [%s]: %r"
+            t = "failed to decompress audio file %r: %r"
             log(t % (abspath, ex))
             wunlink(log, ret, vn.flags if vn else VF_CAREFUL)
 
@@ -174,7 +208,7 @@ def au_unpk(
 
 def ffprobe(
     abspath: str, timeout: int = 60
-) -> tuple[dict[str, tuple[int, Any]], dict[str, list[Any]]]:
+) -> tuple[dict[str, tuple[int, Any]], dict[str, list[Any]], list[Any], dict[str, Any]]:
     cmd = [
         b"ffprobe",
         b"-hide_banner",
@@ -188,8 +222,17 @@ def ffprobe(
     return parse_ffprobe(so)
 
 
-def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[Any]]]:
-    """ffprobe -show_format -show_streams"""
+def parse_ffprobe(
+    txt: str,
+) -> tuple[dict[str, tuple[int, Any]], dict[str, list[Any]], list[Any], dict[str, Any]]:
+    """
+    txt: output from ffprobe -show_format -show_streams
+    returns:
+     * normalized tags
+     * original/raw tags
+     * list of streams
+     * format props
+    """
     streams = []
     fmt = {}
     g = {}
@@ -213,7 +256,7 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
     ret: dict[str, Any] = {}  # processed
     md: dict[str, list[Any]] = {}  # raw tags
 
-    is_audio = fmt.get("format_name") in ["mp3", "ogg", "flac", "wav"]
+    is_audio = fmt.get("format_name") in FMT_AU
     if fmt.get("filename", "").split(".")[-1].lower() in ["m4a", "aac"]:
         is_audio = True
 
@@ -241,6 +284,8 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
                 ["channel_layout", "chs"],
                 ["sample_rate", ".hz"],
                 ["bit_rate", ".aq"],
+                ["bits_per_sample", ".bps"],
+                ["bits_per_raw_sample", ".bprs"],
                 ["duration", ".dur"],
             ]
 
@@ -280,7 +325,7 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
                 ret[rk] = v1
 
     if ret.get("vc") == "ansi":  # shellscript
-        return {}, {}
+        return {}, {}, [], {}
 
     for strm in streams:
         for sk, sv in strm.items():
@@ -329,7 +374,77 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
     zero = int("0")
     zd = {k: (zero, v) for k, v in ret.items()}
 
-    return zd, md
+    return zd, md, streams, fmt
+
+
+def get_cover_from_epub(log: "NamedLogger", abspath: str) -> Optional[IO[bytes]]:
+    import zipfile
+
+    from .dxml import parse_xml
+
+    try:
+        from urlparse import urljoin  # Python2
+    except ImportError:
+        from urllib.parse import urljoin  # Python3
+
+    with zipfile.ZipFile(abspath, "r") as z:
+        # First open the container file to find the package document (.opf file)
+        try:
+            container_root = parse_xml(z.read("META-INF/container.xml").decode())
+        except KeyError:
+            log("epub: no container file found in %s" % (abspath,))
+            return None
+
+        # https://www.w3.org/TR/epub-33/#sec-container.xml-rootfile-elem
+        container_ns = {"": "urn:oasis:names:tc:opendocument:xmlns:container"}
+        # One file could contain multiple package documents, default to the first one
+        rootfile_path = container_root.find("./rootfiles/rootfile", container_ns).get(
+            "full-path"
+        )
+
+        # Then open the first package document to find the path of the cover image
+        try:
+            package_root = parse_xml(z.read(rootfile_path).decode())
+        except KeyError:
+            log("epub: no package document found in %s" % (abspath,))
+            return None
+
+        # https://www.w3.org/TR/epub-33/#sec-package-doc
+        package_ns = {"": "http://www.idpf.org/2007/opf"}
+        # https://www.w3.org/TR/epub-33/#sec-cover-image
+        coverimage_path_node = package_root.find(
+            "./manifest/item[@properties='cover-image']", package_ns
+        )
+        if coverimage_path_node is not None:
+            coverimage_path = coverimage_path_node.get("href")
+        else:
+            # This might be an EPUB2 file, try the legacy way of specifying covers
+            coverimage_path = _get_cover_from_epub2(log, package_root, package_ns)
+
+        # This url is either absolute (in the .epub) or relative to the package document
+        adjusted_cover_path = urljoin(rootfile_path, coverimage_path)
+
+        return z.open(adjusted_cover_path)
+
+
+def _get_cover_from_epub2(
+    log: "NamedLogger", package_root, package_ns
+) -> Optional[str]:
+    # <meta name="cover" content="id-to-cover-image"> in <metadata>, then
+    # <item> in <manifest>
+    cover_id = package_root.find("./metadata/meta[@name='cover']", package_ns).get(
+        "content"
+    )
+
+    if not cover_id:
+        return None
+
+    for node in package_root.iterfind("./manifest/item", package_ns):
+        if node.get("id") == cover_id:
+            cover_path = node.get("href")
+            return cover_path
+
+    return None
 
 
 class MTag(object):
@@ -473,7 +588,7 @@ class MTag(object):
                 sv = str(zv).split("/")[0].strip().lstrip("0")
                 ret[sk] = sv or 0
 
-        # normalize key notation to rkeobo
+        # normalize key notation to rekobo
         okey = ret.get("key")
         if okey:
             key = str(okey).replace(" ", "").replace("maj", "").replace("min", "m")
@@ -553,7 +668,7 @@ class MTag(object):
                 raise Exception()
         except Exception as ex:
             if self.args.mtag_v:
-                self.log("mutagen-err [{}] @ [{}]".format(ex, abspath), "90")
+                self.log("mutagen-err [%s] @ %r" % (ex, abspath), "90")
 
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
@@ -600,7 +715,7 @@ class MTag(object):
         if not bos.path.isfile(abspath):
             return {}
 
-        ret, md = ffprobe(abspath, self.args.mtag_to)
+        ret, md, _, _ = ffprobe(abspath, self.args.mtag_to)
 
         if self.args.mtag_vv:
             for zd in (ret, dict(md)):
@@ -670,8 +785,8 @@ class MTag(object):
                             ret[tag] = zj[tag]
             except:
                 if self.args.mtag_v:
-                    t = "mtag error: tagname {}, parser {}, file {} => {}"
-                    self.log(t.format(tagname, parser.bin, abspath, min_ex()))
+                    t = "mtag error: tagname %r, parser %r, file %r => %r"
+                    self.log(t % (tagname, parser.bin, abspath, min_ex()), 6)
 
         if ap != abspath:
             wunlink(self.log, ap, VF_CAREFUL)

copyparty/multicast.py

@@ -163,6 +163,7 @@ class MCast(object):
             sck.settimeout(None)
             sck.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
             try:
+                # safe for this purpose; https://lwn.net/Articles/853637/
                 sck.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
             except:
                 pass
@@ -182,11 +183,7 @@ class MCast(object):
                 srv.ips[oth_ip.split("/")[0]] = ipaddress.ip_network(oth_ip, False)
 
             # gvfs breaks if a linklocal ip appears in a dns reply
-            ll = {
-                k: v
-                for k, v in srv.ips.items()
-                if k.startswith("169.254") or k.startswith("fe80")
-            }
+            ll = {k: v for k, v in srv.ips.items() if k.startswith(("169.254", "fe80"))}
             rt = {k: v for k, v in srv.ips.items() if k not in ll}
 
             if self.args.ll or not rt:

copyparty/pwhash.py

@@ -15,7 +15,7 @@ try:
         raise Exception()
 
     HAVE_ARGON2 = True
-    from argon2 import __version__ as argon2ver
+    from argon2 import exceptions as argon2ex
 except:
     HAVE_ARGON2 = False
 
@@ -24,17 +24,13 @@ class PWHash(object):
     def __init__(self, args: argparse.Namespace):
         self.args = args
 
-        try:
-            alg, ac = args.ah_alg.split(",")
-        except:
-            alg = args.ah_alg
-            ac = {}
-
+        zsl = args.ah_alg.split(",")
+        alg = zsl[0]
         if alg == "none":
             alg = ""
 
         self.alg = alg
-        self.ac = ac
+        self.ac = zsl[1:]
         if not alg:
             self.on = False
             self.hash = unicode
@@ -90,17 +86,23 @@ class PWHash(object):
         its = 2
         blksz = 8
         para = 4
+        ramcap = 0  # openssl 1.1 = 32 MiB
         try:
             cost = 2 << int(self.ac[0])
             its = int(self.ac[1])
             blksz = int(self.ac[2])
             para = int(self.ac[3])
+            ramcap = int(self.ac[4]) * 1024 * 1024
         except:
             pass
 
+        cfg = {"salt": self.salt, "n": cost, "r": blksz, "p": para, "dklen": 24}
+        if ramcap:
+            cfg["maxmem"] = ramcap
+
         ret = plain.encode("utf-8")
         for _ in range(its):
-            ret = hashlib.scrypt(ret, salt=self.salt, n=cost, r=blksz, p=para, dklen=24)
+            ret = hashlib.scrypt(ret, **cfg)
 
         return "+" + base64.urlsafe_b64encode(ret).decode("utf-8")
 
@@ -145,6 +147,10 @@ class PWHash(object):
     def cli(self) -> None:
         import getpass
 
+        if self.args.usernames:
+            t = "since you have enabled --usernames, please provide username:password"
+            print(t)
+
         while True:
             try:
                 p1 = getpass.getpass("password> ")

copyparty/smbd.py

@@ -263,7 +263,7 @@ class SMB(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         ret = bos.open(ap, flags, *a, mode=chmod, **ka)
         if wr:
@@ -320,7 +320,7 @@ class SMB(object):
 
         self.hub.up2k.handle_mv(uname, "1.7.6.2", vp1, vp2)
         try:
-            bos.makedirs(ap2)
+            bos.makedirs(ap2, vf=vfs2.flags)
         except:
             pass
 
@@ -334,7 +334,7 @@ class SMB(object):
             t = "blocked mkdir (no-write-acc %s): /%s @%s"
             yeet(t % (vfs.axs.uwrite, vpath, uname))
 
-        return bos.mkdir(ap)
+        return bos.mkdir(ap, vfs.flags["chmod_d"])
 
     def _stat(self, vpath: str, *a: Any, **ka: Any) -> os.stat_result:
         try:

copyparty/ssdp.py

@@ -84,7 +84,7 @@ class SSDPr(object):
         name = self.args.doctitle
         zs = zs.strip().format(c(ubase), c(url), c(name), c(self.args.zsid))
         hc.reply(zs.encode("utf-8", "replace"))
-        return False  # close connectino
+        return False  # close connection
 
 
 class SSDPd(MCast):

copyparty/stolen/dnslib/dns.py

@@ -8,7 +8,7 @@ from itertools import chain
 from .bimap import Bimap, BimapError
 from .bit import get_bits, set_bits
 from .buffer import BufferError
-from .label import DNSBuffer, DNSLabel
+from .label import DNSBuffer, DNSLabel, set_avahi_379
 from .ranges import IP4, IP6, H, I, check_bytes
 
 
@@ -426,7 +426,7 @@ class RR(object):
             if rdlength:
                 rdata = RDMAP.get(QTYPE.get(rtype), RD).parse(buffer, rdlength)
             else:
-                rdata = ""
+                rdata = RD(b"a")
             return cls(rname, rtype, rclass, ttl, rdata)
         except (BufferError, BimapError) as e:
             raise DNSError("Error unpacking RR [offset=%d]: %s" % (buffer.offset, e))

copyparty/stolen/dnslib/label.py

@@ -11,6 +11,23 @@ LDH = set(range(33, 127))
 ESCAPE = re.compile(r"\\([0-9][0-9][0-9])")
 
 
+avahi_379 = 0
+
+
+def set_avahi_379():
+    global avahi_379
+    avahi_379 = 1
+
+
+def log_avahi_379(args):
+    global avahi_379
+    if avahi_379 == 2:
+        return
+    avahi_379 = 2
+    t = "Invalid pointer in DNSLabel [offset=%d,pointer=%d,length=%d];\n\033[35m  NOTE: this is probably avahi-bug #379, packet corruption in Avahi's mDNS-reflection feature. Copyparty has a workaround and is OK, but other devices need either --zm4 or --zm6"
+    raise BufferError(t % args)
+
+
 class DNSLabelError(Exception):
     pass
 
@@ -96,8 +113,11 @@ class DNSBuffer(Buffer):
                     )
                 if pointer < self.offset:
                     self.offset = pointer
+                elif avahi_379:
+                    log_avahi_379((self.offset, pointer, len(self.data)))
+                    label.extend(b"a")
+                    break
                 else:
-
                     raise BufferError(
                         "Invalid pointer in DNSLabel [offset=%d,pointer=%d,length=%d]"
                         % (self.offset, pointer, len(self.data))

copyparty/sutil.py

@@ -17,6 +17,9 @@ if True:  # pylint: disable=using-constant-test
     from .util import NamedLogger
 
 
+TAR_NO_OPUS = set("aac|m4a|mp3|oga|ogg|opus|wma".split("|"))
+
+
 class StreamArc(object):
     def __init__(
         self,
@@ -82,9 +85,7 @@ def enthumb(
 ) -> dict[str, Any]:
     rem = f["vp"]
     ext = rem.rsplit(".", 1)[-1].lower()
-    if (fmt == "mp3" and ext == "mp3") or (
-        fmt == "opus" and ext in "aac|m4a|mp3|ogg|opus|wma".split("|")
-    ):
+    if (fmt == "mp3" and ext == "mp3") or (fmt == "opus" and ext in TAR_NO_OPUS):
         raise Exception()
 
     vp = vjoin(vtop, rem.split("/", 1)[1])
@@ -110,7 +111,7 @@ def errdesc(
     report = ["copyparty failed to add the following files to the archive:", ""]
 
     for fn, err in errors:
-        report.extend([" file: {}".format(fn), "error: {}".format(err), ""])
+        report.extend([" file: %r" % (fn,), "error: %s" % (err,), ""])
 
     btxt = "\r\n".join(report).encode("utf-8", "replace")
     btxt = vol_san(list(vfs.all_vols.values()), btxt)

copyparty/svchub.py

@@ -2,8 +2,8 @@
 from __future__ import print_function, unicode_literals
 
 import argparse
+import atexit
 import errno
-import gzip
 import logging
 import os
 import re
@@ -28,6 +28,7 @@ if True:  # pylint: disable=using-constant-test
 
 from .__init__ import ANYWIN, EXE, MACOS, PY2, TYPE_CHECKING, E, EnvParams, unicode
 from .authsrv import BAD_CFG, AuthSrv
+from .bos import bos
 from .cert import ensure_cert
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, HAVE_MUTAGEN
 from .pwhash import HAVE_ARGON2
@@ -38,6 +39,7 @@ from .th_srv import (
     HAVE_FFPROBE,
     HAVE_HEIF,
     HAVE_PIL,
+    HAVE_RAW,
     HAVE_VIPS,
     HAVE_WEBP,
     ThumbSrv,
@@ -50,6 +52,9 @@ from .util import (
     FFMPEG_URL,
     HAVE_PSUTIL,
     HAVE_SQLITE3,
+    HAVE_ZMQ,
+    RE_ANSI,
+    URL_BUG,
     UTC,
     VERSIONS,
     Daemon,
@@ -58,17 +63,25 @@ from .util import (
     HMaccas,
     ODict,
     alltrace,
-    ansi_re,
     build_netmap,
+    expat_ver,
+    gzip,
+    load_ipr,
+    load_ipu,
+    lock_file,
     min_ex,
     mp,
     odfusion,
     pybin,
     start_log_thrs,
     start_stackmon,
+    termsize,
     ub64enc,
 )
 
+if HAVE_SQLITE3:
+    import sqlite3
+
 if TYPE_CHECKING:
     try:
         from .mdns import MDNS
@@ -80,6 +93,11 @@ if PY2:
     range = xrange  # type: ignore
 
 
+VER_IDP_DB = 1
+VER_SESSION_DB = 1
+VER_SHARES_DB = 2
+
+
 class SvcHub(object):
     """
     Hosts all services which cannot be parallelized due to reliance on monolithic resources.
@@ -111,7 +129,7 @@ class SvcHub(object):
         self.stopping = False
         self.stopped = False
         self.reload_req = False
-        self.reloading = 0
+        self.reload_mutex = threading.Lock()
         self.stop_cond = threading.Condition()
         self.nsigs = 3
         self.retcode = 0
@@ -138,6 +156,7 @@ class SvcHub(object):
             args.no_del = True
             args.no_mv = True
             args.hardlink = True
+            args.dav_auth = True
             args.vague_403 = True
             args.nih = True
 
@@ -154,6 +173,7 @@ class SvcHub(object):
         # for non-http clients (ftp, tftp)
         self.bans: dict[str, int] = {}
         self.gpwd = Garda(self.args.ban_pw)
+        self.gpwc = Garda(self.args.ban_pwc)
         self.g404 = Garda(self.args.ban_404)
         self.g403 = Garda(self.args.ban_403)
         self.g422 = Garda(self.args.ban_422, False)
@@ -182,8 +202,14 @@ class SvcHub(object):
 
         if not args.use_fpool and args.j != 1:
             args.no_fpool = True
-            t = "multithreading enabled with -j {}, so disabling fpool -- this can reduce upload performance on some filesystems"
-            self.log("root", t.format(args.j))
+            t = "multithreading enabled with -j {}, so disabling fpool -- this can reduce upload performance on some filesystems, and make some antivirus-softwares "
+            c = 0
+            if ANYWIN:
+                t += "(especially Microsoft Defender) stress your CPU and HDD severely during big uploads"
+                c = 3
+            else:
+                t += "consume more resources (CPU/HDD) than normal"
+            self.log("root", t.format(args.j), c)
 
         if not args.no_fpool and args.j != 1:
             t = "WARNING: ignoring --use-fpool because multithreading (-j{}) is enabled"
@@ -210,7 +236,16 @@ class SvcHub(object):
             t = "WARNING: --s-rd-sz (%d) is larger than --iobuf (%d); this may lead to reduced performance"
             self.log("root", t % (args.s_rd_sz, args.iobuf), 3)
 
-        if args.chpw and args.idp_h_usr:
+        zs = ""
+        if args.th_ram_max < 0.22:
+            zs = "generate thumbnails"
+        elif args.th_ram_max < 1:
+            zs = "generate audio waveforms or spectrograms"
+        if zs:
+            t = "WARNING: --th-ram-max is very small (%.2f GiB); will not be able to %s"
+            self.log("root", t % (args.th_ram_max, zs), 3)
+
+        if args.chpw and args.have_idp_hdrs:
             t = "ERROR: user-changeable passwords is incompatible with IdP/identity-providers; you must disable either --chpw or --idp-h-usr"
             self.log("root", t, 1)
             raise Exception(t)
@@ -221,9 +256,31 @@ class SvcHub(object):
             noch.update([x for x in zsl if x])
         args.chpw_no = noch
 
-        if not self.args.no_ses:
-            self.setup_session_db()
+        if args.ipu:
+            iu, nm = load_ipu(self.log, args.ipu, True)
+            setattr(args, "ipu_iu", iu)
+            setattr(args, "ipu_nm", nm)
 
+        if args.ipr:
+            ipr = load_ipr(self.log, args.ipr, True)
+            setattr(args, "ipr_u", ipr)
+
+        for zs in "ah_salt fk_salt dk_salt".split():
+            if getattr(args, "show_%s" % (zs,)):
+                self.log("root", "effective %s is %s" % (zs, getattr(args, zs)))
+
+        if args.ah_cli or args.ah_gen:
+            args.idp_store = 0
+            args.no_ses = True
+            args.shr = ""
+
+        if args.idp_store and args.have_idp_hdrs:
+            self.setup_db("idp")
+
+        if not self.args.no_ses:
+            self.setup_db("ses")
+
+        args.shr1 = ""
         if args.shr:
             self.setup_share_db()
 
@@ -273,6 +330,8 @@ class SvcHub(object):
             decs.pop("vips", None)
         if not HAVE_PIL:
             decs.pop("pil", None)
+        if not HAVE_RAW:
+            decs.pop("raw", None)
         if not HAVE_FFMPEG or not HAVE_FFPROBE:
             decs.pop("ff", None)
 
@@ -372,33 +431,102 @@ class SvcHub(object):
 
         self.broker = Broker(self)
 
-    def setup_session_db(self) -> None:
+        # create netmaps early to avoid firewall gaps,
+        # but the mutex blocks multiprocessing startup
+        for zs in "ipu_iu ftp_ipa_nm tftp_ipa_nm".split():
+            try:
+                getattr(args, zs).mutex = threading.Lock()
+            except:
+                pass
+        if args.ipr:
+            for nm in args.ipr_u.values():
+                nm.mutex = threading.Lock()
+
+    def _db_onfail_ses(self) -> None:
+        self.args.no_ses = True
+
+    def _db_onfail_idp(self) -> None:
+        self.args.idp_store = 0
+
+    def setup_db(self, which: str) -> None:
+        """
+        the "non-mission-critical" databases; if something looks broken then just nuke it
+        """
+        if which == "ses":
+            native_ver = VER_SESSION_DB
+            db_path = self.args.ses_db
+            desc = "sessions-db"
+            pathopt = "ses-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_session_db
+            failfun = self._db_onfail_ses
+        elif which == "idp":
+            native_ver = VER_IDP_DB
+            db_path = self.args.idp_db
+            desc = "idp-db"
+            pathopt = "idp-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_idp_db
+            failfun = self._db_onfail_idp
+        else:
+            raise Exception("unknown cachetype")
+
+        if not db_path.endswith(".db"):
+            zs = "config option --%s (the %s) was configured to [%s] which is invalid; must be a filepath ending with .db"
+            self.log("root", zs % (pathopt, desc, db_path), 1)
+            raise Exception(BAD_CFG)
+
         if not HAVE_SQLITE3:
-            self.args.no_ses = True
-            t = "WARNING: sqlite3 not available; disabling sessions, will use plaintext passwords in cookies"
-            self.log("root", t, 3)
+            failfun()
+            if which == "ses":
+                zs = "disabling sessions, will use plaintext passwords in cookies"
+            elif which == "idp":
+                zs = "disabling idp-db, will be unable to remember IdP-volumes after a restart"
+            self.log("root", "WARNING: sqlite3 not available; %s" % (zs,), 3)
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
-        create = True
-        db_path = self.args.ses_db
-        self.log("root", "opening sessions-db %s" % (db_path,))
-        for n in range(2):
+        db_lock = db_path + ".lock"
+        try:
+            create = not os.path.getsize(db_path)
+        except:
+            create = True
+        zs = "creating new" if create else "opening"
+        self.log("root", "%s %s %s" % (zs, desc, db_path))
+
+        for tries in range(2):
+            sver = 0
             try:
                 db = sqlite3.connect(db_path)
                 cur = db.cursor()
                 try:
-                    cur.execute("select count(*) from us").fetchone()
-                    create = False
-                    break
+                    zs = "select v from kv where k='sver'"
+                    sver = cur.execute(zs).fetchall()[0][0]
+                    if sver > native_ver:
+                        zs = "this version of copyparty only understands %s v%d and older; the db is v%d"
+                        raise Exception(zs % (desc, native_ver, sver))
+
+                    cur.execute(sanchk_q).fetchone()
                 except:
-                    pass
+                    if sver:
+                        raise
+                    sver = createfun(cur)
+
+                err = self._verify_db(
+                    cur, which, pathopt, db_path, desc, sver, native_ver
+                )
+                if err:
+                    tries = 99
+                    self.args.no_ses = True
+                    self.log("root", err, 3)
+                break
+
             except Exception as ex:
-                if n:
+                if tries or sver > native_ver:
                     raise
-                t = "sessions-db corrupt; deleting and recreating: %r"
-                self.log("root", t % (ex,), 3)
+                t = "%s is unusable; deleting and recreating: %r"
+                self.log("root", t % (desc, ex), 3)
                 try:
                     cur.close()  # type: ignore
                 except:
@@ -407,8 +535,13 @@ class SvcHub(object):
                     db.close()  # type: ignore
                 except:
                     pass
+                try:
+                    os.unlink(db_lock)
+                except:
+                    pass
                 os.unlink(db_path)
 
+    def _create_session_db(self, cur: "sqlite3.Cursor") -> int:
         sch = [
             r"create table kv (k text, v int)",
             r"create table us (un text, si text, t0 int)",
@@ -418,17 +551,74 @@ class SvcHub(object):
             r"create index us_t0 on us(t0)",
             r"insert into kv values ('sver', 1)",
         ]
+        for cmd in sch:
+            cur.execute(cmd)
+        self.log("root", "created new sessions-db")
+        return 1
 
-        assert db  # type: ignore  # !rm
-        assert cur  # type: ignore  # !rm
-        if create:
-            for cmd in sch:
-                cur.execute(cmd)
-            self.log("root", "created new sessions-db")
-            db.commit()
+    def _create_idp_db(self, cur: "sqlite3.Cursor") -> int:
+        sch = [
+            r"create table kv (k text, v int)",
+            r"create table us (un text, gs text)",
+            # username, groups
+            r"create index us_un on us(un)",
+            r"insert into kv values ('sver', 1)",
+        ]
+        for cmd in sch:
+            cur.execute(cmd)
+        self.log("root", "created new idp-db")
+        return 1
 
+    def _verify_db(
+        self,
+        cur: "sqlite3.Cursor",
+        which: str,
+        pathopt: str,
+        db_path: str,
+        desc: str,
+        sver: int,
+        native_ver: int,
+    ) -> str:
+        # ensure writable (maybe owned by other user)
+        db = cur.connection
+
+        try:
+            zil = cur.execute("select v from kv where k='pid'").fetchall()
+            if len(zil) > 1:
+                raise Exception()
+            owner = zil[0][0]
+        except:
+            owner = 0
+
+        if which == "ses":
+            cons = "Will now disable sessions and instead use plaintext passwords in cookies."
+        elif which == "idp":
+            cons = "Each IdP-volume will not become available until its associated user sends their first request."
+        else:
+            raise Exception()
+
+        if not lock_file(db_path + ".lock"):
+            t = "the %s [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --%s or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. %s"
+            return t % (desc, db_path, owner, pathopt, cons)
+
+        vars = (("pid", os.getpid()), ("ts", int(time.time() * 1000)))
+        if owner:
+            # wear-estimate: 2 cells; offsets 0x10, 0x50, 0x19720
+            for k, v in vars:
+                cur.execute("update kv set v=? where k=?", (v, k))
+        else:
+            # wear-estimate: 3~4 cells; offsets 0x10, 0x50, 0x19180, 0x19710, 0x36000, 0x360b0, 0x36b90
+            for k, v in vars:
+                cur.execute("insert into kv values(?, ?)", (k, v))
+
+        if sver < native_ver:
+            cur.execute("delete from kv where k='sver'")
+            cur.execute("insert into kv values('sver',?)", (native_ver,))
+
+        db.commit()
         cur.close()
         db.close()
+        return ""
 
     def setup_share_db(self) -> None:
         al = self.args
@@ -437,7 +627,7 @@ class SvcHub(object):
             al.shr = ""
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
         al.shr = al.shr.strip("/")
         if "/" in al.shr or not al.shr:
@@ -446,35 +636,50 @@ class SvcHub(object):
             raise Exception(t)
 
         al.shr = "/%s/" % (al.shr,)
+        al.shr1 = al.shr[1:]
+
+        # policy:
+        # the shares-db is important, so panic if something is wrong
 
-        create = True
-        modified = False
         db_path = self.args.shr_db
-        self.log("root", "opening shares-db %s" % (db_path,))
-        for n in range(2):
-            try:
-                db = sqlite3.connect(db_path)
-                cur = db.cursor()
-                try:
-                    cur.execute("select count(*) from sh").fetchone()
-                    create = False
-                    break
-                except:
-                    pass
-            except Exception as ex:
-                if n:
-                    raise
-                t = "shares-db corrupt; deleting and recreating: %r"
-                self.log("root", t % (ex,), 3)
-                try:
-                    cur.close()  # type: ignore
-                except:
-                    pass
-                try:
-                    db.close()  # type: ignore
-                except:
-                    pass
-                os.unlink(db_path)
+        db_lock = db_path + ".lock"
+        try:
+            create = not os.path.getsize(db_path)
+        except:
+            create = True
+        zs = "creating new" if create else "opening"
+        self.log("root", "%s shares-db %s" % (zs, db_path))
+
+        sver = 0
+        try:
+            db = sqlite3.connect(db_path)
+            cur = db.cursor()
+            if not create:
+                zs = "select v from kv where k='sver'"
+                sver = cur.execute(zs).fetchall()[0][0]
+                if sver > VER_SHARES_DB:
+                    zs = "this version of copyparty only understands shares-db v%d and older; the db is v%d"
+                    raise Exception(zs % (VER_SHARES_DB, sver))
+
+                cur.execute("select count(*) from sh").fetchone()
+        except Exception as ex:
+            t = "could not open shares-db; will now panic...\nthe following database must be repaired or deleted before you can launch copyparty:\n%s\n\nERROR: %s\n\nadditional details:\n%s\n"
+            self.log("root", t % (db_path, ex, min_ex()), 1)
+            raise
+
+        try:
+            zil = cur.execute("select v from kv where k='pid'").fetchall()
+            if len(zil) > 1:
+                raise Exception()
+            owner = zil[0][0]
+        except:
+            owner = 0
+
+        if not lock_file(db_lock):
+            t = "the shares-db [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --shr-db or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. Will now panic."
+            t = t % (db_path, owner)
+            self.log("root", t, 1)
+            raise Exception(t)
 
         sch1 = [
             r"create table kv (k text, v int)",
@@ -486,34 +691,37 @@ class SvcHub(object):
             r"create index sf_k on sf(k)",
             r"create index sh_k on sh(k)",
             r"create index sh_t1 on sh(t1)",
+            r"insert into kv values ('sver', 2)",
         ]
 
         assert db  # type: ignore  # !rm
         assert cur  # type: ignore  # !rm
-        if create:
-            dver = 2
-            modified = True
+        if not sver:
+            sver = VER_SHARES_DB
             for cmd in sch1 + sch2:
                 cur.execute(cmd)
             self.log("root", "created new shares-db")
-        else:
-            (dver,) = cur.execute("select v from kv where k = 'sver'").fetchall()[0]
 
-        if dver == 1:
-            modified = True
+        if sver == 1:
             for cmd in sch2:
                 cur.execute(cmd)
             cur.execute("update sh set st = 0")
             self.log("root", "shares-db schema upgrade ok")
 
-        if modified:
-            for cmd in [
-                r"delete from kv where k = 'sver'",
-                r"insert into kv values ('sver', %d)" % (2,),
-            ]:
-                cur.execute(cmd)
-            db.commit()
+        if sver < VER_SHARES_DB:
+            cur.execute("delete from kv where k='sver'")
+            cur.execute("insert into kv values('sver',?)", (VER_SHARES_DB,))
 
+        vars = (("pid", os.getpid()), ("ts", int(time.time() * 1000)))
+        if owner:
+            # wear-estimate: same as sessions-db
+            for k, v in vars:
+                cur.execute("update kv set v=? where k=?", (v, k))
+        else:
+            for k, v in vars:
+                cur.execute("insert into kv values(?, ?)", (k, v))
+
+        db.commit()
         cur.close()
         db.close()
 
@@ -578,6 +786,39 @@ class SvcHub(object):
     def sigterm(self) -> None:
         self.signal_handler(signal.SIGTERM, None)
 
+    def sticky_qr(self) -> None:
+        tw, th = termsize()
+        zs1, qr = self.tcpsrv.qr.split("\n", 1)
+        url, colr = zs1.split(" ", 1)
+        nl = len(qr.split("\n"))  # numlines
+        lp = 3 if nl * 2 + 4 < tw else 0  # leftpad
+        lp0 = lp
+        if self.args.qr_pin == 2:
+            url = ""
+        else:
+            while lp and (nl + lp) * 2 + len(url) + 1 > tw:
+                lp -= 1
+            if (nl + lp) * 2 + len(url) + 1 > tw:
+                qr = url + "\n" + qr
+                url = ""
+                nl += 1
+                lp = lp0
+        sh = 1 + th - nl
+        if lp:
+            zs = " " * lp
+            qr = zs + qr.replace("\n", "\n" + zs)
+        if url:
+            url = "%s\033[%d;%dH%s\033[0m" % (colr, sh + 1, (nl + lp) * 2, url)
+        qr = colr + qr
+
+        def unlock():
+            print("\033[s\033[r\033[u", file=sys.stderr)
+
+        atexit.register(unlock)
+        t = "%s\033[%dA" % ("\n" * nl, nl)
+        t = "%s\033[s\033[1;%dr\033[%dH%s%s\033[u" % (t, sh - 1, sh, qr, url)
+        self.pr(t, file=sys.stderr)
+
     def cb_httpsrv_up(self) -> None:
         self.httpsrv_up += 1
         if self.httpsrv_up != self.broker.num_workers:
@@ -590,7 +831,10 @@ class SvcHub(object):
                 break
 
         if self.tcpsrv.qr:
-            self.log("qr-code", self.tcpsrv.qr)
+            if self.args.qr_pin:
+                self.sticky_qr()
+            else:
+                self.log("qr-code", self.tcpsrv.qr)
         else:
             self.log("root", "workers OK\n")
 
@@ -614,8 +858,10 @@ class SvcHub(object):
             (HAVE_FFPROBE, "ffprobe", t_ff + ", read audio/media tags"),
             (HAVE_MUTAGEN, "mutagen", "read audio tags (ffprobe is better but slower)"),
             (HAVE_ARGON2, "argon2", "secure password hashing (advanced users only)"),
+            (HAVE_ZMQ, "pyzmq", "send zeromq messages from event-hooks"),
             (HAVE_HEIF, "pillow-heif", "read .heif images with pillow (rarely useful)"),
             (HAVE_AVIF, "pillow-avif", "read .avif images with pillow (rarely useful)"),
+            (HAVE_RAW, "rawpy", "read RAW images"),
         ]
         if ANYWIN:
             to_check += [
@@ -650,19 +896,11 @@ class SvcHub(object):
                 t += ", "
             t += "\033[0mNG: \033[35m" + sng
 
-        t += "\033[0m, see --deps"
-        self.log("dependencies", t, 6)
+        t += "\033[0m, see --deps (this is fine btw)"
+        self.log("optional-dependencies", t, 6)
 
     def _check_env(self) -> None:
-        try:
-            files = os.listdir(E.cfg)
-        except:
-            files = []
-
-        hits = [x for x in files if x.lower().endswith(".conf")]
-        if hits:
-            t = "WARNING: found config files in [%s]: %s\n  config files are not expected here, and will NOT be loaded (unless your setup is intentionally hella funky)"
-            self.log("root", t % (E.cfg, ", ".join(hits)), 3)
+        al = self.args
 
         if self.args.no_bauth:
             t = "WARNING: --no-bauth disables support for the Android app; you may want to use --bauth-last instead"
@@ -670,6 +908,30 @@ class SvcHub(object):
             if self.args.bauth_last:
                 self.log("root", "WARNING: ignoring --bauth-last due to --no-bauth", 3)
 
+        have_tcp = False
+        for zs in al.i:
+            if not zs.startswith(("unix:", "fd:")):
+                have_tcp = True
+        if not have_tcp:
+            zb = False
+            zs = "z zm zm4 zm6 zmv zmvv zs zsv zv"
+            for zs in zs.split():
+                if getattr(al, zs, False):
+                    setattr(al, zs, False)
+                    zb = True
+            if zb:
+                t = "not listening on any ip-addresses (only unix-sockets and/or FDs); cannot enable zeroconf/mdns/ssdp as requested"
+                self.log("root", t, 3)
+
+        if not self.args.no_dav:
+            from .dxml import DXML_OK
+
+            if not DXML_OK:
+                if not self.args.no_dav:
+                    self.args.no_dav = True
+                    t = "WARNING:\nDisabling WebDAV support because dxml selftest failed. Please report this bug;\n%s\n...and include the following information in the bug-report:\n%s | expat %s\n"
+                    self.log("root", t % (URL_BUG, VERSIONS, expat_ver()), 1)
+
     def _process_config(self) -> bool:
         al = self.args
 
@@ -725,13 +987,20 @@ class SvcHub(object):
             vl = [os.path.expandvars(os.path.expanduser(x)) for x in vl]
             setattr(al, k, vl)
 
-        for k in "lo hist ssl_log".split(" "):
+        for k in "lo hist dbpath ssl_log".split(" "):
             vs = getattr(al, k)
             if vs:
                 vs = os.path.expandvars(os.path.expanduser(vs))
                 setattr(al, k, vs)
 
-        for k in "sus_urls nonsus_urls".split(" "):
+        for k in "idp_adm".split(" "):
+            vs = getattr(al, k)
+            vsa = [x.strip() for x in vs.split(",")]
+            vsa = [x.lower() for x in vsa if x]
+            setattr(al, k + "_set", set(vsa))
+
+        zs = "dav_ua1 sus_urls nonsus_urls ua_nodoc ua_nozip"
+        for k in zs.split(" "):
             vs = getattr(al, k)
             if not vs or vs == "no":
                 setattr(al, k, None)
@@ -751,12 +1020,25 @@ class SvcHub(object):
             al.sus_urls = None
 
         al.xff_hdr = al.xff_hdr.lower()
-        al.idp_h_usr = al.idp_h_usr.lower()
+        al.idp_h_usr = [x.lower() for x in al.idp_h_usr or []]
         al.idp_h_grp = al.idp_h_grp.lower()
         al.idp_h_key = al.idp_h_key.lower()
 
-        al.ftp_ipa_nm = build_netmap(al.ftp_ipa or al.ipa)
-        al.tftp_ipa_nm = build_netmap(al.tftp_ipa or al.ipa)
+        al.idp_hm_usr_p = {}
+        for zs0 in al.idp_hm_usr or []:
+            try:
+                sep = zs0[:1]
+                hn, zs1, zs2 = zs0[1:].split(sep)
+                hn = hn.lower()
+                if hn in al.idp_hm_usr_p:
+                    al.idp_hm_usr_p[hn][zs1] = zs2
+                else:
+                    al.idp_hm_usr_p[hn] = {zs1: zs2}
+            except:
+                raise Exception("invalid --idp-hm-usr [%s]" % (zs0,))
+
+        al.ftp_ipa_nm = build_netmap(al.ftp_ipa or al.ipa, True)
+        al.tftp_ipa_nm = build_netmap(al.tftp_ipa or al.ipa, True)
 
         mte = ODict.fromkeys(DEF_MTE.split(","), True)
         al.mte = odfusion(mte, al.mte)
@@ -768,7 +1050,7 @@ class SvcHub(object):
         al.exp_md = odfusion(exp, al.exp_md.replace(" ", ","))
         al.exp_lg = odfusion(exp, al.exp_lg.replace(" ", ","))
 
-        for k in ["no_hash", "no_idx", "og_ua"]:
+        for k in ["no_hash", "no_idx", "og_ua", "srch_excl"]:
             ptn = getattr(self.args, k)
             if ptn:
                 setattr(self.args, k, re.compile(ptn))
@@ -799,10 +1081,30 @@ class SvcHub(object):
         except:
             raise Exception("invalid --mv-retry [%s]" % (self.args.mv_retry,))
 
+        al.js_utc = "false" if al.localtime else "true"
+
         al.tcolor = al.tcolor.lstrip("#")
         if len(al.tcolor) == 3:  # fc5 => ffcc55
             al.tcolor = "".join([x * 2 for x in al.tcolor])
 
+        zs = al.u2sz
+        zsl = zs.split(",")
+        if len(zsl) not in (1, 3):
+            t = "invalid --u2sz; must be either one number, or a comma-separated list of three numbers (min,default,max)"
+            raise Exception(t)
+        if len(zsl) < 3:
+            zsl = ["1", zs, zs]
+        zi2 = 1
+        for zs in zsl:
+            zi = int(zs)
+            # arbitrary constraint (anything above 2 GiB is probably unintended)
+            if zi < 1 or zi > 2047:
+                raise Exception("invalid --u2sz; minimum is 1, max is 2047")
+            if zi < zi2:
+                raise Exception("invalid --u2sz; values must be equal or ascending")
+            zi2 = zi
+        al.u2sz = ",".join(zsl)
+
         return True
 
     def _ipa2re(self, txt) -> Optional[re.Pattern]:
@@ -875,7 +1177,7 @@ class SvcHub(object):
 
         fn = sel_fn
         try:
-            os.makedirs(os.path.dirname(fn))
+            bos.makedirs(os.path.dirname(fn))
         except:
             pass
 
@@ -892,6 +1194,9 @@ class SvcHub(object):
 
             lh = codecs.open(fn, "w", encoding="utf-8", errors="replace")
 
+        if getattr(self.args, "free_umask", False):
+            os.fchmod(lh.fileno(), 0o644)
+
         argv = [pybin] + self.argv
         if hasattr(shlex, "quote"):
             argv = [shlex.quote(x) for x in argv]
@@ -970,41 +1275,18 @@ class SvcHub(object):
             except:
                 self.log("root", "ssdp startup failed;\n" + min_ex(), 3)
 
-    def reload(self) -> str:
-        with self.up2k.mutex:
-            if self.reloading:
-                return "cannot reload; already in progress"
-            self.reloading = 1
-
-        Daemon(self._reload, "reloading")
-        return "reload initiated"
-
-    def _reload(self, rescan_all_vols: bool = True, up2k: bool = True) -> None:
-        with self.up2k.mutex:
-            if self.reloading != 1:
-                return
-            self.reloading = 2
+    def reload(self, rescan_all_vols: bool, up2k: bool) -> str:
+        t = "config has been reloaded"
+        with self.reload_mutex:
             self.log("root", "reloading config")
             self.asrv.reload(9 if up2k else 4)
             if up2k:
                 self.up2k.reload(rescan_all_vols)
+                t += "; volumes are now reinitializing"
             else:
                 self.log("root", "reload done")
             self.broker.reload()
-            self.reloading = 0
-
-    def _reload_blocking(self, rescan_all_vols: bool = True, up2k: bool = True) -> None:
-        while True:
-            with self.up2k.mutex:
-                if self.reloading < 2:
-                    self.reloading = 1
-                    break
-            time.sleep(0.05)
-
-        # try to handle multiple pending IdP reloads at once:
-        time.sleep(0.2)
-
-        self._reload(rescan_all_vols=rescan_all_vols, up2k=up2k)
+        return t
 
     def _reload_sessions(self) -> None:
         with self.asrv.mutex:
@@ -1018,7 +1300,7 @@ class SvcHub(object):
 
             if self.reload_req:
                 self.reload_req = False
-                self.reload()
+                self.reload(True, True)
 
         self.shutdown()
 
@@ -1182,11 +1464,18 @@ class SvcHub(object):
 
             fmt = "\033[36m%s \033[33m%-21s \033[0m%s\n"
             if self.no_ansi:
-                fmt = "%s %-21s %s\n"
+                if c == 1:
+                    fmt = "%s %-21s CRIT: %s\n"
+                elif c == 3:
+                    fmt = "%s %-21s WARN: %s\n"
+                elif c == 6:
+                    fmt = "%s %-21s  BTW: %s\n"
+                else:
+                    fmt = "%s %-21s  LOG: %s\n"
                 if "\033" in msg:
-                    msg = ansi_re.sub("", msg)
+                    msg = RE_ANSI.sub("", msg)
                 if "\033" in src:
-                    src = ansi_re.sub("", src)
+                    src = RE_ANSI.sub("", src)
             elif c:
                 if isinstance(c, int):
                     msg = "\033[3%sm%s\033[0m" % (c, msg)
@@ -1227,7 +1516,7 @@ class SvcHub(object):
                 raise
 
     def check_mp_support(self) -> str:
-        if MACOS:
+        if MACOS and not os.environ.get("PRTY_FORCE_MP"):
             return "multiprocessing is wonky on mac osx;"
         elif sys.version_info < (3, 3):
             return "need python 3.3 or newer for multiprocessing;"
@@ -1247,7 +1536,7 @@ class SvcHub(object):
             return False
 
         try:
-            if mp.cpu_count() <= 1:
+            if mp.cpu_count() <= 1 and not os.environ.get("PRTY_FORCE_MP"):
                 raise Exception()
         except:
             self.log("svchub", "only one CPU detected; multiprocessing disabled")

copyparty/szip.py

@@ -4,12 +4,11 @@ from __future__ import print_function, unicode_literals
 import calendar
 import stat
 import time
-import zlib
 
 from .authsrv import AuthSrv
 from .bos import bos
 from .sutil import StreamArc, errdesc
-from .util import min_ex, sanitize_fn, spack, sunpack, yieldfile
+from .util import min_ex, sanitize_fn, spack, sunpack, yieldfile, zlib
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Generator, Optional
@@ -55,6 +54,7 @@ def gen_fdesc(sz: int, crc32: int, z64: bool) -> bytes:
 
 def gen_hdr(
     h_pos: Optional[int],
+    z64: bool,
     fn: str,
     sz: int,
     lastmod: int,
@@ -71,7 +71,6 @@ def gen_hdr(
     # appnote 4.5 / zip 3.0 (2008) / unzip 6.0 (2009) says to add z64
     # extinfo for values which exceed H, but that becomes an off-by-one
     # (can't tell if it was clamped or exactly maxval), make it obvious
-    z64 = sz >= 0xFFFFFFFF
     z64v = [sz, sz] if z64 else []
     if h_pos and h_pos >= 0xFFFFFFFF:
         # central, also consider ptr to original header
@@ -100,7 +99,7 @@ def gen_hdr(
 
     # spec says to put zeros when !crc if bit3 (streaming)
     # however infozip does actual sz and it even works on winxp
-    # (same reasning for z64 extradata later)
+    # (same reasoning for z64 extradata later)
     vsz = 0xFFFFFFFF if z64 else sz
     ret += spack(b"<LL", vsz, vsz)
 
@@ -245,6 +244,7 @@ class StreamZip(StreamArc):
 
         sz = st.st_size
         ts = st.st_mtime
+        h_pos = self.pos
 
         crc = 0
         if self.pre_crc:
@@ -253,8 +253,12 @@ class StreamZip(StreamArc):
 
             crc &= 0xFFFFFFFF
 
-        h_pos = self.pos
-        buf = gen_hdr(None, name, sz, ts, self.utf8, crc, self.pre_crc)
+        # some unzip-programs expect a 64bit data-descriptor
+        # even if the only 32bit-exceeding value is the offset,
+        # so force that by placeholdering the filesize too
+        z64 = h_pos >= 0xFFFFFFFF or sz >= 0xFFFFFFFF
+
+        buf = gen_hdr(None, z64, name, sz, ts, self.utf8, crc, self.pre_crc)
         yield self._ct(buf)
 
         for buf in yieldfile(src, self.args.iobuf):
@@ -267,8 +271,6 @@ class StreamZip(StreamArc):
 
         self.items.append((name, sz, ts, crc, h_pos))
 
-        z64 = sz >= 4 * 1024 * 1024 * 1024
-
         if z64 or not self.pre_crc:
             buf = gen_fdesc(sz, crc, z64)
             yield self._ct(buf)
@@ -307,7 +309,8 @@ class StreamZip(StreamArc):
 
             cdir_pos = self.pos
             for name, sz, ts, crc, h_pos in self.items:
-                buf = gen_hdr(h_pos, name, sz, ts, self.utf8, crc, self.pre_crc)
+                z64 = h_pos >= 0xFFFFFFFF or sz >= 0xFFFFFFFF
+                buf = gen_hdr(h_pos, z64, name, sz, ts, self.utf8, crc, self.pre_crc)
                 mbuf += self._ct(buf)
                 if len(mbuf) >= 16384:
                     yield mbuf

copyparty/tcpsrv.py

@@ -25,8 +25,8 @@ from .util import (
     termsize,
 )
 
-if True:
-    from typing import Generator, Union
+if True:  # pylint: disable=using-constant-test
+    from typing import Generator, Optional, Union
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -95,7 +95,7 @@ class TcpSrv(object):
                                 continue
 
                             # binding 0.0.0.0 after :: fails on dualstack
-                            # but is necessary on non-dualstakc
+                            # but is necessary on non-dualstack
                             if successful_binds:
                                 continue
 
@@ -151,9 +151,15 @@ class TcpSrv(object):
             if just_ll or self.args.ll:
                 ll_ok.add(ip.split("/")[0])
 
+        listening_on = []
+        for ip, ports in sorted(ok.items()):
+            for port in sorted(ports):
+                listening_on.append("%s %s" % (ip, port))
+
         qr1: dict[str, list[int]] = {}
         qr2: dict[str, list[int]] = {}
         msgs = []
+        accessible_on = []
         title_tab: dict[str, dict[str, int]] = {}
         title_vars = [x[1:] for x in self.args.wintitle.split(" ") if x.startswith("$")]
         t = "available @ {}://{}:{}/  (\033[33m{}\033[0m)"
@@ -169,6 +175,10 @@ class TcpSrv(object):
                 ):
                     continue
 
+                zs = "%s %s" % (ip, port)
+                if zs not in accessible_on:
+                    accessible_on.append(zs)
+
                 proto = " http"
                 if self.args.http_only:
                     pass
@@ -219,6 +229,14 @@ class TcpSrv(object):
         else:
             print("\n", end="")
 
+        for fn, ls in (
+            (self.args.wr_h_eps, listening_on),
+            (self.args.wr_h_aon, accessible_on),
+        ):
+            if fn:
+                with open(fn, "wb") as f:
+                    f.write(("\n".join(ls)).encode("utf-8"))
+
         if self.args.qr or self.args.qrs:
             self.qr = self._qr(qr1, qr2)
 
@@ -227,8 +245,10 @@ class TcpSrv(object):
 
     def _listen(self, ip: str, port: int) -> None:
         uds_perm = uds_gid = -1
+        bound: Optional[socket.socket] = None
+        tcp = False
+
         if "unix:" in ip:
-            tcp = False
             ipv = socket.AF_UNIX
             uds = ip.split(":")
             ip = uds[-1]
@@ -241,7 +261,12 @@ class TcpSrv(object):
                     import grp
 
                     uds_gid = grp.getgrnam(uds[2]).gr_gid
+        elif "fd:" in ip:
+            fd = ip[3:]
+            bound = socket.socket(fileno=int(fd))
 
+            tcp = bound.proto == socket.IPPROTO_TCP
+            ipv = bound.family
         elif ":" in ip:
             tcp = True
             ipv = socket.AF_INET6
@@ -249,7 +274,7 @@ class TcpSrv(object):
             tcp = True
             ipv = socket.AF_INET
 
-        srv = socket.socket(ipv, socket.SOCK_STREAM)
+        srv = bound or socket.socket(ipv, socket.SOCK_STREAM)
 
         if not ANYWIN or self.args.reuseaddr:
             srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
@@ -264,9 +289,13 @@ class TcpSrv(object):
         except:
             pass  # will create another ipv4 socket instead
 
-        if not ANYWIN and self.args.freebind:
+        if getattr(self.args, "freebind", False):
             srv.setsockopt(socket.SOL_IP, socket.IP_FREEBIND, 1)
 
+        if bound:
+            self.srv.append(srv)
+            return
+
         try:
             if tcp:
                 srv.bind((ip, port))
@@ -371,7 +400,7 @@ class TcpSrv(object):
             if self.args.q:
                 print(msg)
 
-            self.hub.broker.say("listen", srv)
+            self.hub.broker.say("httpsrv.listen", srv)
 
         self.srv = srvs
         self.bound = bound
@@ -379,7 +408,7 @@ class TcpSrv(object):
         self._distribute_netdevs()
 
     def _distribute_netdevs(self):
-        self.hub.broker.say("set_netdevs", self.netdevs)
+        self.hub.broker.say("httpsrv.set_netdevs", self.netdevs)
         self.hub.start_zeroconf()
         gencert(self.log, self.args, self.netdevs)
         self.hub.restart_ftpd()
@@ -402,24 +431,24 @@ class TcpSrv(object):
             if not netdevs:
                 continue
 
-            added = "nothing"
-            removed = "nothing"
+            add = []
+            rem = []
             for k, v in netdevs.items():
                 if k not in self.netdevs:
-                    added = "{} = {}".format(k, v)
+                    add.append("\n\033[32m  added %s = %s" % (k, v))
             for k, v in self.netdevs.items():
                 if k not in netdevs:
-                    removed = "{} = {}".format(k, v)
+                    rem.append("\n\033[33mremoved %s = %s" % (k, v))
 
-            t = "network change detected:\n  added {}\033[0;33m\nremoved {}"
-            self.log("tcpsrv", t.format(added, removed), 3)
+            t = "network change detected:%s%s"
+            self.log("tcpsrv", t % ("".join(add), "".join(rem)), 3)
             self.netdevs = netdevs
             self._distribute_netdevs()
 
     def detect_interfaces(self, listen_ips: list[str]) -> dict[str, Netdev]:
         from .stolen.ifaddr import get_adapters
 
-        listen_ips = [x for x in listen_ips if "unix:" not in x]
+        listen_ips = [x for x in listen_ips if not x.startswith(("unix:", "fd:"))]
 
         nics = get_adapters(True)
         eps: dict[str, Netdev] = {}
@@ -548,7 +577,7 @@ class TcpSrv(object):
         ip = None
         ips = list(t1) + list(t2)
         qri = self.args.qri
-        if self.args.zm and not qri:
+        if self.args.zm and not qri and ips:
             name = self.args.name + ".local"
             t1[name] = next(v for v in (t1 or t2).values())
             ips = [name] + ips
@@ -565,8 +594,7 @@ class TcpSrv(object):
         if not ip:
             return ""
 
-        if ":" in ip:
-            ip = "[{}]".format(ip)
+        hip = "[%s]" % (ip,) if ":" in ip else ip
 
         if self.args.http_only:
             https = ""
@@ -578,7 +606,7 @@ class TcpSrv(object):
         ports = t1.get(ip, t2.get(ip, []))
         dport = 443 if https else 80
         port = "" if dport in ports or not ports else ":{}".format(ports[0])
-        txt = "http{}://{}{}/{}".format(https, ip, port, self.args.qrl)
+        txt = "http{}://{}{}/{}".format(https, hip, port, self.args.qrl)
 
         btxt = txt.encode("utf-8")
         if PY2:
@@ -586,6 +614,10 @@ class TcpSrv(object):
 
         fg = self.args.qr_fg
         bg = self.args.qr_bg
+        nocolor = fg == -1
+        if nocolor:
+            fg = 0
+
         pad = self.args.qrp
         zoom = self.args.qrz
         qrc = QrCode.encode_binary(btxt)
@@ -613,6 +645,8 @@ class TcpSrv(object):
 
         qr = qr.replace("\n", "\033[K\n") + "\033[K"  # win10do
         cc = " \033[0;38;5;{0};47;48;5;{1}m" if fg else " \033[0;30;47m"
+        if nocolor:
+            cc = " \033[0m"
         t = cc + "\n{2}\033[999G\033[0m\033[J"
         t = t.format(fg, bg, qr)
         if ANYWIN:

copyparty/tftpd.py

@@ -36,7 +36,20 @@ from partftpy.TftpShared import TftpException
 from .__init__ import EXE, PY2, TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .util import UTC, BytesIO, Daemon, ODict, exclude_dotfiles, min_ex, runhook, undot
+from .util import (
+    FN_EMB,
+    UTC,
+    BytesIO,
+    Daemon,
+    ODict,
+    exclude_dotfiles,
+    min_ex,
+    runhook,
+    set_fperms,
+    undot,
+    vjoin,
+    vsplit,
+)
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Union
@@ -166,7 +179,7 @@ class Tftpd(object):
         if "::" in ips:
             ips.append("0.0.0.0")
 
-        ips = [x for x in ips if "unix:" not in x]
+        ips = [x for x in ips if not x.startswith(("unix:", "fd:"))]
 
         if self.args.tftp4:
             ips = [x for x in ips if ":" not in x]
@@ -244,16 +257,25 @@ class Tftpd(object):
         for srv in srvs:
             srv.stop()
 
-    def _v2a(self, caller: str, vpath: str, perms: list, *a: Any) -> tuple[VFS, str]:
+    def _v2a(
+        self, caller: str, vpath: str, perms: list, *a: Any
+    ) -> tuple[VFS, str, str]:
         vpath = vpath.replace("\\", "/").lstrip("/")
         if not perms:
             perms = [True, True]
 
         debug('%s("%s", %s) %s\033[K\033[0m', caller, vpath, str(a), perms)
         vfs, rem = self.asrv.vfs.get(vpath, "*", *perms)
+        if perms[1] and "*" not in vfs.axs.uread and "wo_up_readme" not in vfs.flags:
+            zs, fn = vsplit(vpath)
+            if fn.lower() in FN_EMB:
+                vpath = vjoin(zs, "_wo_" + fn)
+                vfs, rem = self.asrv.vfs.get(vpath, "*", *perms)
+
         if not vfs.realpath:
             raise Exception("unmapped vfs")
-        return vfs, vfs.canonical(rem)
+
+        return vfs, vpath, vfs.canonical(rem)
 
     def _ls(self, vpath: str, raddress: str, rport: int, force=False) -> Any:
         # generate file listing if vpath is dir.txt and return as file object
@@ -263,18 +285,20 @@ class Tftpd(object):
             if not ptn or not ptn.match(fn.lower()):
                 return None
 
+        tsdt = datetime.fromtimestamp
         vn, rem = self.asrv.vfs.get(vpath, "*", True, False)
         fsroot, vfs_ls, vfs_virt = vn.ls(
             rem,
             "*",
             not self.args.no_scandir,
             [[True, False]],
+            throw=True,
         )
         dnames = set([x[0] for x in vfs_ls if stat.S_ISDIR(x[1].st_mode)])
         dirs1 = [(v.st_mtime, v.st_size, k + "/") for k, v in vfs_ls if k in dnames]
         fils1 = [(v.st_mtime, v.st_size, k) for k, v in vfs_ls if k not in dnames]
         real1 = dirs1 + fils1
-        realt = [(datetime.fromtimestamp(mt, UTC), sz, fn) for mt, sz, fn in real1]
+        realt = [(tsdt(max(0, mt), UTC), sz, fn) for mt, sz, fn in real1]
         reals = [
             (
                 "%04d-%02d-%02d %02d:%02d:%02d"
@@ -330,7 +354,7 @@ class Tftpd(object):
         else:
             raise Exception("bad mode %s" % (mode,))
 
-        vfs, ap = self._v2a("open", vpath, [rd, wr])
+        vfs, vpath, ap = self._v2a("open", vpath, [rd, wr])
         if wr:
             if "*" not in vfs.axs.uwrite:
                 yeet("blocked write; folder not world-writable: /%s" % (vpath,))
@@ -356,7 +380,7 @@ class Tftpd(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         if not self.args.tftp_nols and bos.path.isdir(ap):
             return self._ls(vpath, "", 0, True)
@@ -364,18 +388,24 @@ class Tftpd(object):
         if not a:
             a = (self.args.iobuf,)
 
-        return open(ap, mode, *a, **ka)
+        ret = open(ap, mode, *a, **ka)
+        if wr and "fperms" in vfs.flags:
+            set_fperms(ret, vfs.flags)
+
+        return ret
 
     def _mkdir(self, vpath: str, *a) -> None:
-        vfs, ap = self._v2a("mkdir", vpath, [])
+        vfs, _, ap = self._v2a("mkdir", vpath, [False, True])
         if "*" not in vfs.axs.uwrite:
             yeet("blocked mkdir; folder not world-writable: /%s" % (vpath,))
 
-        return bos.mkdir(ap)
+        bos.mkdir(ap, vfs.flags["chmod_d"])
+        if "chown" in vfs.flags:
+            bos.chown(ap, vfs.flags["uid"], vfs.flags["gid"])
 
     def _unlink(self, vpath: str) -> None:
         # return bos.unlink(self._v2a("stat", vpath, *a)[1])
-        vfs, ap = self._v2a("delete", vpath, [True, False, False, True])
+        vfs, _, ap = self._v2a("delete", vpath, [True, False, False, True])
 
         try:
             inf = bos.stat(ap)
@@ -399,7 +429,7 @@ class Tftpd(object):
 
     def _p_exists(self, vpath: str) -> bool:
         try:
-            ap = self._v2a("p.exists", vpath, [False, False])[1]
+            ap = self._v2a("p.exists", vpath, [False, False])[2]
             bos.stat(ap)
             return True
         except:
@@ -407,7 +437,7 @@ class Tftpd(object):
 
     def _p_isdir(self, vpath: str) -> bool:
         try:
-            st = bos.stat(self._v2a("p.isdir", vpath, [False, False])[1])
+            st = bos.stat(self._v2a("p.isdir", vpath, [False, False])[2])
             ret = stat.S_ISDIR(st.st_mode)
             return ret
         except:

copyparty/th_cli.py

@@ -1,13 +1,15 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import errno
 import os
+import stat
 
 from .__init__ import TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .th_srv import HAVE_WEBP, thumb_path
-from .util import Cooldown
+from .th_srv import EXTS_AC, HAVE_WEBP, thumb_path
+from .util import Cooldown, Pebkac
 
 if True:  # pylint: disable=using-constant-test
     from typing import Optional, Union
@@ -16,6 +18,9 @@ if TYPE_CHECKING:
     from .httpsrv import HttpSrv
 
 
+IOERROR = "reading the file was denied by the server os; either due to filesystem permissions, selinux, apparmor, or similar:\n%r"
+
+
 class ThumbCli(object):
     def __init__(self, hsrv: "HttpSrv") -> None:
         self.broker = hsrv.broker
@@ -31,11 +36,15 @@ class ThumbCli(object):
             if not c:
                 raise Exception()
         except:
-            c = {k: set() for k in ["thumbable", "pil", "vips", "ffi", "ffv", "ffa"]}
+            c = {
+                k: set()
+                for k in ["thumbable", "pil", "vips", "raw", "ffi", "ffv", "ffa"]
+            }
 
         self.thumbable = c["thumbable"]
         self.fmt_pil = c["pil"]
         self.fmt_vips = c["vips"]
+        self.fmt_raw = c["raw"]
         self.fmt_ffi = c["ffi"]
         self.fmt_ffv = c["ffv"]
         self.fmt_ffa = c["ffa"]
@@ -57,13 +66,17 @@ class ThumbCli(object):
         if is_vid and "dvthumb" in dbv.flags:
             return None
 
-        want_opus = fmt in ("opus", "caf", "mp3")
+        want_opus = fmt in EXTS_AC
         is_au = ext in self.fmt_ffa
         is_vau = want_opus and ext in self.fmt_ffv
         if is_au or is_vau:
             if want_opus:
                 if self.args.no_acode:
                     return None
+                elif fmt == "caf" and self.args.no_caf:
+                    fmt = "mp3"
+                elif fmt == "owa" and self.args.no_owa:
+                    fmt = "mp3"
             else:
                 if "dathumb" in dbv.flags:
                     return None
@@ -79,7 +92,7 @@ class ThumbCli(object):
         if rem.startswith(".hist/th/") and rem.split(".")[-1] in ["webp", "jpg", "png"]:
             return os.path.join(ptop, rem)
 
-        if fmt[:1] in "jw":
+        if fmt[:1] in "jw" and fmt != "wav":
             sfmt = fmt[:1]
 
             if sfmt == "j" and self.args.th_no_jpg:
@@ -109,18 +122,18 @@ class ThumbCli(object):
             fmt = sfmt
 
         elif fmt[:1] == "p" and not is_au and not is_vid:
-            t = "cannot thumbnail [%s]: png only allowed for waveforms"
-            self.log(t % (rem), 6)
+            t = "cannot thumbnail %r: png only allowed for waveforms"
+            self.log(t % (rem,), 6)
             return None
 
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
         tpaths = [tpath]
-        if fmt == "w":
+        if fmt[:1] == "w" and fmt != "wav":
             # also check for jpg (maybe webp is unavailable)
             tpaths.append(tpath.rsplit(".", 1)[0] + ".jpg")
 
@@ -153,8 +166,22 @@ class ThumbCli(object):
         if abort:
             return None
 
-        if not bos.path.getsize(os.path.join(ptop, rem)):
-            return None
+        ap = os.path.join(ptop, rem)
+        try:
+            st = bos.stat(ap)
+            if not st.st_size or not stat.S_ISREG(st.st_mode):
+                return None
+
+            with open(ap, "rb", 4) as f:
+                if not f.read(4):
+                    raise Exception()
+        except OSError as ex:
+            if ex.errno == errno.ENOENT:
+                raise Pebkac(404)
+            else:
+                raise Pebkac(500, IOERROR % (ex,))
+        except Exception as ex:
+            raise Pebkac(500, IOERROR % (ex,))
 
         x = self.broker.ask("thumbsrv.get", ptop, rem, mtime, fmt)
         return x.get()  # type: ignore

copyparty/th_srv.py

@@ -2,10 +2,13 @@
 from __future__ import print_function, unicode_literals
 
 import hashlib
+import io
 import logging
 import os
+import re
 import shutil
 import subprocess as sp
+import tempfile
 import threading
 import time
 
@@ -18,22 +21,22 @@ from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, au_unpk, ffprobe
 from .util import BytesIO  # type: ignore
 from .util import (
     FFMPEG_URL,
+    VF_CAREFUL,
     Cooldown,
     Daemon,
-    Pebkac,
     afsenc,
+    atomic_move,
     fsenc,
     min_ex,
     runcmd,
     statdir,
     ub64enc,
     vsplit,
-    wrename,
     wunlink,
 )
 
 if True:  # pylint: disable=using-constant-test
-    from typing import Optional, Union
+    from typing import Any, Optional, Union
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -47,6 +50,13 @@ HAVE_HEIF = False
 HAVE_AVIF = False
 HAVE_WEBP = False
 
+EXTS_TH = set(["jpg", "webp", "png"])
+EXTS_AC = set(["opus", "owa", "caf", "mp3", "flac", "wav"])
+EXTS_SPEC_SAFE = set("aif aiff flac mp3 opus wav".split())
+
+PTN_TS = re.compile("^-?[0-9a-f]{8,10}$")
+
+
 try:
     if os.environ.get("PRTY_NO_PIL"):
         raise Exception()
@@ -76,7 +86,10 @@ try:
         if os.environ.get("PRTY_NO_PIL_HEIF"):
             raise Exception()
 
-        from pyheif_pillow_opener import register_heif_opener
+        try:
+            from pillow_heif import register_heif_opener
+        except ImportError:
+            from pyheif_pillow_opener import register_heif_opener
 
         register_heif_opener()
         HAVE_HEIF = True
@@ -87,6 +100,10 @@ try:
         if os.environ.get("PRTY_NO_PIL_AVIF"):
             raise Exception()
 
+        if ".avif" in Image.registered_extensions():
+            HAVE_AVIF = True
+            raise Exception()
+
         import pillow_avif  # noqa: F401  # pylint: disable=unused-import
 
         HAVE_AVIF = True
@@ -99,14 +116,28 @@ except:
 
 try:
     if os.environ.get("PRTY_NO_VIPS"):
-        raise Exception()
+        raise ImportError()
 
     HAVE_VIPS = True
     import pyvips
 
     logging.getLogger("pyvips").setLevel(logging.WARNING)
-except:
+except Exception as e:
     HAVE_VIPS = False
+    if not isinstance(e, ImportError):
+        logging.warning("libvips found, but failed to load: " + str(e))
+
+
+try:
+    if os.environ.get("PRTY_NO_RAW"):
+        raise Exception()
+
+    HAVE_RAW = True
+    import rawpy
+
+    logging.getLogger("rawpy").setLevel(logging.WARNING)
+except:
+    HAVE_RAW = False
 
 
 th_dir_cache = {}
@@ -140,7 +171,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
     h = hashlib.sha512(afsenc(fn)).digest()
     fn = ub64enc(h).decode("ascii")[:24]
 
-    if fmt in ("opus", "caf", "mp3"):
+    if fmt in EXTS_AC:
         cat = "ac"
     else:
         fc = fmt[:1]
@@ -161,11 +192,15 @@ class ThumbSrv(object):
 
         self.mutex = threading.Lock()
         self.busy: dict[str, list[threading.Condition]] = {}
+        self.untemp: dict[str, list[str]] = {}
         self.ram: dict[str, float] = {}
         self.memcond = threading.Condition(self.mutex)
         self.stopping = False
+        self.rm_nullthumbs = True  # forget failed conversions on startup
         self.nthr = max(1, self.args.th_mt)
 
+        self.exts_spec_unsafe = set(self.args.th_spec_cnv.split(","))
+
         self.q: Queue[Optional[tuple[str, str, str, VFS]]] = Queue(self.nthr * 4)
         for n in range(self.nthr):
             Daemon(self.worker, "thumb-{}-{}".format(n, self.nthr))
@@ -188,11 +223,19 @@ class ThumbSrv(object):
         if self.args.th_clean:
             Daemon(self.cleaner, "thumb.cln")
 
-        self.fmt_pil, self.fmt_vips, self.fmt_ffi, self.fmt_ffv, self.fmt_ffa = [
+        (
+            self.fmt_pil,
+            self.fmt_vips,
+            self.fmt_raw,
+            self.fmt_ffi,
+            self.fmt_ffv,
+            self.fmt_ffa,
+        ) = [
             set(y.split(","))
             for y in [
                 self.args.th_r_pil,
                 self.args.th_r_vips,
+                self.args.th_r_raw,
                 self.args.th_r_ffi,
                 self.args.th_r_ffv,
                 self.args.th_r_ffa,
@@ -215,6 +258,9 @@ class ThumbSrv(object):
         if "vips" in self.args.th_dec:
             self.thumbable |= self.fmt_vips
 
+        if "raw" in self.args.th_dec:
+            self.thumbable |= self.fmt_raw
+
         if "ff" in self.args.th_dec:
             for zss in [self.fmt_ffi, self.fmt_ffv, self.fmt_ffa]:
                 self.thumbable |= zss
@@ -239,7 +285,7 @@ class ThumbSrv(object):
     def get(self, ptop: str, rem: str, mtime: float, fmt: str) -> Optional[str]:
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
@@ -249,10 +295,11 @@ class ThumbSrv(object):
         with self.mutex:
             try:
                 self.busy[tpath].append(cond)
-                self.log("joined waiting room for %s" % (tpath,))
+                self.log("joined waiting room for %r" % (tpath,))
             except:
                 thdir = os.path.dirname(tpath)
-                bos.makedirs(os.path.join(thdir, "w"))
+                chmod = bos.MKD_700 if self.args.free_umask else bos.MKD_755
+                bos.makedirs(os.path.join(thdir, "w"), vf=chmod)
 
                 inf_path = os.path.join(thdir, "dir.txt")
                 if not bos.path.exists(inf_path):
@@ -266,11 +313,11 @@ class ThumbSrv(object):
             allvols = list(self.asrv.vfs.all_vols.values())
             vn = next((x for x in allvols if x.realpath == ptop), None)
             if not vn:
-                self.log("ptop [{}] not in {}".format(ptop, allvols), 3)
-                vn = self.asrv.vfs.all_aps[0][1]
+                self.log("ptop %r not in %s" % (ptop, allvols), 3)
+                vn = self.asrv.vfs.all_aps[0][1][0]
 
             self.q.put((abspath, tpath, fmt, vn))
-            self.log("conv {} :{} \033[0m{}".format(tpath, fmt, abspath), c=6)
+            self.log("conv %r :%s \033[0m%r" % (tpath, fmt, abspath), 6)
 
         while not self.stopping:
             with self.mutex:
@@ -295,6 +342,7 @@ class ThumbSrv(object):
             "thumbable": self.thumbable,
             "pil": self.fmt_pil,
             "vips": self.fmt_vips,
+            "raw": self.fmt_raw,
             "ffi": self.fmt_ffi,
             "ffv": self.fmt_ffv,
             "ffa": self.fmt_ffa,
@@ -334,10 +382,13 @@ class ThumbSrv(object):
                 ap_unpk = abspath
 
             if not bos.path.exists(tpath):
-                want_mp3 = tpath.endswith(".mp3")
-                want_opus = tpath.endswith(".opus") or tpath.endswith(".caf")
-                want_png = tpath.endswith(".png")
-                want_au = want_mp3 or want_opus
+                tex = tpath.rsplit(".", 1)[-1]
+                want_mp3 = tex == "mp3"
+                want_opus = tex in ("opus", "owa", "caf")
+                want_flac = tex == "flac"
+                want_wav = tex == "wav"
+                want_png = tex == "png"
+                want_au = want_mp3 or want_opus or want_flac or want_wav
                 for lib in self.args.th_dec:
                     can_au = lib == "ff" and (
                         ext in self.fmt_ffa or ext in self.fmt_ffv
@@ -347,11 +398,17 @@ class ThumbSrv(object):
                         funs.append(self.conv_pil)
                     elif lib == "vips" and ext in self.fmt_vips:
                         funs.append(self.conv_vips)
+                    elif lib == "raw" and ext in self.fmt_raw:
+                        funs.append(self.conv_raw)
                     elif can_au and (want_png or want_au):
                         if want_opus:
                             funs.append(self.conv_opus)
                         elif want_mp3:
                             funs.append(self.conv_mp3)
+                        elif want_flac:
+                            funs.append(self.conv_flac)
+                        elif want_wav:
+                            funs.append(self.conv_wav)
                         elif want_png:
                             funs.append(self.conv_waves)
                             png_ok = True
@@ -375,14 +432,18 @@ class ThumbSrv(object):
                     fun(ap_unpk, ttpath, fmt, vn)
                     break
                 except Exception as ex:
-                    msg = "{} could not create thumbnail of {}\n{}"
-                    msg = msg.format(fun.__name__, abspath, min_ex())
+                    msg = "%s could not create thumbnail of %r\n%s"
+                    msg = msg % (fun.__name__, abspath, min_ex())
                     c: Union[str, int] = 1 if "<Signals.SIG" in msg else "90"
                     self.log(msg, c)
                     if getattr(ex, "returncode", 0) != 321:
                         if fun == funs[-1]:
-                            with open(ttpath, "wb") as _:
-                                pass
+                            try:
+                                with open(ttpath, "wb") as _:
+                                    pass
+                            except Exception as ex:
+                                t = "failed to create the file [%s]: %r"
+                                self.log(t % (ttpath, ex), 3)
                     else:
                         # ffmpeg may spawn empty files on windows
                         try:
@@ -394,14 +455,25 @@ class ThumbSrv(object):
                 wunlink(self.log, ap_unpk, vn.flags)
 
             try:
-                wrename(self.log, ttpath, tpath, vn.flags)
-            except:
+                atomic_move(self.log, ttpath, tpath, vn.flags)
+            except Exception as ex:
+                if not os.path.exists(tpath):
+                    t = "failed to move  [%s]  to  [%s]:  %r"
+                    self.log(t % (ttpath, tpath, ex), 3)
                 pass
 
+            untemp = []
             with self.mutex:
                 subs = self.busy[tpath]
                 del self.busy[tpath]
                 self.ram.pop(ttpath, None)
+                untemp = self.untemp.pop(ttpath, None) or []
+
+            for ap in untemp:
+                try:
+                    wunlink(self.log, ap, VF_CAREFUL)
+                except:
+                    pass
 
             for x in subs:
                 with x:
@@ -440,35 +512,38 @@ class ThumbSrv(object):
 
         return im
 
+    def conv_image_pil(self, im: "Image.Image", tpath: str, fmt: str, vn: VFS) -> None:
+        try:
+            im = self.fancy_pillow(im, fmt, vn)
+        except Exception as ex:
+            self.log("fancy_pillow {}".format(ex), "90")
+            im.thumbnail(self.getres(vn, fmt))
+
+        fmts = ["RGB", "L"]
+        args = {"quality": 40}
+
+        if tpath.endswith(".webp"):
+            # quality 80 = pillow-default
+            # quality 75 = ffmpeg-default
+            # method 0 = pillow-default, fast
+            # method 4 = ffmpeg-default
+            # method 6 = max, slow
+            fmts.extend(("RGBA", "LA"))
+            args["method"] = 6
+        else:
+            # default q = 75
+            args["progressive"] = True
+
+        if im.mode not in fmts:
+            # print("conv {}".format(im.mode))
+            im = im.convert("RGB")
+
+        im.save(tpath, **args)
+
     def conv_pil(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         self.wait4ram(0.2, tpath)
         with Image.open(fsenc(abspath)) as im:
-            try:
-                im = self.fancy_pillow(im, fmt, vn)
-            except Exception as ex:
-                self.log("fancy_pillow {}".format(ex), "90")
-                im.thumbnail(self.getres(vn, fmt))
-
-            fmts = ["RGB", "L"]
-            args = {"quality": 40}
-
-            if tpath.endswith(".webp"):
-                # quality 80 = pillow-default
-                # quality 75 = ffmpeg-default
-                # method 0 = pillow-default, fast
-                # method 4 = ffmpeg-default
-                # method 6 = max, slow
-                fmts.extend(("RGBA", "LA"))
-                args["method"] = 6
-            else:
-                # default q = 75
-                args["progressive"] = True
-
-            if im.mode not in fmts:
-                # print("conv {}".format(im.mode))
-                im = im.convert("RGB")
-
-            im.save(tpath, **args)
+            self.conv_image_pil(im, tpath, fmt, vn)
 
     def conv_vips(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         self.wait4ram(0.2, tpath)
@@ -491,9 +566,53 @@ class ThumbSrv(object):
         assert img  # type: ignore  # !rm
         img.write_to_file(tpath, Q=40)
 
+    def conv_raw(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        self.wait4ram(0.2, tpath)
+        with rawpy.imread(abspath) as raw:
+            thumb = raw.extract_thumb()
+        if thumb.format == rawpy.ThumbFormat.JPEG and tpath.endswith(".jpg"):
+            # if we have a jpg thumbnail and no webp output is available,
+            # just write the jpg directly (it'll be the wrong size, but it's fast)
+            with open(tpath, "wb") as f:
+                f.write(thumb.data)
+        if HAVE_VIPS:
+            crops = ["centre", "none"]
+            if "f" in fmt:
+                crops = ["none"]
+            w, h = self.getres(vn, fmt)
+            kw = {"height": h, "size": "down", "intent": "relative"}
+
+            for c in crops:
+                try:
+                    kw["crop"] = c
+                    if thumb.format == rawpy.ThumbFormat.BITMAP:
+                        img = pyvips.Image.new_from_array(
+                            thumb.data, interpretation="rgb"
+                        )
+                        img = img.thumbnail_image(w, **kw)
+                    else:
+                        img = pyvips.Image.thumbnail_buffer(thumb.data, w, **kw)
+                    break
+                except:
+                    if c == crops[-1]:
+                        raise
+
+            assert img  # type: ignore  # !rm
+            img.write_to_file(tpath, Q=40)
+        elif HAVE_PIL:
+            if thumb.format == rawpy.ThumbFormat.BITMAP:
+                im = Image.fromarray(thumb.data, "RGB")
+            else:
+                im = Image.open(io.BytesIO(thumb.data))
+            self.conv_image_pil(im, tpath, fmt, vn)
+        else:
+            raise Exception(
+                "either pil or vips is needed to process embedded bitmap thumbnails in raw files"
+            )
+
     def conv_ffmpeg(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         self.wait4ram(0.2, tpath)
-        ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        ret, _, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if not ret:
             return
 
@@ -504,6 +623,17 @@ class ThumbSrv(object):
             dur = ret[".dur"][1] if ".dur" in ret else 4
             seek = [b"-ss", "{:.0f}".format(dur / 3).encode("utf-8")]
 
+        self._ffmpeg_im(abspath, tpath, fmt, vn, seek, b"0:v:0")
+
+    def _ffmpeg_im(
+        self,
+        abspath: str,
+        tpath: str,
+        fmt: str,
+        vn: VFS,
+        seek: list[bytes],
+        imap: bytes,
+    ) -> None:
         scale = "scale={0}:{1}:force_original_aspect_ratio="
         if "f" in fmt:
             scale += "decrease,setsar=1:1"
@@ -522,7 +652,7 @@ class ThumbSrv(object):
         cmd += seek
         cmd += [
             b"-i", fsenc(abspath),
-            b"-map", b"0:v:0",
+            b"-map", imap,
             b"-vf", bscale,
             b"-frames:v", b"1",
             b"-metadata:s:v:0", b"rotate=0",
@@ -543,11 +673,11 @@ class ThumbSrv(object):
             ]
 
         cmd += [fsenc(tpath)]
-        self._run_ff(cmd, vn)
+        self._run_ff(cmd, vn, "convt")
 
-    def _run_ff(self, cmd: list[bytes], vn: VFS, oom: int = 400) -> None:
+    def _run_ff(self, cmd: list[bytes], vn: VFS, kto: str, oom: int = 400) -> None:
         # self.log((b" ".join(cmd)).decode("utf-8"))
-        ret, _, serr = runcmd(cmd, timeout=vn.flags["convt"], nice=True, oom=oom)
+        ret, _, serr = runcmd(cmd, timeout=vn.flags[kto], nice=True, oom=oom)
         if not ret:
             return
 
@@ -591,7 +721,7 @@ class ThumbSrv(object):
         raise sp.CalledProcessError(ret, (cmd[0], b"...", cmd[-1]))
 
     def conv_waves(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
-        ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        ret, _, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
 
@@ -629,7 +759,7 @@ class ThumbSrv(object):
         # fmt: on
 
         cmd += [fsenc(tpath)]
-        self._run_ff(cmd, vn)
+        self._run_ff(cmd, vn, "convt")
 
         if "pngquant" in vn.flags:
             wtpath = tpath + ".png"
@@ -648,22 +778,70 @@ class ThumbSrv(object):
                 except:
                     pass
             else:
-                wrename(self.log, wtpath, tpath, vn.flags)
+                atomic_move(self.log, wtpath, tpath, vn.flags)
+
+    def conv_emb_cv(
+        self, abspath: str, tpath: str, fmt: str, vn: VFS, strm: dict[str, Any]
+    ) -> None:
+        self.wait4ram(0.2, tpath)
+        self._ffmpeg_im(
+            abspath, tpath, fmt, vn, [], b"0:" + strm["index"].encode("ascii")
+        )
 
     def conv_spec(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
-        ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        ret, raw, strms, ctnr = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
 
+        want_spec = vn.flags.get("th_spec_p", 1)
+        if want_spec < 2:
+            for strm in strms:
+                if (
+                    strm.get("codec_type") == "video"
+                    and strm.get("DISPOSITION:attached_pic") == "1"
+                ):
+                    return self.conv_emb_cv(abspath, tpath, fmt, vn, strm)
+
+        if not want_spec:
+            raise Exception("spectrograms forbidden by volflag")
+
+        fext = abspath.split(".")[-1].lower()
+
         # https://trac.ffmpeg.org/ticket/10797
         # expect 1 GiB every 600 seconds when duration is tricky;
         # simple filetypes are generally safer so let's special-case those
-        safe = ("flac", "wav", "aif", "aiff", "opus")
-        coeff = 1800 if abspath.split(".")[-1].lower() in safe else 600
-        dur = ret[".dur"][1] if ".dur" in ret else 300
+        coeff = 1800 if fext in EXTS_SPEC_SAFE else 600
+        dur = ret[".dur"][1] if ".dur" in ret else 900
         need = 0.2 + dur / coeff
         self.wait4ram(need, tpath)
 
+        infile = abspath
+        if dur >= 900 or fext in self.exts_spec_unsafe:
+            with tempfile.NamedTemporaryFile(suffix=".spec.flac", delete=False) as f:
+                f.write(b"h")
+                infile = f.name
+                try:
+                    self.untemp[tpath].append(infile)
+                except:
+                    self.untemp[tpath] = [infile]
+
+            # fmt: off
+            cmd = [
+                b"ffmpeg",
+                b"-nostdin",
+                b"-v", b"error",
+                b"-hide_banner",
+                b"-i", fsenc(abspath),
+                b"-map", b"0:a:0",
+                b"-ac", b"1",
+                b"-ar", b"48000",
+                b"-sample_fmt", b"s16",
+                b"-t", b"900",
+                b"-y", fsenc(infile),
+            ]
+            # fmt: on
+            self._run_ff(cmd, vn, "convt")
+
         fc = "[0:a:0]aresample=48000{},showspectrumpic=s="
         if "3" in fmt:
             fc += "1280x1024,crop=1420:1056:70:48[o]"
@@ -683,7 +861,7 @@ class ThumbSrv(object):
             b"-nostdin",
             b"-v", b"error",
             b"-hide_banner",
-            b"-i", fsenc(abspath),
+            b"-i", fsenc(infile),
             b"-filter_complex", fc.encode("utf-8"),
             b"-map", b"[o]",
             b"-frames:v", b"1",
@@ -704,7 +882,7 @@ class ThumbSrv(object):
             ]
 
         cmd += [fsenc(tpath)]
-        self._run_ff(cmd, vn)
+        self._run_ff(cmd, vn, "convt")
 
     def conv_mp3(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         quality = self.args.q_mp3.lower()
@@ -712,7 +890,7 @@ class ThumbSrv(object):
             raise Exception("disabled in server config")
 
         self.wait4ram(0.2, tpath)
-        tags, rawtags = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        tags, rawtags, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in tags:
             raise Exception("not audio")
 
@@ -743,58 +921,173 @@ class ThumbSrv(object):
             fsenc(tpath)
         ]
         # fmt: on
-        self._run_ff(cmd, vn, oom=300)
+        self._run_ff(cmd, vn, "aconvt", oom=300)
+
+    def conv_flac(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        if self.args.no_acode or not self.args.allow_flac:
+            raise Exception("flac not permitted in server config")
+
+        self.wait4ram(0.2, tpath)
+        tags, _, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        if "ac" not in tags:
+            raise Exception("not audio")
+
+        self.log("conv2 flac", 6)
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map", b"0:a:0",
+            b"-c:a", b"flac",
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, "aconvt", oom=300)
+
+    def conv_wav(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        if self.args.no_acode or not self.args.allow_wav:
+            raise Exception("wav not permitted in server config")
+
+        self.wait4ram(0.2, tpath)
+        tags, _, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        if "ac" not in tags:
+            raise Exception("not audio")
+
+        bits = tags[".bps"][1]
+        if bits == 0.0:
+            bits = tags[".bprs"][1]
+
+        codec = b"pcm_s32le"
+        if bits <= 16.0:
+            codec = b"pcm_s16le"
+        elif bits <= 24.0:
+            codec = b"pcm_s24le"
+
+        self.log("conv2 wav", 6)
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map", b"0:a:0",
+            b"-c:a", codec,
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, "aconvt", oom=300)
 
     def conv_opus(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         if self.args.no_acode or not self.args.q_opus:
             raise Exception("disabled in server config")
 
         self.wait4ram(0.2, tpath)
-        tags, rawtags = ffprobe(abspath, int(vn.flags["convt"] / 2))
+        tags, rawtags, _, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in tags:
             raise Exception("not audio")
 
+        sq = "%dk" % (self.args.q_opus,)
+        bq = sq.encode("ascii")
+        if tags["ac"][1] == "opus":
+            enc = "-c:a copy"
+        else:
+            enc = "-c:a libopus -b:a " + sq
+
+        fun = self._conv_caf if fmt == "caf" else self._conv_owa
+
+        fun(abspath, tpath, tags, rawtags, enc, bq, vn)
+
+    def _conv_owa(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        if tpath.endswith(".owa"):
+            container = b"webm"
+            tagset = [b"-map_metadata", b"-1"]
+        else:
+            container = b"opus"
+            tagset = self.big_tags(rawtags)
+
+        self.log("conv2 %s [%s]" % (container, enc), 6)
+        benc = enc.encode("ascii").split(b" ")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+        ] + tagset + [
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", container,
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, "aconvt", oom=300)
+
+    def _conv_caf(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        tmp_opus = tpath + ".opus"
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
+
         try:
             dur = tags[".dur"][1]
         except:
             dur = 0
 
-        src_opus = abspath.lower().endswith(".opus") or tags["ac"][1] == "opus"
-        want_caf = tpath.endswith(".caf")
-        tmp_opus = tpath
-        if want_caf:
-            tmp_opus = tpath + ".opus"
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
+        self.log("conv2 caf-tmp [%s]" % (enc,), 6)
+        benc = enc.encode("ascii").split(b" ")
 
-        caf_src = abspath if src_opus else tmp_opus
-        bq = ("%dk" % (self.args.q_opus,)).encode("ascii")
-
-        if not want_caf or not src_opus:
-            # fmt: off
-            cmd = [
-                b"ffmpeg",
-                b"-nostdin",
-                b"-v", b"error",
-                b"-hide_banner",
-                b"-i", fsenc(abspath),
-            ] + self.big_tags(rawtags) + [
-                b"-map", b"0:a:0",
-                b"-c:a", b"libopus",
-                b"-b:a", bq,
-                fsenc(tmp_opus)
-            ]
-            # fmt: on
-            self._run_ff(cmd, vn, oom=300)
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map_metadata", b"-1",
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", b"opus",
+            fsenc(tmp_opus)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, "aconvt", oom=300)
 
         # iOS fails to play some "insufficiently complex" files
         # (average file shorter than 8 seconds), so of course we
         # fix that by mixing in some inaudible pink noise :^)
         # 6.3 sec seems like the cutoff so lets do 7, and
         # 7 sec of psyqui-musou.opus @ 3:50 is 174 KiB
-        if want_caf and (dur < 20 or bos.path.getsize(caf_src) < 256 * 1024):
+        sz = bos.path.getsize(tmp_opus)
+        if dur < 20 or sz < 256 * 1024:
+            zs = bq.decode("ascii")
+            self.log("conv2 caf-transcode; dur=%d sz=%d q=%s" % (dur, sz, zs), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
@@ -811,17 +1104,18 @@ class ThumbSrv(object):
                 fsenc(tpath)
             ]
             # fmt: on
-            self._run_ff(cmd, vn, oom=300)
+            self._run_ff(cmd, vn, "aconvt", oom=300)
 
-        elif want_caf:
+        else:
             # simple remux should be safe
+            self.log("conv2 caf-remux; dur=%d sz=%d" % (dur, sz), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
                 b"-nostdin",
                 b"-v", b"error",
                 b"-hide_banner",
-                b"-i", fsenc(abspath if src_opus else tmp_opus),
+                b"-i", fsenc(tmp_opus),
                 b"-map_metadata", b"-1",
                 b"-map", b"0:a:0",
                 b"-c:a", b"copy",
@@ -829,13 +1123,12 @@ class ThumbSrv(object):
                 fsenc(tpath)
             ]
             # fmt: on
-            self._run_ff(cmd, vn, oom=300)
+            self._run_ff(cmd, vn, "aconvt", oom=300)
 
-        if tmp_opus != tpath:
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
 
     def big_tags(self, raw_tags: dict[str, list[str]]) -> list[bytes]:
         ret = []
@@ -862,7 +1155,6 @@ class ThumbSrv(object):
     def cleaner(self) -> None:
         interval = self.args.th_clean
         while True:
-            time.sleep(interval)
             ndirs = 0
             for vol, histpath in self.asrv.vfs.histtab.items():
                 if histpath.startswith(vol):
@@ -876,6 +1168,8 @@ class ThumbSrv(object):
                     self.log("\033[Jcln err in %s: %r" % (histpath, ex), 3)
 
             self.log("\033[Jcln ok; rm {} dirs".format(ndirs))
+            self.rm_nullthumbs = False
+            time.sleep(interval)
 
     def clean(self, histpath: str) -> int:
         ret = 0
@@ -890,13 +1184,15 @@ class ThumbSrv(object):
 
     def _clean(self, cat: str, thumbpath: str) -> int:
         # self.log("cln {}".format(thumbpath))
-        exts = ["jpg", "webp", "png"] if cat == "th" else ["opus", "caf", "mp3"]
+        exts = EXTS_TH if cat == "th" else EXTS_AC
         maxage = getattr(self.args, cat + "_maxage")
         now = time.time()
         prev_b64 = None
         prev_fp = ""
         try:
-            t1 = statdir(self.log_func, not self.args.no_scandir, False, thumbpath)
+            t1 = statdir(
+                self.log_func, not self.args.no_scandir, False, thumbpath, False
+            )
             ents = sorted(list(t1))
         except:
             return 0
@@ -929,6 +1225,8 @@ class ThumbSrv(object):
             # thumb file
             try:
                 b64, ts, ext = f.split(".")
+                if len(ts) > 8 and PTN_TS.match(ts):
+                    ts = "yeahokay"
                 if len(b64) != 24 or len(ts) != 8 or ext not in exts:
                     raise Exception()
             except:
@@ -937,6 +1235,10 @@ class ThumbSrv(object):
 
                 continue
 
+            if self.rm_nullthumbs and not inf.st_size:
+                bos.unlink(fp)
+                continue
+
             if b64 == prev_b64:
                 self.log("rm replaced [{}]".format(fp))
                 bos.unlink(prev_fp)

copyparty/u2idx.py

@@ -70,6 +70,9 @@ class U2idx(object):
         self.log_func("u2idx", msg, c)
 
     def shutdown(self) -> None:
+        if not HAVE_SQLITE3:
+            return
+
         for cur in self.cur.values():
             db = cur.connection
             try:
@@ -80,6 +83,12 @@ class U2idx(object):
             cur.close()
             db.close()
 
+        for cur in (self.mem_cur, self.sh_cur):
+            if cur:
+                db = cur.connection
+                cur.close()
+                db.close()
+
     def fsearch(
         self, uname: str, vols: list[VFS], body: dict[str, Any]
     ) -> list[dict[str, Any]]:
@@ -95,7 +104,7 @@ class U2idx(object):
         uv: list[Union[str, int]] = [wark[:16], wark]
 
         try:
-            return self.run_query(uname, vols, uq, uv, False, 99999)[0]
+            return self.run_query(uname, vols, uq, uv, False, True, 99999)[0]
         except:
             raise Pebkac(500, min_ex())
 
@@ -125,9 +134,9 @@ class U2idx(object):
         assert sqlite3  # type: ignore  # !rm
 
         ptop = vn.realpath
-        histpath = self.asrv.vfs.histtab.get(ptop)
+        histpath = self.asrv.vfs.dbpaths.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no dbpath for %r" % (ptop,))
             return None
 
         db_path = os.path.join(histpath, "up2k.db")
@@ -142,7 +151,7 @@ class U2idx(object):
                 db = sqlite3.connect(uri, timeout=2, uri=True, check_same_thread=False)
                 cur = db.cursor()
                 cur.execute('pragma table_info("up")').fetchone()
-                self.log("ro: {}".format(db_path))
+                self.log("ro: %r" % (db_path,))
             except:
                 self.log("could not open read-only: {}\n{}".format(uri, min_ex()))
                 # may not fail until the pragma so unset it
@@ -152,7 +161,7 @@ class U2idx(object):
             # on windows, this steals the write-lock from up2k.deferred_init --
             # seen on win 10.0.17763.2686, py 3.10.4, sqlite 3.37.2
             cur = sqlite3.connect(db_path, timeout=2, check_same_thread=False).cursor()
-            self.log("opened {}".format(db_path))
+            self.log("opened %r" % (db_path,))
 
         self.cur[ptop] = cur
         return cur
@@ -301,7 +310,7 @@ class U2idx(object):
                 q += " lower({}) {} ? ) ".format(field, oper)
 
         try:
-            return self.run_query(uname, vols, q, va, have_mt, lim)
+            return self.run_query(uname, vols, q, va, have_mt, True, lim)
         except Exception as ex:
             raise Pebkac(500, repr(ex))
 
@@ -312,9 +321,11 @@ class U2idx(object):
         uq: str,
         uv: list[Union[str, int]],
         have_mt: bool,
+        sort: bool,
         lim: int,
     ) -> tuple[list[dict[str, Any]], list[str], bool]:
-        if self.args.srch_dbg:
+        dbg = self.args.srch_dbg
+        if dbg:
             t = "searching across all %s volumes in which the user has 'r' (full read access):\n  %s"
             zs = "\n  ".join(["/%s = %s" % (x.vpath, x.realpath) for x in vols])
             self.log(t % (len(vols), zs), 5)
@@ -357,14 +368,14 @@ class U2idx(object):
             if not cur:
                 continue
 
-            excl = []
-            for vp2 in self.asrv.vfs.all_vols.keys():
-                if vp2.startswith((vtop + "/").lstrip("/")) and vtop != vp2:
-                    excl.append(vp2[len(vtop) :].lstrip("/"))
+            dots = flags.get("dotsrch") and uname in vol.axs.udot
+            zs = "srch_re_dots" if dots else "srch_re_nodot"
+            rex: re.Pattern = flags.get(zs)  # type: ignore
 
-            if self.args.srch_dbg:
-                t = "searching in volume /%s (%s), excludelist %s"
-                self.log(t % (vtop, ptop, excl), 5)
+            if dbg:
+                t = "searching in volume /%s (%s), excluding %s"
+                self.log(t % (vtop, ptop, rex.pattern), 5)
+                rex_cfg: Optional[re.Pattern] = flags.get("srch_excl")
 
             self.active_cur = cur
 
@@ -377,29 +388,31 @@ class U2idx(object):
 
             sret = []
             fk = flags.get("fk")
-            dots = flags.get("dotsrch") and uname in vol.axs.udot
             fk_alg = 2 if "fka" in flags else 1
             c = cur.execute(uq, tuple(vuv))
             for hit in c:
-                w, ts, sz, rd, fn, ip, at = hit[:7]
+                w, ts, sz, rd, fn = hit[:5]
 
                 if rd.startswith("//") or fn.startswith("//"):
                     rd, fn = s3dec(rd, fn)
 
-                if rd in excl or any([x for x in excl if rd.startswith(x + "/")]):
-                    if self.args.srch_dbg:
-                        zs = vjoin(vjoin(vtop, rd), fn)
-                        t = "database inconsistency in volume '/%s'; ignoring: %s"
-                        self.log(t % (vtop, zs), 1)
+                vp = vjoin(vjoin(vtop, rd), fn)
+
+                if vp in seen_rps:
                     continue
 
-                rp = quotep("/".join([x for x in [vtop, rd, fn] if x]))
-                if not dots and "/." in ("/" + rp):
-                    continue
-
-                if rp in seen_rps:
+                if rex.search(vp):
+                    if dbg:
+                        if rex_cfg and rex_cfg.search(vp):  # type: ignore
+                            self.log("filtered by srch_excl: %s" % (vp,), 6)
+                        elif not dots and "/." in ("/" + vp):
+                            pass
+                        else:
+                            t = "database inconsistency in volume '/%s'; ignoring: %s"
+                            self.log(t % (vtop, vp), 1)
                     continue
 
+                rp = quotep(vp)
                 if not fk:
                     suf = ""
                 else:
@@ -421,7 +434,7 @@ class U2idx(object):
                 if lim < 0:
                     break
 
-                if self.args.srch_dbg:
+                if dbg:
                     t = "in volume '/%s': hit: %s"
                     self.log(t % (vtop, rp), 5)
 
@@ -451,14 +464,15 @@ class U2idx(object):
             ret.extend(sret)
             # print("[{}] {}".format(ptop, sret))
 
-            if self.args.srch_dbg:
+            if dbg:
                 t = "in volume '/%s': got %d hits, %d total so far"
                 self.log(t % (vtop, len(sret), len(ret)), 5)
 
         done_flag.append(True)
         self.active_id = ""
 
-        ret.sort(key=itemgetter("rp"))
+        if sort:
+            ret.sort(key=itemgetter("rp"))
 
         return ret, list(taglist.keys()), lim < 0 and not clamped
 

copyparty/web/baguettebox.js

@@ -32,7 +32,7 @@ window.baguetteBox = (function () {
         scrollCSS = ['', ''],
         scrollTimer = 0,
         re_i = /^[^?]+\.(a?png|avif|bmp|gif|heif|jpe?g|jfif|svg|webp)(\?|$)/i,
-        re_v = /^[^?]+\.(webm|mkv|mp4)(\?|$)/i,
+        re_v = /^[^?]+\.(webm|mkv|mp4|m4v|mov)(\?|$)/i,
         anims = ['slideIn', 'fadeIn', 'none'],
         data = {},  // all galleries
         imagesElements = [],
@@ -48,6 +48,7 @@ window.baguetteBox = (function () {
 
     var onFSC = function (e) {
         isFullscreen = !!document.fullscreenElement;
+        clmod(document.documentElement, 'bb_fsc', isFullscreen);
     };
 
     var overlayClickHandler = function (e) {
@@ -402,7 +403,7 @@ window.baguetteBox = (function () {
             if (isFullscreen)
                 document.exitFullscreen();
             else
-                (vid() || ebi('bbox-overlay')).requestFullscreen();
+                ebi('bbox-overlay').requestFullscreen();
         }
         catch (ex) {
             if (IPHONE)
@@ -592,9 +593,7 @@ window.baguetteBox = (function () {
             preloadPrev(currentIndex);
         });
 
-        clmod(ebi('bbox-btns'), 'off');
-        clmod(btnPrev, 'off');
-        clmod(btnNext, 'off');
+        show_buttons(0);
 
         updateOffset();
         overlay.style.display = 'block';
@@ -633,6 +632,9 @@ window.baguetteBox = (function () {
         catch (ex) { }
         isFullscreen = false;
 
+        if (toast.tag == 'bb-ded')
+            toast.hide();
+
         if (dtor || overlay.style.display === 'none')
             return;
 
@@ -668,6 +670,7 @@ window.baguetteBox = (function () {
             if (v == keep)
                 continue;
 
+            unbind(v, 'error', lerr);
             v.src = '';
             v.load();
 
@@ -695,6 +698,28 @@ window.baguetteBox = (function () {
         }
     }
 
+    function lerr() {
+        var t;
+        try {
+            t = this.getAttribute('src');
+            t = uricom_dec(t.split('/').pop().split('?')[0]);
+        }
+        catch (ex) { }
+
+        t = 'Failed to open ' + (t?t:'file');
+        console.log('bb-ded', t);
+        t += '\n\nEither the file is corrupt, or your browser does not understand the file format or codec';
+
+        try {
+            t += "\n\nerr#" + this.error.code + ", " + this.error.message;
+        }
+        catch (ex) { }
+ 
+        this.ded = esc(t);
+        if (this === vidimg())
+            toast.err(20, this.ded, 'bb-ded');
+    }
+
     function loadImage(index, callback) {
         var imageContainer = imagesElements[index];
         var galleryItem = currentGallery[index];
@@ -739,7 +764,8 @@ window.baguetteBox = (function () {
         var image = mknod(is_vid ? 'video' : 'img');
         clmod(imageContainer, 'vid', is_vid);
 
-        image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
+        bind(image, 'error', lerr);
+        bind(image, is_vid ? 'loadedmetadata' : 'load', function () {
             // Remove loader element
             qsr('#baguette-img-' + index + ' .bbox-spinner');
             if (!options.async && callback)
@@ -749,6 +775,8 @@ window.baguetteBox = (function () {
         if (is_vid) {
             image.volume = clamp(fcfg_get('vol', dvol / 100), 0, 1);
             image.setAttribute('controls', 'controls');
+            image.setAttribute('playsinline', '1');
+            // ios ignores poster
             image.onended = vidEnd;
             image.onplay = function () { show_buttons(1); };
             image.onpause = function () { show_buttons(); };
@@ -816,6 +844,12 @@ window.baguetteBox = (function () {
         });
         updateOffset();
 
+        var im = vidimg();
+        if (im && im.ded)
+            toast.err(20, im.ded, 'bb-ded');
+        else if (toast.tag == 'bb-ded')
+            toast.hide();
+
         if (options.animation == 'none')
             unvid(vid());
         else

copyparty/web/browser.html

@@ -109,7 +109,7 @@
 {%- for f in files %}
 <tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td>
 {%- if f.tags is defined %}
-	{%- for k in taglist %}<td>{{ f.tags[k] }}</td>{%- endfor %}
+	{%- for k in taglist %}<td>{{ f.tags[k]|e }}</td>{%- endfor %}
 {%- endif %}<td>{{ f.ext }}</td><td>{{ f.dt }}</td></tr>
 {%- endfor %}
 
@@ -124,24 +124,21 @@
 
 </div>
 
-	{%- if srv_info %}
 	<div id="srv_info"><span>{{ srv_info }}</span></div>
-	{%- endif %}
 
 	<div id="widget"></div>
 
 	<script>
-		var SR = {{ r|tojson }},
+		var SR = "{{ r }}",
+			CGV1 = {{ cgv1 }},
 			CGV = {{ cgv|tojson }},
 			TS = "{{ ts }}",
 			dtheme = "{{ dtheme }}",
 			srvinf = "{{ srv_info }}",
-			s_name = "{{ s_name }}",
 			lang = "{{ lang }}",
 			dfavico = "{{ favico }}",
-			have_tags_idx = {{ have_tags_idx|tojson }},
+			have_tags_idx = {{ have_tags_idx }},
 			sb_lg = "{{ sb_lg }}",
-			txt_ext = "{{ txt_ext }}",
 			logues = {{ logues|tojson if sb_lg else "[]" }},
 			ls0 = {{ ls0|tojson }};