mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2025-08-17 09:02:15 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity

add nosparse volflag + update s3 readme:

Browse source 
may improve upload performance in some particular uncommon scenarios,
for example if hdd-writes are uncached, and/or the hdd is drastically
slower than the network throughput

one particular usecase where nosparse *might* improve performance
is when the upload destination is cloud-storage provided by FUSE
(for example an s3 bucket) but this is educated guesswork
This commit is contained in:
ed 2025-01-19 16:28:40 +00:00
parent d9ae067e52
commit 917380ddbb
3 changed files with 16 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -1545,12 +1545,16 @@ connecting to an aws s3 bucket and similar
there is no built-in support for this, but you can use FUSE-software such as [rclone](https://rclone.org/) / [geesefs](https://github.com/yandex-cloud/geesefs) / [JuiceFS](https://juicefs.com/en/) to first mount your cloud storage as a local disk, and then let copyparty use (a folder in) that disk as a volume there is no built-in support for this, but you can use FUSE-software such as [rclone](https://rclone.org/) / [geesefs](https://github.com/yandex-cloud/geesefs) / [JuiceFS](https://juicefs.com/en/) to first mount your cloud storage as a local disk, and then let copyparty use (a folder in) that disk as a volume
you may experience poor upload performance this way, but that can sometimes be fixed by specifying the volflag `sparse` to force the use of sparse files; this has improved the upload speeds from `1.5 MiB/s` to over `80 MiB/s` in one case, but note that you are also more likely to discover funny bugs in your FUSE software this way, so buckle up you will probably get decent speeds with the default config, however most likely restricted to using one TCP connection per file, so the upload-client won't be able to send multiple chunks in parallel
> before [v1.13.5](https://github.com/9001/copyparty/releases/tag/v1.13.5) it was recommended to use the volflag `sparse` to force-allow multiple chunks in parallel; this would improve the upload-speed from `1.5 MiB/s` to over `80 MiB/s` at the risk of provoking latent bugs in S3 or JuiceFS. But v1.13.5 added chunk-stitching, so this is now probably much less important. On the contrary, `nosparse` *may* now increase performance in some cases. Please try all three options (default, `sparse`, `nosparse`) as the optimal choice depends on your network conditions and software stack (both the FUSE-driver and cloud-server)
someone has also tested geesefs in combination with [gocryptfs](https://nuetzlich.net/gocryptfs/) with surprisingly good results, getting 60 MiB/s upload speeds on a gbit line, but JuiceFS won with 80 MiB/s using its built-in encryption someone has also tested geesefs in combination with [gocryptfs](https://nuetzlich.net/gocryptfs/) with surprisingly good results, getting 60 MiB/s upload speeds on a gbit line, but JuiceFS won with 80 MiB/s using its built-in encryption
you may improve performance by specifying larger values for `--iobuf` / `--s-rd-sz` / `--s-wr-sz` you may improve performance by specifying larger values for `--iobuf` / `--s-rd-sz` / `--s-wr-sz`
> if you've experimented with this and made interesting observations, please share your findings so we can add a section with specific recommendations :-)
## hiding from google ## hiding from google

1
copyparty/cfg.py
View file

@ -144,6 +144,7 @@ flagcats = {
"noclone": "take dupe data from clients, even if available on HDD", "noclone": "take dupe data from clients, even if available on HDD",
"nodupe": "rejects existing files (instead of linking/cloning them)", "nodupe": "rejects existing files (instead of linking/cloning them)",
"sparse": "force use of sparse files, mainly for s3-backed storage", "sparse": "force use of sparse files, mainly for s3-backed storage",
"nosparse": "deny use of sparse files, mainly for slow storage",
"daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files", "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
"nosub": "forces all uploads into the top folder of the vfs", "nosub": "forces all uploads into the top folder of the vfs",
"magic": "enables filetype detection for nameless uploads", "magic": "enables filetype detection for nameless uploads",

14
copyparty/up2k.py
View file

@ -4905,7 +4905,8 @@ class Up2k(object):
except: except:
pass pass
xbu = self.flags[job["ptop"]].get("xbu") vf = self.flags[job["ptop"]]
xbu = vf.get("xbu")
ap_chk = djoin(pdir, job["name"]) ap_chk = djoin(pdir, job["name"])
vp_chk = djoin(job["vtop"], job["prel"], job["name"]) vp_chk = djoin(job["vtop"], job["prel"], job["name"])
if xbu: if xbu:
@ -4935,7 +4936,7 @@ class Up2k(object):
if x: if x:
zvfs = vfs zvfs = vfs
pdir, _, job["name"], (vfs, rem) = x pdir, _, job["name"], (vfs, rem) = x
job["vcfg"] = vfs.flags job["vcfg"] = vf = vfs.flags
job["ptop"] = vfs.realpath job["ptop"] = vfs.realpath
job["vtop"] = vfs.vpath job["vtop"] = vfs.vpath
job["prel"] = rem job["prel"] = rem
@ -4985,8 +4986,13 @@ class Up2k(object):
fs = self.fstab.get(pdir) fs = self.fstab.get(pdir)
if fs == "ok": if fs == "ok":
pass pass
elif "sparse" in self.flags[job["ptop"]]: elif "nosparse" in vf:
t = "volflag 'sparse' is forcing use of sparse files for uploads to [%s]" t = "volflag 'nosparse' is preventing creation of sparse files for uploads to [%s]"
self.log(t % (job["ptop"],))
relabel = True
sprs = False
elif "sparse" in vf:
t = "volflag 'sparse' is forcing creation of sparse files for uploads to [%s]"
self.log(t % (job["ptop"],)) self.log(t % (job["ptop"],))
relabel = True relabel = True
else: else: