add rclone optimization, closes #21

This commit is contained in:
ed 2023-04-01 10:21:21 +00:00
parent 6a312ca856
commit 2a37e81bd8
3 changed files with 12 additions and 10 deletions

View file

@ -1206,7 +1206,9 @@ sync folders to/from copyparty
the commandline uploader [up2k.py](https://github.com/9001/copyparty/tree/hovudstraum/bin#up2kpy) with `--dr` is the best way to sync a folder to copyparty; verifies checksums and does files in parallel, and deletes unexpected files on the server after upload has finished which makes file-renames really cheap (it'll rename serverside and skip uploading)
alternatively there is [rclone](./docs/rclone.md) which allows for bidirectional sync and is *way* more flexible (stream files straight from sftp/s3/gcs to copyparty for instance), although syncing to copyparty is about 5x slower than up2k.py if you have many small files in particular
alternatively there is [rclone](./docs/rclone.md) which allows for bidirectional sync and is *way* more flexible (stream files straight from sftp/s3/gcs to copyparty for instance), although there is no integrity check and it won't work with files over 100 MiB if copyparty is behind cloudflare
* starting from rclone v1.63 (currently [in beta](https://beta.rclone.org/?filter=latest)), rclone will also be faster than up2k.py
## mount as drive
@ -1215,11 +1217,10 @@ a remote copyparty server as a local filesystem; go to the control-panel and cl
alternatively, some alternatives roughly sorted by speed (unreproducible benchmark), best first:
* [rclone-http](./docs/rclone.md) (25s), read-only
* [rclone-webdav](./docs/rclone.md) (25s), read/WRITE ((v1.63-beta)[https://beta.rclone.org/?filter=latest])
* [rclone-http](./docs/rclone.md) (26s), read-only
* [partyfuse.py](./bin/#partyfusepy) (35s), read-only
* [rclone-ftp](./docs/rclone.md) (47s), read/WRITE
* [rclone-webdav](./docs/rclone.md) (51s), read/WRITE
* copyparty-1.5.0's webdav server is faster than rclone-1.60.0 (69s)
* [partyfuse.py](./bin/#partyfusepy) (71s), read-only
* davfs2 (103s), read/WRITE, *very fast* on small files
* [win10-webdav](#webdav-server) (138s), read/WRITE
* [win10-smb2](#smb-server) (387s), read/WRITE

View file

@ -43,10 +43,9 @@
<h1>WebDAV</h1>
<div class="os win">
<p><em>note: rclone-FTP is a bit faster, so {% if args.ftp or args.ftps %}try that first{% else %}consider enabling FTP in server settings{% endif %}</em></p>
<p>if you can, install <a href="https://winfsp.dev/rel/">winfsp</a>+<a href="https://downloads.rclone.org/rclone-current-windows-amd64.zip">rclone</a> and then paste this in cmd:</p>
<pre>
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>W:</b>
</pre>
{% if s %}
@ -71,7 +70,7 @@
</pre>
<p>or you can use rclone instead, which is much slower but doesn't require root (plus it keeps lastmodified on upload):</p>
<pre>
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>mp</b>
</pre>
{% if s %}

View file

@ -29,11 +29,13 @@ echo type = webdav
echo vendor = owncloud
echo url = http://127.0.0.1:3923/
echo headers = Cookie,cppwd=hunter2
echo pacer_min_sleep = 0.01ms
echo(
echo [cpp-ro]
echo type = http
echo url = http://127.0.0.1:3923/
echo headers = Cookie,cppwd=hunter2
echo pacer_min_sleep = 0.01ms
) > %userprofile%\.config\rclone\rclone.conf
```
@ -48,11 +50,13 @@ type = webdav
vendor = owncloud
url = http://127.0.0.1:3923/
headers = Cookie,cppwd=hunter2
pacer_min_sleep = 0.01ms
[cpp-ro]
type = http
url = http://127.0.0.1:3923/
headers = Cookie,cppwd=hunter2
pacer_min_sleep = 0.01ms
EOF
```
@ -74,8 +78,6 @@ note that the up2k client [up2k.py](https://github.com/9001/copyparty/tree/hovud
rclone sync /usr/share/icons/ cpp-rw:fds/
```
TODO: rclone bug? `--transfers=4` doesn't seem to do anything (it does one request at a time), doesn't matter if the webdav server is copyparty or rclone
# use rclone as server too, replacing copyparty