Commit graph

48 commits

Author SHA1 Message Date
KevinXuxuxu 9c19753546 [u2c.py] Fix unicode files argument handling for py2.7 2025-08-01 14:39:57 +00:00
ed ecced0c4f2 v1.17.1 2025-05-18 22:34:16 +00:00
ed 261236e302 st_mtime can be -11644473600 on win64 fat16 vhd 2025-05-18 21:34:38 +00:00
ed e9f78ea70c up2k: tristate option for overwriting files; closes #139
adds a third possible value for the `replace` property in handshakes:

* absent or False: never overwrite an existing file on the server,
   and instead generate a new filename to avoid collision

* True: always overwrite existing files on the server

* "mt": only overwrite if client's last-modified is more recent
   (this is the new option)

the new UI button toggles between all three options,
defaulting to never-overwrite
2025-02-19 21:58:56 +00:00
ed 1fe30363c7 u2c: option to print download links 2025-01-27 01:35:36 +00:00
ed 4c04798aa5 u2c: fix hash-calculator mode
it produced the correct chunk-hashes with --chs
but the total file-hash was wrong regardless
2025-01-21 22:04:20 +00:00
ed 946a8c5baa u2c: fix windowtitle 2024-12-19 18:02:29 +00:00
ed 9163780000 u2c: misc windows fixes
* support globbing/wildcards on windows

* add `osc 9;4` to show upload progress in the taskbar
   (currently windows-only; linux is picking it up)

* workaround msys2-terminal not normalizing
   absolute paths which contain whitespace

* show a helpful "now hashing..." while the
   first file is being hashed, since it kinda
   looks like a deadlock on windows otherwise
2024-12-06 18:44:05 +00:00
ed 1a1d731043 misc cleanup:
* u2c: remove superfluous pathsep (harmless)
* tl: new strings
2024-12-02 22:52:39 +00:00
ed b10843d051 cosmetic eta improvements:
* u2c: strip hh:mm:ss past 30 days
* u2js: fix "infini.ty" in elapsed-times
2024-11-23 19:58:25 +00:00
ed c784e5285e u2c: adaptive connection:keepalive expiration 2024-11-10 17:43:40 +00:00
ed 119e88d87b bubble OS-filesystem errors to client
send a 500 or 404 if a folder is inaccessible or does not exist

previously it would return an empty directory listing instead
2024-11-02 17:38:17 +00:00
ed a9b4436cdc up2k: improve upload retry/timeout
* `js:` make handshake retries more aggressive
* `u2c:` reduce chunks timeout + ^
* `main:` reduce tcp timeout to 128sec (js is 42s)
* `httpcli:` less confusing log messages
2024-10-18 16:24:31 +00:00
ed fc8298c468 up2k: avoid cloudflare upload size-limit
previously, the biggest file that could be uploaded through
cloudflare was 383 GiB, due to max num chunks being 4096

`--u2sz`, which takes three ints (min-size, target, max-size)
can now be used to enforce a max chunksize; chunks larger
than max-size gets split into smaller subchunks / chunklets

subchunks cannot be stitched/joined, and subchunks of the
same chunk must be uploaded sequentially, one at a time

if a subchunk fails due to bitflips or connection-loss,
then the entire chunk must (and will) be reuploaded
2024-10-16 19:29:08 +00:00
ed b2770a2087 u2c: support more crazy filenames
newlines, invalid utf8, and worst of all... %20 (whitespace)

due to up2k protocol limitations,
filenames are normalized when they hit the server,
but folders get to keep their intended jank
2024-10-15 23:01:07 +00:00
ed cba1878bb2 u2c: don't get stuck at fifos and such 2024-10-15 22:53:55 +00:00
ed a2e037d6af u2c: fix chunksize calculation
files which were exactly 128 GiB large would fail
(you can't make this shit up)
2024-10-15 22:39:48 +00:00
ed 65a2b6a223 u2c: fix excessive FDs
it would open separate FDs for all chunks to be uploaded...

open and close files as they are needed during upload instead
2024-10-15 22:30:15 +00:00
ed 414de88925 u2c v2.2 2024-10-13 22:07:41 +00:00
ed 53ffd245dd u2c: fix progress indicator for resumed uploads 2024-10-13 22:07:07 +00:00
ed cf1b756206 u2c: option to list chunk hashes 2024-10-13 22:06:02 +00:00
ed b9c5c7bbde u2c: early exclude + fix py2 perf
* don't traverse into excluded folders
* 3.5x faster on python2.7
2024-09-23 17:20:04 +00:00
ed 9daeed923f u2c: remove all deps to become 3x faster on small files
reduces performance on python 2.7, but that is ok

also fixes `unknown encoding: idna` due to cpython race
2024-09-22 18:07:36 +00:00
ed 08848be784 u2c: add hashgen mode + fix shutdown lag 2024-09-06 00:31:25 +00:00
ed f356faa278 u2c: support multiple exclusion patterns 2024-08-22 20:03:25 +00:00
ed 0da719f4c2 up2k: shrink request headers
v1.13.5 made some proxies angry with its massive chunklists

when stitching chunks, only list the first chunk hash in full,
and include a truncated hash for the consecutive chunks

should be enough for logfiles to make sense
and to smoketest that clients are behaving
2024-08-08 18:24:18 +00:00
ed 8916bce306 u2c fixes:
* `--sz` was num.chunks, not the intended MiB
* crash on exit with `-z` and no modified files
* summary upload elapsed-time could exceed wallclock
2024-07-26 19:28:47 +00:00
ed 987bce2182 u2c fixes:
* don't stitch across deduplicated blocks
* print speed/time for hash/upload
* more compact json in handshakes
2024-07-22 20:55:32 +00:00
ed 132a83501e add chunk stitching; twice as fast long-distance uploads:
rather than sending each file chunk as a separate HTTP request,
sibling chunks will now be fused together into larger HTTP POSTs
which results in unreasonably huge speed boosts on some routes
( `2.6x` from Norway to US-East,  `1.6x` from US-West to Finland )

the `x-up2k-hash` request header now takes a comma-separated list
of chunk hashes, which must all be sibling chunks, resulting in
one large consecutive range of file data as the post body

a new global-option `--u2sz`, default `1,64,96`, sets the target
request size as 64 MiB, allowing the settings ui to specify any
value between 1 and 96 MiB, which is cloudflare's max value

this does not cause any issues for resumable uploads; thanks to the
streaming HTTP POST parser, each chunk will be verified and written
to disk as they arrive, meaning only the untransmitted chunks will
have to be resent in the event of a connection drop -- of course
assuming there are no misconfigured WAFs or caching-proxies

the previous up2k approach of uploading each chunk in a separate HTTP
POST was inefficient in many real-world scenarios, mainly due to TCP
window-scaling behaving erratically in some IXPs / along some routes

a particular link from Norway to Virginia,US is unusably slow for
the first 4 MiB, only reaching optimal speeds after 100 MiB, and
then immediately resets the scale when the request has been sent;
connection reuse does not help in this case

on this route, the basic-uploader was somehow faster than up2k
with 6 parallel uploads; only time i've seen this
2024-07-21 23:35:37 +00:00
ed bef9617638 u2c.exe: explain that https is disabled 2024-06-01 22:26:47 +00:00
ed 87c60a1ec9 ensure OS signals hit main-thread as intended;
use sigmasks to block SIGINT, SIGTERM, SIGUSR1 from all other threads

also initiate shutdown by calling sighandler directly,
in case this misses anything and that is still unreliable
(discovered by `--exit=idx` being noop once in a blue moon)
2024-05-09 22:28:16 +00:00
ed 439cb7f85b u2c: add --ow (previously part of --dr) 2024-04-20 16:36:10 +00:00
ed fd552842d4 fix other possible division-by-zeros;
u2c: also fix exe detection
2024-02-17 23:19:11 +00:00
ed b2e233403d u2c: apply exclude-filter to deletion too
if a file gets synced and you later add an exclude-filter for it,
delete the file from the server as if it doesn't exist locally
2024-01-27 18:49:25 +00:00
ed 55eb692134 up2k: add option to touch existing files to match local 2024-01-24 20:36:41 +00:00
ed 86419b8f47 suboptimizations and some future safeguards 2024-01-10 23:20:42 +01:00
ed b4031e8d43 forgot to bump this... oh well, at least the exe is correct 2023-12-08 02:16:40 +00:00
ed 2077dca66f u2c: when deleting from server, heed request size limit 2023-12-08 00:54:57 +00:00
ed 8c52b88767 make linters happier 2023-11-30 17:33:07 +00:00
ed c6744dc483 u2c: configurable retry delay 2023-11-11 14:46:00 +00:00
ed b9997d677d u2c: give up on files with bitflips 2023-11-11 14:30:46 +00:00
ed 10defe6aef u2c: make -x case-insensitive 2023-11-11 14:02:01 +00:00
ed fcc3336760 v1.9.8 2023-10-06 17:50:35 +00:00
ed 14497b2425 docs:
* mention cloudflare-specific nginx config

versus.md:
* seafile has a size limit on zip downloads
* seafile and nextcloud are slow at uploading many small files

u2c: improve error message in funky environments
2023-08-25 21:57:26 +00:00
ed 033b50ae1b u2c: exclude files by regex 2023-08-15 00:45:12 +02:00
ed deca082623 v1.7.1 2023-05-07 18:34:39 +00:00
ed 4295923b76 rename up2k.py (client) to u2c.py 2023-05-07 15:37:52 +00:00
ed 572aa4b26c rename up2k.py (client) to u2c.py 2023-05-07 15:35:56 +00:00
Renamed from bin/up2k.py (Browse further)