mirrors/copyparty
1
0
Fork 0
mirror of https://github.com/9001/copyparty.git synced 2025-08-18 01:22:13 -06:00
Code Issues Actions Packages Projects Releases Wiki Activity
3775 commits 4 branches 344 tags 55 MiB
Commit graph

459 commits

Author SHA1 Message Date
ed e1d78d8b23 increase timeout of unfinished uploads from 6 to 24 hours 
plus make it configurable
 2023-10-20 18:31:28 +00:00
ed 4b5a0787ab option to show upload timestamps in directory listing; 
enable with -mte +.ip_at
or volflag mte=+.ip_at

worst-case performance impact: 18%
 2023-10-17 17:51:27 +00:00
ed dc2b67f155 ui-button to use upload-time instead of local last-modified 2023-10-15 08:46:23 +00:00
ed 753e3cfbaf revert 68c6794d (v1.6.2) and fix it better: 
moving deduplicated files between volumes could drop some links
 2023-10-07 22:25:44 +00:00
ed 0dc3c23b42 add alternative filekey generator; closes #52 2023-10-06 13:41:22 +00:00
ed a62f744a18 prevent losing an out-of-volume index 
if the server is started while an external drive is not mounted,
it would drop the database because all the files are missing
 2023-09-22 23:05:07 +00:00
ed c1efd227b7 fix inconsistent use of symlink mtimes in database; 
on upload, dupes are by default handled by symlinking to the existing
copy on disk, writing the uploader's local mtime into the symlink mtime,
which is also what gets indexed in the db

this worked as intended, however during an -e2dsa rescan on startup the
symlink destination timestamps would be used instead, causing a reindex
and the resulting loss of uploader metadata (ip, timestamp)

will now always use the symlink's mtime;
worst-case 1% slower startup (no dhash)

this change will cause a reindex of incorrectly indexed files, however
as this has already happened at least once due to the bug being fixed,
there will be no additional loss of metadata
 2023-09-01 20:29:55 +00:00
ed 5a38311481 mark offline volumes in directory tree sidebar 2023-08-26 19:00:46 +00:00
ed cafe53c055 v1.9.0 2023-08-20 22:02:40 +00:00
ed 135ece3fbd immediately allow uploading an interrupted and 
deleted incomplete upload to another location
 2023-08-20 19:16:35 +00:00
ed bd3640d256 change to openmetrics 2023-08-20 18:50:14 +00:00
ed fc0405c8f3 add prometheus metrics; closes #49 2023-08-20 17:58:06 +00:00
ed 015b87ee99 performance / cosmetic: 
* js: use .call instead of .bind when possible
* when running without e2d, the message on startup regarding
  unfinished uploads didn't show the correct filesystem path
 2023-08-16 19:32:43 +00:00
ed 406e413594 hint at additional context in exceptions 2023-08-15 01:42:13 +02:00
ed 1441ccee4f v1.8.4 2023-07-18 07:46:22 +00:00
ed 8f59afb159 fix another race (unpost): 
unposting could collide with most other database-related activities,
causing one or the other to fail.
luckily the unprotected query performed by the unpost API happens to be
very cheap, so also the most likely to fail, and would succeed upon a
manual reattempt from the UI.
even in the worst case scenario, there would be no unrecoverable damage
as the next rescan would auto-repair any resulting inconsistencies.
 2023-07-14 15:21:14 +00:00
ed 77f1e51444 fix unlikely race (e2tsr): 
if someone with admin rights refreshes the homepage exactly as the
directory indexer decides to `_drop_caches`, the indexer thread would
die and the up2k instance would become inoperable...
luckily the probability of hitting this by chance is absolutely minimal,
and the worst case scenario is having to restart copyparty if this
happens immediately after startup; there is no risk of database damage
 2023-07-14 15:20:25 +00:00
ed b54b7213a7 more thumbnailer configs available as volflags: 
--th-convt = convt
--th-no-crop = nocrop
--th-size = thsize
 2023-07-11 22:15:37 +00:00
ed e197895c10 support hashed passwords; closes #39 2023-06-25 21:50:33 +00:00
ed 8b0cf2c982 volflags to limit volume size / num files; closes #40 2023-06-19 00:42:45 +00:00
ed e64b87b99b dont hardlink symlinks (they could be relative) 2023-05-12 20:41:09 +00:00
ed b1359f039f linter cleanup 2023-05-07 14:38:30 +00:00
ed 83178d0836 preserve empty folders (closes #23): 
* when deleting files, do not cascade upwards through empty folders
* when moving folders, also move any empty folders inside

the only remaining action which autoremoves empty folders is
files getting deleted as they expire volume lifetimes

also prevents accidentally moving parent folders into subfolders
(even though that actually worked surprisingly well)
 2023-04-29 11:30:43 +00:00
ed cb6de0387d a bit faster 2023-04-26 19:56:27 +00:00
ed 209b49d771 remind sqlite we have indexes 2023-03-30 21:45:58 +00:00
ed e0c4829ec6 verify covers against db instead of fs 2023-03-15 19:48:43 +00:00
ed b0cc396bca v1.6.8 2023-03-12 16:10:07 +00:00
ed 2be2e9a0d8 index folder thumbs in db 2023-03-11 11:43:29 +00:00
ed 974ca773be just to be extra sure 2023-03-09 21:49:29 +00:00
ed c98fff1647 fix chunkpost-handshake race (affects --no-dedup only); 
a handshake arriving in the middle of the final chunk could cause
dupes to become empty -- worst case leading to loss of data
 2023-03-05 19:45:50 +00:00
ed db95e880b2 thats not how it works 2023-02-28 22:19:06 +00:00
ed 1c2c9db8f0 retain upload time (but not ip) on file reindex 2023-02-26 19:09:24 +00:00
ed 7ea183baef let http thread handle upload verification plugins 2023-02-26 19:07:49 +00:00
ed ab87fac6d8 db got the wrong lastmod when linking dupes 2023-02-26 18:52:04 +00:00
ed 1e3b7eee3b dont rmdir volume top on cleanup 2023-02-26 18:28:37 +00:00
ed 4de028fc3b let controlpanel rescan button override lack of e2dsa 2023-02-26 18:27:10 +00:00
ed 604e5dfaaf improve error handling / messages 2023-02-26 18:26:13 +00:00
ed 05e0c2ec9e add xiu (batching hook; runs on idle after uploads) + 
bunch of tweaks/fixes for hooks
 2023-02-26 18:23:32 +00:00
ed 6eba9feffe condense uploads listing on view change 2023-02-14 21:58:15 +00:00
ed 8f61e1568c transcode chiptunes to opus; 
* new audio/MPT formats: apac bonk dfpwm ilbc it itgz itr itz mo3 mod mptm mt2 okt s3gz s3m s3r s3z xm xmgz xmr xmz xpk
* new image/PIL formats: blp dcx emf eps fits flc fli fpx im j2k j2p psd spi wmf
 2023-02-11 11:17:37 +00:00
ed a4b56c74c7 support long filepaths on win7 + misc windows fixes 2023-02-10 18:37:37 +00:00
ed 7e2ff9825e ensure -e2tsr takes effect by ignoring dhash 2023-02-08 22:33:02 +00:00
ed 2b644fa81b don't alias randomized filenames 2023-02-04 13:41:43 +00:00
ed 0aa88590d0 should generalize this somehow 2023-02-02 22:35:13 +00:00
ed e67b695b23 show filekeys in recent-uploads ui 2023-02-02 21:22:51 +00:00
ed a918c285bf up2k-ui: button to randomize upload filenames 2023-02-01 22:26:18 +00:00
ed 1e20eafbe0 volflag to randomize all upload filenames 2023-02-01 21:58:01 +00:00
ed eaa6b3d0be mute some startup noise 2023-01-29 04:33:28 +00:00
ed 68c6794d33 rewrite other symlinks after the actual move; 
fixes volumes where symlinking is disabled
 2023-01-28 01:14:29 +00:00
ed e965b9b9e2 mkdir missing volumes on startup 2023-01-27 21:52:28 +00:00
ed 9b7b9262aa promote dedup control to volflags 2023-01-25 21:46:15 +00:00
ed f8e3e87a52 add event hooks 2023-01-22 23:35:31 +00:00
ed 8a7135cf41 support fat32 time precision, avoiding rescans 
posted from warzaw airport otw to japan
 2022-12-20 22:19:32 +01:00
ed 79aa40fdea cosmetic fixes 2022-12-14 23:12:51 +00:00
ed 447ed5ab37 windows fixes 2022-12-12 21:59:50 +00:00
ed 02ad4bfab2 ensure consistency between db tables 2022-12-10 22:13:21 +00:00
ed 56b73dcc8a up2k: add option to replace existing file 2022-12-10 19:22:16 +00:00
ed 7704b9c8a2 sqlite durability profiles 2022-12-10 10:01:33 +00:00
ed 999b7ae919 safer to merge wal on startup instead 2022-12-09 19:58:13 +00:00
ed 195eb53995 merge wal on shutdown 2022-12-07 23:09:40 +00:00
ed 06fa78f54a windows: set .hist folder hidden 2022-12-07 22:56:30 +00:00
ed fbed322d3b option to skip database syncs entirely 2022-12-07 21:35:04 +00:00
ed 9b0f519e4e switch to wal for ~2x faster uploads 2022-12-07 20:52:17 +00:00
ed 067cc23346 docs + cleanup 2022-12-03 18:58:56 +00:00
ed 89c9f45fd0 add option for cross-volume dedupe 2022-12-02 17:25:37 +00:00
ed 3a800585bc u2cli: server is allowed to reject dupes 2022-11-29 22:09:32 +00:00
ed 0aa7d12704 add option to disable .hist/up2k.snap 2022-11-15 22:16:53 +00:00
ed 4203fc161b misc 2022-10-30 16:31:04 +00:00
ed 54e1c8d261 remove 697 GiB upload filesize limit 2022-10-30 12:51:20 +00:00
ed 73bd2df2c6 more metadata-parser debug options 2022-10-29 21:59:59 +00:00
ed edad3246e0 make pylance happier 2022-10-29 20:40:25 +00:00
ed 4bcd30da6b cleaner daemon instancing 2022-10-23 12:05:44 +02:00
ed c003dfab03 unbold ansi grays 2022-10-19 15:30:17 +02:00
ed 20c6b82bec replace magic numbers with errno.* 2022-10-19 15:21:48 +02:00
ed 118bdcc26e 120x faster folder moves/renames 2022-10-08 19:11:03 +02:00
ed 78fa96f0f4 add unpost sanchk 2022-10-08 18:23:41 +02:00
ed b8a93e74bf fix default upload expiration + ux 2022-10-07 21:38:01 +02:00
ed b299aaed93 fix some cases of deleted files not being forgotten 2022-09-26 22:19:46 +02:00
ed 4f4f106c48 add ability to forget uploads by deleting the files 2022-09-25 14:24:01 +02:00
ed e70e926a40 support up2k uploads from old browsertabs 2022-09-24 14:35:51 +02:00
ed 0b87a4a810 allow setting lifetimes from up2k ui 2022-09-19 23:49:07 +02:00
ed 09cea66aa8 add ability to set lifetime per-file during upload 2022-09-18 13:12:38 +02:00
ed 5e9bc1127d fix windows symlink creation 2022-09-17 13:27:54 +02:00
ed 0006f933a2 hmac uploader-ip when avoiding filename collisions 2022-09-11 08:27:45 +02:00
ed 590c763659 add unforgetti beam 2022-09-07 08:09:32 +02:00
ed 11d1267f8c option to keep files in index when deleted 2022-09-07 01:07:21 +02:00
ed 995cd10df8 bump timeouts for zfs / bursty filesystems 2022-09-04 21:21:54 +02:00
ed c930d8f773 add mtp debug mode 2022-09-03 19:58:10 +02:00
ed d48a7d2398 provide tagparsers with uploader info 2022-08-15 20:23:17 +02:00
ed 7a460de3c2 windows db fix 2022-08-15 18:01:28 +02:00
ed 1c69ccc6cd v1.3.12 2022-08-13 00:58:49 +02:00
ed 9ccd327298 add directory hashing (boots ~3x faster) 2022-08-12 23:17:18 +02:00
ed 11df36f3cf add option to exit after scanning volumes 2022-08-12 21:20:13 +02:00
ed ad18b6e15e stop reindexing empty files on startup 2022-08-12 16:31:36 +02:00
ed 92ed4ba3f8 parallelize python hashing too 2022-08-10 23:12:01 +02:00
ed 4aaa111925 v1.3.9 2022-08-04 00:39:37 +02:00
ed e31248f018 include version info on startup and in crash dumps 2022-08-04 00:11:52 +02:00
ed 4e7455268a tag-scanner perf 2022-08-03 22:33:20 +02:00
ed 680f8ae814 add xdev/xvol indexing guards 2022-08-03 22:20:28 +02:00
ed 90555a4cea clean-shutdown while hashing huge files 2022-08-03 21:06:10 +02:00
ed 5384c2e0f5 reentrant cleanup 2022-08-02 20:56:05 +02:00
ed 0ee574a718 forget uploads that failed to initialize 2022-08-02 06:15:18 +02:00
ed dac2fad48e v1.3.8 2022-07-27 16:07:26 +02:00
ed 77f624b01e improve shumantime + use it everywhere 2022-07-27 15:07:04 +02:00
ed e24ffebfc8 indicate write-activity on splashpage 2022-07-27 14:53:15 +02:00
ed 660705a436 defer volume reindexing on db activity 2022-07-27 11:48:47 +02:00
ed 76adac7c72 up2k-hook-ytid: add mp4/webm/mkv metadata scanner 2022-07-26 22:09:18 +02:00
ed 5dc75ebb67 async e2ts / e2v + forget deleted shadowed 2022-07-26 12:47:40 +02:00
ed d686ce12b6 lsof db on stuck transaction 2022-07-25 02:07:59 +02:00
ed 10430b347f fix dumb prisonparty bug 2022-07-22 20:49:35 +02:00
ed 4a76663fb2 ensure free disk space 2022-07-17 22:33:08 +02:00
ed 3fa377a580 sqlite diag 2022-07-16 20:43:26 +02:00
ed c2a0b1b4c6 autopotato 2022-07-15 02:39:32 +02:00
ed 48b957f1d5 add -e2v (file integrity checker) 2022-07-13 00:48:39 +02:00
ed 3683984c8d abort volume indexing on ^C 2022-07-12 21:46:07 +02:00
ed 6f75b02723 misc 2022-07-12 03:16:30 +02:00
ed d1a03c6d17 zerobyte semantics 2022-07-11 23:17:33 +02:00
ed c6c31702c2 cheaper file deletion 2022-07-11 01:50:18 +02:00
ed c5b04f6fef mtp daisychaining 2022-07-08 22:29:05 +02:00
ed 4792c2770d fix a spin 2022-07-03 02:39:15 +02:00
ed 1c0071d9ce perf 2022-07-03 01:40:30 +02:00
ed 5d3034c231 detect sparse support from st_blocks 2022-06-23 18:23:42 +02:00
ed fe67c52ead configurable list of sparse-supporting filesystems + 
close nonsparse files after each write to force flush
 2022-06-19 22:38:52 +02:00
ed 190e11f7ea update deps + misc 2022-06-16 21:43:40 +02:00
ed 62fbb04c9d allow moving files between filesystems 2022-06-16 18:46:50 +02:00
ed 728dc62d0b optimize nonsparse uploads (fat32, exfat, hpfs) 2022-06-16 17:51:42 +02:00
ed 438384425a add types, isort, errorhandling 2022-06-16 01:07:15 +02:00
ed 4e5a323c62 more cleanup 2022-06-08 01:05:35 +02:00
ed 4c4b3790c7 fix read-spin on d/c during json post + errorhandling 2022-06-07 19:02:52 +02:00
ed bd60b464bb fix misleading log-msg 2022-06-07 14:12:55 +02:00
ed 0143380306 help the query planner 2022-05-13 01:41:39 +02:00
ed 1d9f9f221a louder 2022-05-12 20:55:37 +02:00
ed 2075a8b18c skip nonregular files when indexing filesystem 2022-05-09 19:56:17 +00:00
ed 4878eb2c45 support symlinks as volume root 2022-04-30 20:26:26 +02:00
ed 0d9567575a avoid hashing busy uploads during rescan 2022-04-29 23:16:23 +02:00
ed e82f176289 fix deadlock on rescan during upload 2022-04-29 23:14:51 +02:00
ed 2933dce3ef mtime blank uploads + helptext 2022-04-24 22:58:11 +02:00
ed f35e41baf1 allow unposting with write-only access 2022-04-16 23:35:04 +02:00
ed 36c20bbe53 fix setting mtime on windows 2022-04-15 20:08:55 +02:00
ed cba9e5b669 add hardlinks (symlink alternative) for up2k dedup 2022-04-15 19:13:53 +02:00
ed 951fdb27ca dont scan orphaned volumes 2022-04-14 17:11:51 +02:00
ed f0988c0e32 filter some volflags from up2k dump 2022-04-08 21:56:24 +02:00
ed c10c70c1e5 misc 2022-03-04 21:30:31 +01:00
ed 1044aa071b deal with consecutive dupes even without sqlite 2021-12-06 23:51:44 +01:00
ed 241ef5b99d preserve mtimes when juggling symlinks 2021-12-04 01:58:04 +01:00
ed c9b1c48c72 sizelimit registry + persist without e2d 2021-11-16 21:31:24 +01:00
ed 12b3a62586 fix dumb mistakes 2021-11-15 20:13:16 +01:00
ed b206c5d64e handle multiple simultaneous uploads of the same file 2021-11-14 15:03:11 +01:00
ed 807f272895 missed one 2021-11-06 18:33:32 +01:00
ed f050647b43 rescan volumes on sigusr1 2021-11-06 18:20:31 +01:00
ed f5f9e3ac97 reduce rescan/lifetime wakeups 2021-10-27 22:23:03 +02:00
ed 744727087a better rmtree semantics 2021-10-27 09:40:20 +02:00
ed 6dade82d2c run tag scrapers in parallel on new uploads 2021-10-27 00:47:50 +02:00
ed 6b737bf1d7 abort tagging if the file has poofed 2021-10-27 00:11:58 +02:00
ed e4ca4260bb support mounting entire disks on windows 2021-10-20 00:51:00 +02:00
ed 4f97d7cf8d normalize collision suffix 2021-10-19 00:49:35 +02:00
ed 088899d59f fix unpost in jumpvols 2021-10-18 21:08:31 +02:00
ed 2f021a0c2b skip indexing files by regex 2021-10-12 01:40:19 +02:00
ed d1be37a04a nice 2021-10-09 01:33:27 +02:00
ed b0fd8bf7d4 optimize indexer for huge filesystems 2021-10-09 01:24:19 +02:00
ed a6935b0293 allow uploading empty files 2021-10-02 23:34:12 +02:00
ed 5886a42901 url escaping 2021-09-26 16:59:02 +02:00
ed f10d03c238 add --no-symlink 2021-09-26 13:49:29 +02:00
ed f9a66ffb0e up2k: fully parallelize handshakes/uploads 2021-09-26 12:57:16 +02:00
ed 0bb9154747 catch more tagparser panics 2021-09-26 03:56:30 +02:00
ed 30c3f45072 fix deleting recently uploaded files without e2d 2021-09-26 03:45:16 +02:00
ed 0d5ca67f32 up2k-srv: add option to reuse file-handles 2021-09-26 03:44:22 +02:00
ed 076d32dee5 up2k-srv: try all dupes for matching path 2021-09-24 19:21:19 +02:00
ed 32a1c7c5d5 cosmetic 2021-09-18 02:07:29 +02:00
ed 0e884df486 keep empty folders after deleting all files 2021-09-15 23:31:49 +02:00
ed 02e3d44998 fix move/delete without -e2d (thx exci) 2021-09-07 21:20:34 +02:00
ed 21662c9f3f error-message cleanup 2021-09-07 20:51:07 +02:00
ed ca16c4108d add options to disallow renaming/moving dotfiles 2021-09-06 00:17:35 +02:00
ed 4802f8cf07 better msg when unposting a deleted file 2021-08-24 00:24:50 +02:00
ed 5f42237f2c v0.13.5 2021-08-16 08:40:26 +02:00
ed c1d77e1041 add upload lifetimes 2021-08-09 22:17:41 +02:00
ed 8d755d41e0 per-volume rescan interval 2021-08-09 01:31:20 +02:00
ed ee86b06676 compat + perf + ux 2021-08-08 04:02:58 +02:00
ed 9a45549b66 adding upload rules 2021-08-07 03:45:50 +02:00
ed ae3a01038b v0.12.12 2021-08-06 11:10:04 +02:00
ed 451d757996 fix renaming single symlinks 2021-08-03 20:12:51 +02:00
ed a918cc67eb only drop tags when its safe 2021-07-31 13:19:02 +02:00
ed c164fc58a2 add unpost 2021-07-29 23:53:08 +02:00
ed 0c625a4e62 store upload ip and time 2021-07-29 00:30:10 +02:00
ed 129d33f1a0 mv/del: recursive rmdir 2021-07-27 19:15:58 +02:00
ed 1ad7a3f378 await and monitor workers on startup 2021-07-27 15:48:00 +00:00
ed 7671d791fa rename works + more symlink fixes 2021-07-26 17:44:20 +02:00
ed 980c6fc810 add scheduled rescans + fix mv bugs 2021-07-26 02:34:56 +02:00
ed fb40a484c5 mv(folder) works 2021-07-26 01:26:58 +02:00
ed daa9dedcaa rm works 2021-07-26 00:29:28 +02:00
ed e648252479 mv works (at least in trivial cases) 2021-07-25 21:15:43 +02:00
ed 27cce086c6 fileman ui 2021-07-25 01:09:14 +02:00
ed 05a1a31cab too soon 2021-07-24 22:20:02 +02:00
ed d020527c6f centralize mojibake support stuff 2021-07-24 21:56:55 +02:00
ed 4451485664 mv/rm (serverside), 100% untested 2021-07-24 20:08:31 +02:00
ed a4e1a3738a more deletion progress 2021-07-23 23:42:07 +02:00
ed e3684e25f8 treat symlinks as regular files in db 2021-07-22 19:34:40 +02:00
ed 1359213196 prefer native sqlite3 backup (journal-aware) 2021-07-22 19:10:42 +02:00
ed 8eb8043a3d fix 3rdparty namecase 2021-07-18 22:50:29 +02:00
ed ebc9de02b0 case-insensitive tag search 2021-07-18 21:34:36 +02:00