LemmyNet
/
pict-rs
mirror of https://git.asonix.dog/asonix/pict-rs
2
0
Fork 0
Code Releases Activity
685 Commits 24 Branches 312 Tags 14 MiB
Commit Graph

294 Commits

Author SHA1 Message Date
asonix 85c5ce955d Clippy 2022-09-30 20:02:46 -05:00
asonix d86d9a3228 Put proper file extension on videos 2022-09-30 20:00:14 -05:00
asonix 8eb2293808 Enable selecting video and audio codecs for uploaded media 2022-09-30 19:38:11 -05:00
asonix f4542efcc1 Share ffprobe details code 2022-09-30 17:43:40 -05:00
asonix 3ceaaa391c Update clap 2022-09-28 18:23:41 -05:00
asonix 656b0225af Clippy 2022-09-27 23:23:06 -05:00
asonix 29cab025c6 Improve 0.3 migration code, repo traces 2022-09-27 23:19:52 -05:00
asonix 85f20bbe0e Add new resize options, update README with current APIs 2022-09-26 18:14:25 -05:00
asonix 9db057fdc5 Add admin endpoint to fetch identifier from an alias 2022-09-25 20:59:57 -05:00
asonix 17dab63662 Clippy 2022-09-25 20:44:24 -05:00
asonix 80c83eb491 Add ffprobe for details inspection - vastly improve video detection speed 2022-09-25 20:39:09 -05:00
asonix 5449bb82f1 Better identify video uploads with ffprobe 2022-09-25 19:34:51 -05:00
asonix 21e3e63ac3 Instrument upload store methods 2022-09-25 18:46:50 -05:00
asonix 51d8621015 Defensively format errors in traces to avoid deadlocks 2022-09-25 18:46:26 -05:00
asonix 890478e794 Support audio in uploaded videos, allow webm uploads 2022-09-25 18:16:37 -05:00
asonix c57a48db8a Start introducing more video controls 2022-09-25 17:36:07 -05:00
asonix c9a74a73ca Reduce redundancy in process module 2022-09-25 17:35:52 -05:00
asonix 50d118a2a7 Implement image pre-processing on upload 2022-09-25 15:17:33 -05:00
asonix 04bc586a26 Fix hasher test 2022-09-25 09:09:05 -05:00
asonix dc95417846 Improve concurrency for upload streams part 2 2022-09-25 08:27:11 -05:00
asonix bf3c47e457 Improve concurrency for upload streams 2022-09-24 22:07:06 -05:00
asonix a6adde874e Implement s3 multipart uploads 2022-09-24 20:33:59 -05:00
asonix 25209e29c0 Make it compile 2022-09-24 17:18:53 -05:00
asonix ff1771e016 More migration work for rusty-s3 2022-09-24 14:18:49 -05:00
asonix ab7fd9aaf7 Start work on using rusty-s3 instead of rust-s3 2022-09-24 13:39:27 -05:00
asonix c5b107eb4c clippy 2022-09-11 10:04:37 -05:00
asonix 7f49af58fe Update actix-form-data, impl Store & Repo for Data<T> 2022-09-10 19:29:22 -05:00
asonix e0bc95f785 Remove unneeded reborrow 2022-08-15 15:00:12 -05:00
Aode (lion) 547934071a Don't migrate from the old database if it doesn't exist 2022-08-14 11:49:20 -05:00
Aode (lion) e0543b2c1c Add HEAD endpoints for process, serve 2022-07-17 11:15:43 -05:00
asonix cb9a594f24 Fix content type on already processed images 2022-06-21 11:27:22 -05:00
asonix dc73bdbad4 Make already-claimed a 400 2022-06-05 23:24:58 -05:00
Aode (lion) 2db76ad8c5 Update dependencies 2022-04-29 15:54:54 -05:00
Aode (lion) 8ee26ccf4f change some import paths 2022-04-27 20:13:49 -05:00
Aode (Lion) 4ac20546ce Improve migration from 0.3 formats 2022-04-24 17:10:15 -05:00
Aode (Lion) b3cbda1337 Add internal endpoint for cleaning variants 2022-04-11 16:56:39 -05:00
Aode (Lion) e50ccfb48c Update deps 2022-04-11 15:17:54 -05:00
Aode (lion) 05533f7e3a Use rfc3339 dates for details responses 2022-04-08 13:36:06 -05:00
Aode (lion) 7436f15267 Dry download client logic 2022-04-08 13:16:22 -05:00
Aode (lion) 5f19b63166 Extract details processing 2022-04-08 13:03:00 -05:00
Aode (lion) 31c5a36c77 Better instrument drops, jobs. Properly disarm backgrounded downloads 2022-04-08 12:51:33 -05:00
Aode (lion) 55d5e43cd2 Consolidate endpoints for downloading media, cached or backgrounded 2022-04-08 12:05:14 -05:00
Aode (lion) 07c61cbe87 Add download backgrounded 2022-04-07 19:07:30 -05:00
Aode (lion) 8781bc8f28 Add follows-from relation for background tasks 2022-04-07 13:28:28 -05:00
Aode (lion) e493e90dd4 Defensive tracing 2022-04-07 12:56:40 -05:00
Aode (lion) c80d207a87 Improve tracing 2022-04-06 21:40:49 -05:00
Aode (Lion) 05e2cf5e08 Cache on Alias rather than Hash 
Make sled repo blocking tasks share parent span
 2022-04-06 12:14:12 -05:00
Aode (Lion) 865f4ee09c Caching works: 
- Insert new bucket for current access time before removing
    previous access time record.
- Add instrument to more methods for more precise tracing
 2022-04-05 21:47:35 -05:00
Aode (Lion) 6cdae7b318 Add 'cache' endpoint for ingesting ephemeral media 
By default, cached media should only stick around for 7 days, however
The timeout is reset every time media is accessed, so only obscure
cached media will be flushed from the cache. This '7 days' number is
configurable through the commandline run options as --media-cache-duration
and in the pict-rs.toml file as [media] cache_duration
 2022-04-05 20:29:30 -05:00
Aode (lion) b0670c2f66 Move migrate to old 2022-04-03 15:07:31 -05:00
Aode (lion) 409b7d4c54 Remove unused conversions 2022-04-03 12:51:21 -05:00
Aode (lion) 29f0774331 Add ability to claim uploads 2022-04-02 21:15:39 -05:00
Aode (lion) 8734dfbdc7 Add backgrounding routes 
- Accept backgrounded uploads
- Allow backgrounded processing

Still TODO:
- Endpoint for waiting on/claiming an upload
 2022-04-02 20:56:29 -05:00
Aode (Lion) c4d014597e Better job recovery 2022-04-02 18:53:03 -05:00
Aode (Lion) 77a400c7ca Implement UploadRepo 2022-04-02 17:41:00 -05:00
Aode (Lion) 132e395e5c Multiple items: 
- Reduce duplicate work in generate job
- Use hash instead of identifier for unique processing
- Move motion ident generation behind concurrent processor lock
 2022-04-02 17:40:08 -05:00
Aode (Lion) 09f53b9ce6 Move away from UploadManager to direct repo & store actions 2022-04-02 16:44:03 -05:00
Aode (lion) 6ed592c432 In Progress: process jobs 
- Is this a good idea? it definitely will make interacting with pict-rs harder. Maybe it's best not to do this
 2022-04-01 16:51:12 -05:00
Aode (lion) c0d8e0e8e3 Support multiple queues for job processor 2022-04-01 11:51:46 -05:00
Aode (lion) 09281d9ae8 Move stream timeout to all response bodies 2022-03-29 16:48:26 -05:00
Aode (lion) 63d66050c8 Enable buffering iterator items for iterstream 2022-03-29 16:18:00 -05:00
Aode (lion) 5adb3fde89 Move some Stream implementations into 'stream' module 2022-03-29 15:59:17 -05:00
Aode (lion) 0e490ff54a Remove custom BytesFreezer stream 2022-03-29 14:28:08 -05:00
Aode (lion) 25f4480809 Remove custom stream error mapping types 2022-03-29 14:17:20 -05:00
Aode (lion) 8226a3571d Enable multi-threaded job processing 2022-03-29 13:18:47 -05:00
Aode (lion) 602d1ea935 Begin implementing queue, put cleanups in it 2022-03-29 12:51:16 -05:00
Aode (lion) e25a4781a8 Remove unneeded variable 2022-03-29 11:09:23 -05:00
Aode (lion) 99d042f36e Allow disabling gif/mp4 2022-03-29 11:04:56 -05:00
Aode (Lion) eb5e39c634 Use color-eyre 2022-03-28 20:47:46 -05:00
Aode (lion) 1291bf8beb Divorce reqwest from main application's spans 2022-03-28 18:42:22 -05:00
Aode (lion) 1662f153ce Numerous changes: 
- Improve error printing (display chain in addition to spantrace)
- Fix migration (read main identifier from identifier tree, not filename tree)
- Ensure uniqueness for processed images in ConcurrentProcessor (use source identifier in addition to thumbnail path, include extension in thumbnail path)
- Update default log levels (make pict-rs quieter)
- Add timeout for serving images from object storage (5 seconds)
 2022-03-28 15:34:36 -05:00
Aode (Lion) 3792a8923a Finish redoing config probably 2022-03-27 23:27:07 -05:00
Aode (lion) ca28f68ef5 Start redoing config again 2022-03-27 19:10:06 -05:00
Aode (lion) 37e6b21b55 Fix Range, consolidate errors, test object storage 2022-03-26 20:45:12 -05:00
Aode (lion) 15b52ba6ec Get it compiling again 2022-03-26 16:49:23 -05:00
Aode (lion) 323016f994 Build out migration path 2022-03-25 18:47:50 -05:00
Aode (lion) 750ce4782e Rework configuration 2022-03-24 22:06:29 -05:00
Aode (lion) d6567fbbbd Implement sled repo 2022-03-24 17:09:15 -05:00
Aode (lion) a0c99d05eb Create initial trait representation for database 2022-03-24 13:16:41 -05:00
Aode (Lion) 2ad536db17 Promote console from compile flag to runtime flag 2022-03-21 22:41:51 -05:00
Aode (Lion) f13f870a92 Only use console config when console feature is enabled 2022-03-21 22:05:32 -05:00
Aode (Lion) 62615be371 Allow configuring console buffer capacity 2022-03-21 21:56:19 -05:00
Aode (Lion) d30415bb49 Don't debug PathStore 2022-03-21 21:43:38 -05:00
Aode (Lion) 9a8dc0e44f Fix sled cache capacity help text 2022-03-21 21:18:51 -05:00
Aode (Lion) da159c69cb Try setting a lower default cache capacity 2022-03-21 21:15:43 -05:00
Aode (lion) 21b6e1dc08 Use actix-web's Range header 2022-03-01 11:23:15 -06:00
Aode (Lion) 73137ede21 Release candidate 1 2022-02-26 12:22:30 -06:00
Aode (lion) 0e948a2bf5 Add flag for console 2022-02-01 10:10:52 -06:00
Aode (Lion) 445c99dbf1 Re-use existing BytesMut when streaming file 2022-01-13 18:11:46 -06:00
Aode (lion) f79d38df8d Update rust-runner image, improve tokio-console support 2021-12-19 12:59:51 -06:00
Aode (lion) c48808c1e1 Update deps 2021-12-18 20:24:22 -06:00
Aode (lion) 44d171252b console 2021-12-16 20:46:46 -06:00
Aode (lion) c445615c47 Update to latest betas 2021-12-12 14:48:37 -06:00
Aode (lion) e209a282d9 No more follows-from for process 2021-12-06 21:11:43 -06:00
Aode (lion) 11739cd42c Remove fancy span following code 2021-12-06 20:35:22 -06:00
Aode (lion) a47f2a3428 Log on new span, no pretty 2021-12-06 18:48:07 -06:00
Aode (lion) 920f1f154c Improve follows-from and parent span relationships 2021-12-06 12:43:03 -06:00
Aode (lion) 42aed5165c Update tracing-awc, add drone 2021-12-03 14:21:31 -06:00
Aode (lion) 7a3d1bef5a Only log closing spans, not opening spans 2021-11-24 21:54:27 -06:00
Aode (lion) 5f5c1d3530 Merge branch 'main' of git.asonix.dog:asonix/pict-rs into main 2021-11-23 16:31:19 -06:00