fe2a499110
Fix sending upload parts (set size), optimize small file uploads
2022-10-01 13:00:07 -05:00
0307dc5a3b
Introduce BytesStream to simplify aggregating, forwarding bytes
2022-10-01 12:06:33 -05:00
c767804430
Default to vp9 for uploaded video
2022-10-01 10:17:32 -05:00
85c5ce955d
Clippy
2022-09-30 20:02:46 -05:00
d86d9a3228
Put proper file extension on videos
2022-09-30 20:00:14 -05:00
8eb2293808
Enable selecting video and audio codecs for uploaded media
2022-09-30 19:38:11 -05:00
f4542efcc1
Share ffprobe details code
2022-09-30 17:43:40 -05:00
3ceaaa391c
Update clap
2022-09-28 18:23:41 -05:00
656b0225af
Clippy
2022-09-27 23:23:06 -05:00
29cab025c6
Improve 0.3 migration code, repo traces
2022-09-27 23:19:52 -05:00
85f20bbe0e
Add new resize options, update README with current APIs
2022-09-26 18:14:25 -05:00
9db057fdc5
Add admin endpoint to fetch identifier from an alias
2022-09-25 20:59:57 -05:00
17dab63662
Clippy
2022-09-25 20:44:24 -05:00
80c83eb491
Add ffprobe for details inspection - vastly improve video detection speed
2022-09-25 20:39:09 -05:00
5449bb82f1
Better identify video uploads with ffprobe
2022-09-25 19:34:51 -05:00
21e3e63ac3
Instrument upload store methods
2022-09-25 18:46:50 -05:00
51d8621015
Defensively format errors in traces to avoid deadlocks
2022-09-25 18:46:26 -05:00
890478e794
Support audio in uploaded videos, allow webm uploads
2022-09-25 18:16:37 -05:00
c57a48db8a
Start introducing more video controls
2022-09-25 17:36:07 -05:00
c9a74a73ca
Reduce redundancy in process module
2022-09-25 17:35:52 -05:00
50d118a2a7
Implement image pre-processing on upload
2022-09-25 15:17:33 -05:00
04bc586a26
Fix hasher test
2022-09-25 09:09:05 -05:00
dc95417846
Improve concurrency for upload streams part 2
2022-09-25 08:27:11 -05:00
bf3c47e457
Improve concurrency for upload streams
2022-09-24 22:07:06 -05:00
a6adde874e
Implement s3 multipart uploads
2022-09-24 20:33:59 -05:00
25209e29c0
Make it compile
2022-09-24 17:18:53 -05:00
ff1771e016
More migration work for rusty-s3
2022-09-24 14:18:49 -05:00
ab7fd9aaf7
Start work on using rusty-s3 instead of rust-s3
2022-09-24 13:39:27 -05:00
c5b107eb4c
clippy
2022-09-11 10:04:37 -05:00
7f49af58fe
Update actix-form-data, impl Store & Repo for Data<T>
2022-09-10 19:29:22 -05:00
e0bc95f785
Remove unneeded reborrow
2022-08-15 15:00:12 -05:00
547934071a
Don't migrate from the old database if it doesn't exist
2022-08-14 11:49:20 -05:00
e0543b2c1c
Add HEAD endpoints for process, serve
2022-07-17 11:15:43 -05:00
cb9a594f24
Fix content type on already processed images
2022-06-21 11:27:22 -05:00
dc73bdbad4
Make already-claimed a 400
2022-06-05 23:24:58 -05:00
2db76ad8c5
Update dependencies
2022-04-29 15:54:54 -05:00
8ee26ccf4f
change some import paths
2022-04-27 20:13:49 -05:00
4ac20546ce
Improve migration from 0.3 formats
2022-04-24 17:10:15 -05:00
b3cbda1337
Add internal endpoint for cleaning variants
2022-04-11 16:56:39 -05:00
e50ccfb48c
Update deps
2022-04-11 15:17:54 -05:00
05533f7e3a
Use rfc3339 dates for details responses
2022-04-08 13:36:06 -05:00
7436f15267
Dry download client logic
2022-04-08 13:16:22 -05:00
5f19b63166
Extract details processing
2022-04-08 13:03:00 -05:00
31c5a36c77
Better instrument drops, jobs. Properly disarm backgrounded downloads
2022-04-08 12:51:33 -05:00
55d5e43cd2
Consolidate endpoints for downloading media, cached or backgrounded
2022-04-08 12:05:14 -05:00
07c61cbe87
Add download backgrounded
2022-04-07 19:07:30 -05:00
8781bc8f28
Add follows-from relation for background tasks
2022-04-07 13:28:28 -05:00
e493e90dd4
Defensive tracing
2022-04-07 12:56:40 -05:00
c80d207a87
Improve tracing
2022-04-06 21:40:49 -05:00
05e2cf5e08
Cache on Alias rather than Hash
...
Make sled repo blocking tasks share parent span
2022-04-06 12:14:12 -05:00
865f4ee09c
Caching works:
...
- Insert new bucket for current access time before removing
previous access time record.
- Add instrument to more methods for more precise tracing
2022-04-05 21:47:35 -05:00
6cdae7b318
Add 'cache' endpoint for ingesting ephemeral media
...
By default, cached media should only stick around for 7 days, however
The timeout is reset every time media is accessed, so only obscure
cached media will be flushed from the cache. This '7 days' number is
configurable through the commandline run options as --media-cache-duration
and in the pict-rs.toml file as [media] cache_duration
2022-04-05 20:29:30 -05:00
b0670c2f66
Move migrate to old
2022-04-03 15:07:31 -05:00
409b7d4c54
Remove unused conversions
2022-04-03 12:51:21 -05:00
29f0774331
Add ability to claim uploads
2022-04-02 21:15:39 -05:00
8734dfbdc7
Add backgrounding routes
...
- Accept backgrounded uploads
- Allow backgrounded processing
Still TODO:
- Endpoint for waiting on/claiming an upload
2022-04-02 20:56:29 -05:00
c4d014597e
Better job recovery
2022-04-02 18:53:03 -05:00
77a400c7ca
Implement UploadRepo
2022-04-02 17:41:00 -05:00
132e395e5c
Multiple items:
...
- Reduce duplicate work in generate job
- Use hash instead of identifier for unique processing
- Move motion ident generation behind concurrent processor lock
2022-04-02 17:40:08 -05:00
09f53b9ce6
Move away from UploadManager to direct repo & store actions
2022-04-02 16:44:03 -05:00
6ed592c432
In Progress: process jobs
...
- Is this a good idea? it definitely will make interacting with pict-rs harder. Maybe it's best not to do this
2022-04-01 16:51:12 -05:00
c0d8e0e8e3
Support multiple queues for job processor
2022-04-01 11:51:46 -05:00
09281d9ae8
Move stream timeout to all response bodies
2022-03-29 16:48:26 -05:00
63d66050c8
Enable buffering iterator items for iterstream
2022-03-29 16:18:00 -05:00
5adb3fde89
Move some Stream implementations into 'stream' module
2022-03-29 15:59:17 -05:00
0e490ff54a
Remove custom BytesFreezer stream
2022-03-29 14:28:08 -05:00
25f4480809
Remove custom stream error mapping types
2022-03-29 14:17:20 -05:00
8226a3571d
Enable multi-threaded job processing
2022-03-29 13:18:47 -05:00
602d1ea935
Begin implementing queue, put cleanups in it
2022-03-29 12:51:16 -05:00
e25a4781a8
Remove unneeded variable
2022-03-29 11:09:23 -05:00
99d042f36e
Allow disabling gif/mp4
2022-03-29 11:04:56 -05:00
eb5e39c634
Use color-eyre
2022-03-28 20:47:46 -05:00
1291bf8beb
Divorce reqwest from main application's spans
2022-03-28 18:42:22 -05:00
1662f153ce
Numerous changes:
...
- Improve error printing (display chain in addition to spantrace)
- Fix migration (read main identifier from identifier tree, not filename tree)
- Ensure uniqueness for processed images in ConcurrentProcessor (use source identifier in addition to thumbnail path, include extension in thumbnail path)
- Update default log levels (make pict-rs quieter)
- Add timeout for serving images from object storage (5 seconds)
2022-03-28 15:34:36 -05:00
3792a8923a
Finish redoing config probably
2022-03-27 23:27:07 -05:00
ca28f68ef5
Start redoing config again
2022-03-27 19:10:06 -05:00
37e6b21b55
Fix Range, consolidate errors, test object storage
2022-03-26 20:45:12 -05:00
15b52ba6ec
Get it compiling again
2022-03-26 16:49:23 -05:00
323016f994
Build out migration path
2022-03-25 18:47:50 -05:00
750ce4782e
Rework configuration
2022-03-24 22:06:29 -05:00
d6567fbbbd
Implement sled repo
2022-03-24 17:09:15 -05:00
a0c99d05eb
Create initial trait representation for database
2022-03-24 13:16:41 -05:00
2ad536db17
Promote console from compile flag to runtime flag
2022-03-21 22:41:51 -05:00
f13f870a92
Only use console config when console feature is enabled
2022-03-21 22:05:32 -05:00
62615be371
Allow configuring console buffer capacity
2022-03-21 21:56:19 -05:00
d30415bb49
Don't debug PathStore
2022-03-21 21:43:38 -05:00
9a8dc0e44f
Fix sled cache capacity help text
2022-03-21 21:18:51 -05:00
da159c69cb
Try setting a lower default cache capacity
2022-03-21 21:15:43 -05:00
21b6e1dc08
Use actix-web's Range header
2022-03-01 11:23:15 -06:00
73137ede21
Release candidate 1
2022-02-26 12:22:30 -06:00
0e948a2bf5
Add flag for console
2022-02-01 10:10:52 -06:00
445c99dbf1
Re-use existing BytesMut when streaming file
2022-01-13 18:11:46 -06:00
f79d38df8d
Update rust-runner image, improve tokio-console support
2021-12-19 12:59:51 -06:00
c48808c1e1
Update deps
2021-12-18 20:24:22 -06:00
44d171252b
console
2021-12-16 20:46:46 -06:00
c445615c47
Update to latest betas
2021-12-12 14:48:37 -06:00
e209a282d9
No more follows-from for process
2021-12-06 21:11:43 -06:00
11739cd42c
Remove fancy span following code
2021-12-06 20:35:22 -06:00
a47f2a3428
Log on new span, no pretty
2021-12-06 18:48:07 -06:00
920f1f154c
Improve follows-from and parent span relationships
2021-12-06 12:43:03 -06:00
42aed5165c
Update tracing-awc, add drone
2021-12-03 14:21:31 -06:00
7a3d1bef5a
Only log closing spans, not opening spans
2021-11-24 21:54:27 -06:00
5f5c1d3530
Merge branch 'main' of git.asonix.dog:asonix/pict-rs into main
2021-11-23 16:31:19 -06:00
5b44de2070
clippy
2021-11-23 16:31:15 -06:00
79afd0e8ea
Update deps
2021-11-23 12:29:42 -06:00
0d153ca239
Make stores take filename into account when saving files
2021-11-16 09:45:16 -06:00
c5d6610276
Properly handle out-of-bounds range requests
2021-11-03 12:04:59 -05:00
4c8da2b414
Keep a reqwest client around for sharing an object storage connection pool
2021-11-02 17:21:00 -05:00
ad7c323985
Update readme, cleanup migration key on completion
2021-11-01 10:44:49 -05:00
93c268fc58
Flush at every migrated file
2021-11-01 10:38:11 -05:00
ccb9f49b8b
Implement storage migration
2021-10-31 21:11:35 -05:00
cf50da30ec
Use rust-s3 fork with asyncwrite support
2021-10-31 12:35:11 -05:00
76e3d37009
Clean up ObjectStore's Debug impl
2021-10-28 21:07:31 -05:00
366db1e1fe
Make object storage with minio work
2021-10-28 20:59:11 -05:00
657152e1a8
Don't println
2021-10-28 00:18:18 -05:00
f596edc363
Improve configuration documentation
2021-10-28 00:17:37 -05:00
f9f4fc63d6
Optionally support s3-compatible storage (untested)
2021-10-27 23:06:03 -05:00
26a2401027
Use tmp files for mp4s
2021-10-23 14:14:12 -05:00
6f04595c3b
Remove unneeded mime conversions
2021-10-23 12:35:07 -05:00
36b83f114a
Merge branch 'main' into asonix/generic-pictrs-storage
2021-10-22 23:49:47 -05:00
48557bc2ea
Make pict-rs generic over file storage
2021-10-22 23:48:56 -05:00
2295837ea4
2021
2021-10-21 16:36:31 -05:00
f67aeb92fa
Keep better track of gif/mp4 thumbnail
2021-10-20 21:36:18 -05:00
1767942e31
Remove recursion
2021-10-20 20:36:10 -05:00
d69d261999
Make it actually compilable
2021-10-20 20:30:21 -05:00
4aefc0403a
Remove more boxes
2021-10-20 20:13:39 -05:00
877390878b
Remove more boxes
2021-10-20 19:28:40 -05:00
e7c3e2c96c
Rename stream module
2021-10-20 19:00:41 -05:00
e7b4e4d1cc
Remove a few boxes
2021-10-20 18:58:32 -05:00
922200673e
Update to latest betas
2021-10-20 18:16:53 -05:00
9559a8b202
Update cli flags
2021-10-18 23:51:04 -05:00
3fe66129be
Update opentelemetry url info
2021-10-18 23:44:56 -05:00
34038a0f16
Fix restructure, upload
2021-10-18 23:37:11 -05:00
195f614a4b
Restructure attempt 1
2021-10-18 20:29:06 -05:00
0f3be6566e
Split upload_manager files
2021-10-18 18:44:05 -05:00
2fb8d3e39a
Reduce required Unpin bounds with pin-project-lite
2021-10-18 18:02:33 -05:00
4c5482b3a8
Add more instrumentation
2021-10-13 20:31:40 -05:00
786d8469ee
It works
2021-10-13 19:50:07 -05:00
09cb2a53b0
Rewrite to avoid direct AsyncX impls
2021-10-13 19:06:53 -05:00
2318fd9dca
Enable io-uring
2021-10-12 23:16:31 -05:00
23e6657208
Re-enable skipping validation on import, strip when converting to prescribed format
2021-09-26 11:02:19 -05:00
15dc79d10e
Instrument a couple more functions
2021-09-25 17:09:55 -05:00
f390c62cbf
Wrap BytesFreezer in span
2021-09-25 16:39:10 -05:00
ba68bcbde3
Add better span information to commands, spawned tasks
2021-09-25 15:23:05 -05:00
7cb6a15acd
Wire up cross-application propagation
2021-09-19 14:36:53 -05:00
79a0fa32ee
Remove tracing-error workaround
2021-09-18 16:34:23 -05:00
5e9d923d03
Instrument for OpenTelemetry
2021-09-18 16:29:30 -05:00
06e402ec72
Don't enter across await points
2021-09-16 17:51:20 -05:00
acba660c37
Update to latest actix-form-data
2021-09-13 20:50:51 -05:00
5d3e6f50b3
Instrument better with Tracing
2021-09-13 20:22:42 -05:00
79bd562d8e
Clippy lints
2021-09-12 10:42:44 -05:00
68e4f1f4c8
Make uploads cancel-safe
2021-09-11 19:53:26 -05:00
bf1a16d7d3
Abort process writer task if reader is dropped
2021-09-11 16:35:38 -05:00
0ffd66c74e
Add and document request deadlines
2021-09-11 15:31:00 -05:00
c1b2c7706a
Add back range header support for thumbnail on generation
2021-09-09 16:35:07 -05:00
45606f4579
Add application-level dimension limiting, bail on failed processes
2021-09-09 14:16:12 -05:00
a0f7148924
Cargo fmt
2021-09-06 12:09:25 -05:00
0d5ce07b20
Convert OnceCell to Lazy, fmt
2021-09-06 12:08:13 -05:00
ac79a9d5cd
Use futures-util instead of hand-implemented types
2021-09-05 20:00:31 -05:00
80044616f9
Make CancelSafeProcessor impl more concise, format code
2021-09-05 19:05:58 -05:00
810831ca73
Don't process images that are already being processed
2021-09-05 17:15:30 -05:00
e88b9a024a
Add semaphore around process operations
2021-09-04 16:01:48 -05:00
e21fd29c09
Drop unneeded dependencies
2021-09-04 14:20:31 -05:00
08c3169d3f
Fewer streams
2021-09-04 12:42:40 -05:00
1119ed740e
WIP: Try to pass AsyncRead types around instead of Streams
2021-09-03 19:53:53 -05:00
43f94a3d5d
Fix imagemagick policy for gifs
2021-08-31 12:14:23 -05:00
ee215bfa5e
Version v0.3.0-alpha.21-in-mem
2021-08-31 11:02:30 -05:00
a6f2082b37
Spawn new task to avoid complexity/bugs
2021-08-31 10:17:08 -05:00
c1d4e3b87e
Start work on piping bytes around from memory instead of going to disk and back
2021-08-30 21:19:47 -05:00
3578303104
Version v0.3.0-alpha.19
2021-08-29 14:56:28 -05:00
18c6481fff
Print outfile when converting / processing
2021-08-29 14:16:55 -05:00
3066e4d350
Use safe_move_file to create thumbnail image
2021-08-29 12:12:22 -05:00
98ec2c5a90
Magick on tmp files, allow tmp files in policy
2021-08-29 11:51:56 -05:00
91207e1a6f
Version v0.3.0-alpha.15
2021-08-29 00:05:02 -05:00
7c371c182a
Version v0.3.0-alpha.14
2021-08-28 23:15:29 -05:00
db95c85bdf
Version v0.3.0-alpha.13-shell-out-r4
2021-08-28 22:05:49 -05:00
d4733321ed
Version v0.3.0-alpha.13-shell-out-r3
2021-08-28 20:37:53 -05:00
edd6bb4a60
Replace all bindings with calls to external binaries
2021-08-28 17:15:14 -05:00
7fd707c8df
Begin work on spawning processes rather than binding to c libs
2021-08-25 21:46:11 -05:00
a2c17a950b
Update actix deps
2021-06-26 13:10:28 -05:00
120184dd0c
Update dependencies
2021-06-19 14:39:41 -05:00
5e452ef27e
Update actix deps
2021-04-17 14:59:42 -05:00
7aa32b9b3b
Update actix deps
2021-03-09 20:51:03 -06:00
79b26a1dda
Update to actix-web 4.0.0-beta.3
2021-02-10 16:57:42 -06:00
9797e2e3fc
Go back to actix-fs
2021-01-18 19:54:39 -06:00
8c1f60101b
Add module for Range Header operations
...
Switch from actix-fs to async-fs
2021-01-18 17:11:32 -06:00
John Doe
da237eacae
fmt
2021-01-14 11:26:17 -05:00
John Doe
c663337bcd
ranged support for process endpoint, bugfixes
2021-01-14 08:52:11 -05:00
John Doe
61061f0451
range header support
2021-01-13 20:08:04 -05:00
c0520d181b
Randomize tmp folder name, clean tmp folder on exit
2021-01-05 11:51:13 -06:00
2e9fb2a830
Update download route to return 'details'
2021-01-05 00:28:17 -06:00
0acdfffe08
Return details in upload method, clippy nits
2020-12-25 20:39:19 -06:00
a750364e27
Expose cropping
2020-12-16 18:20:06 -06:00
c0a20588d3
Add content type, width, height to Details, add details endpoints
2020-12-10 12:49:10 -06:00
0bf7e0f688
Add initial details implementation
...
Details is an object generated for each image, containing
it's created_at date to properly set response headers. It
also can be expanding in the future to include an image's
dimensions, which we can expose via an API endpoint.
In addition to this, Details allows us the ability to
re-generate processed images. For now, all images which
do not yet have an associated Details will be
re-generated upon access. Details is set for each
thumbnail on generation, so this will only happen once
for each image.
2020-12-09 23:05:07 -06:00
4bd99431e2
Generate image bytes based on request format
2020-11-01 14:25:21 -06:00
4ce7b79b86
Safely create parent tmp dir when needed
2020-10-16 17:48:42 -05:00
b64e3615be
Switch many migration logs to debug, flush changes to new db
2020-10-16 16:02:48 -05:00
577d358da1
Enhance migrations to be more resilient to panics
2020-10-16 15:28:45 -05:00
f4f1ae6939
Update migration string
2020-10-10 19:38:07 -05:00
04e5d45679
Add Last-Modified header, extend MaxAge cache
2020-10-02 15:38:52 -05:00
6de19d17b6
Clean migration, don't use db tree
2020-09-14 16:42:31 -05:00
2515de00c2
Stable actix
2020-09-14 10:52:18 -05:00
4a3bd42c14
ffmpeg features, sled 034
2020-09-10 11:12:42 -05:00
7c0a407568
Add privileged purging, alias <-> filename relation queries
2020-07-11 17:22:45 -05:00
fd809e4a0b
Make 'import' a protected method
2020-07-11 16:28:49 -05:00
de3e04a411
Support mp4
2020-07-11 16:01:09 -05:00
d28d56fb9a
Fix odd-pixeled gif conversion
2020-07-11 15:49:33 -05:00
510f955cc0
Migrate from 0.32.0-rc1
2020-07-11 15:40:40 -05:00
cc37f0dd8a
Remove redundant default info, update readme help, add 'check' helper
2020-06-25 18:17:05 -05:00
a746793187
Enable mp4 with ffmpeg
2020-06-24 11:58:46 -05:00
eb685f99e7
Use sample for thumbnail generation
2020-06-16 21:05:06 -05:00
6dab3b30e8
Purge image
2020-06-16 18:13:22 -05:00
0fef7abc67
Validate with magickwand
2020-06-16 16:34:37 -05:00
df85faff8b
Start work on webp support
2020-06-16 15:55:24 -05:00
0801c1141f
dev build works with imagemagick
2020-06-15 14:20:25 -05:00
84c10edd0e
Improve bounds check, don't save variant unless it's been changed
2020-06-14 23:10:30 -05:00
928fb8f277
Replace 'move' with 'copy+delete'
2020-06-14 22:35:00 -05:00
154914e61a
Use rexiv2 for metadata removal
2020-06-14 21:41:45 -05:00
24f09408fb
Further clean tracing logs
2020-06-14 13:56:42 -05:00
3b4c5b646a
Do better logging
2020-06-14 10:07:31 -05:00
68721fb269
Replace log with tracing, only hardcode 'info' if RUST_LOG is empty
2020-06-13 21:28:06 -05:00
53345823bb
Better handle upload errors
2020-06-13 14:04:33 -05:00
6de89a3318
Don't use image crate to validate gifs
2020-06-11 11:46:00 -05:00
bfdfb737ed
Update deps
2020-06-10 19:34:25 -05:00
9b45751f70
Add import API for filename preservation
2020-06-10 19:26:54 -05:00
1fbb5005ef
Keep track of image variants rather than walking fs on delete
2020-06-09 18:40:01 -05:00
9a3b8dd4d7
Add GET for deleting images
2020-06-09 17:37:48 -05:00
0e7d4fc34f
Upgrading rust-musl and alpine, adding filter env var.
2020-06-07 18:29:22 -04:00
15cf50e3b1
Merge remote-tracking branch 'asonix/master'
2020-06-07 16:21:29 -04:00
a7bc7c0121
Adding docker and docker-compose.
2020-06-07 16:21:25 -04:00
4f55251310
Add filter whitelist
2020-06-07 14:12:19 -05:00
35aca6a426
Unneeded mut
2020-06-07 13:03:36 -05:00
ab8a24f25e
Identity doesn't alter path
2020-06-07 13:02:49 -05:00
fe0082a23b
Add a concept of 'processors' to allow multiple image transformations
2020-06-07 12:51:45 -05:00
d71b86a81e
Put image crate actions behind web::block
2020-06-07 11:34:44 -05:00
fc1ae4be49
Add endpoint for downloading remote images
2020-06-07 10:59:58 -05:00
1b526f4c1f
Remove redundant code
2020-06-06 22:24:33 -05:00
b7a66c7863
Remove outdated TODO
2020-06-06 22:21:42 -05:00
18e7aa8d06
Enable compression
2020-06-06 21:14:31 -05:00
459db42a88
Add optional image format coersion
2020-06-06 20:44:26 -05:00
981e146696
Add StructOpt
2020-06-06 19:54:06 -05:00
fcf18ceca5
Background deletions
2020-06-06 19:33:29 -05:00
d60862a0d0
Implement delete tokens
2020-06-06 19:29:15 -05:00
7298e500cb
Introduce alias system
2020-06-06 17:43:33 -05:00
65b83c6a06
Split into multiple files
2020-06-06 16:41:17 -05:00
6d7673dfcc
A Simple duplicate-resolving image host
2020-06-06 16:02:26 -05:00
