8f50a15b25
Reduce reliance on futures-util
2023-08-23 11:59:42 -05:00
0352ada9d8
Don't doulbe-open repo
2023-08-16 16:39:41 -05:00
88d4cc1a6a
Exit after repo migration
2023-08-16 16:34:42 -05:00
f95e61e402
Expose repo migrations
2023-08-16 16:32:19 -05:00
ba27a1a223
Build out repo-repo migration
2023-08-16 16:09:40 -05:00
1559d57f0a
Don't overwrite existing variants
2023-08-16 15:12:16 -05:00
22cfbe979d
Note how many failures have occurred for a given hash
2023-08-16 13:29:22 -05:00
ee88823d41
clippy
2023-08-16 12:43:12 -05:00
521cdd5b9d
Enforce format on new details
2023-08-16 12:36:18 -05:00
f3c6239979
Allow skipping details generation if already generated
2023-08-16 11:57:09 -05:00
4d909ba32a
Make migration concurrent
2023-08-16 11:47:36 -05:00
ac996ff832
Remove unused error
2023-08-15 21:25:04 -05:00
1de257bb07
Ensure access values are unique
2023-08-15 21:18:25 -05:00
bd3b668d81
Don't error on duplicate - happens on retried migration
2023-08-15 20:28:42 -05:00
9fe7410dd6
Add logging to migration
2023-08-15 19:33:19 -05:00
1ee77b19d8
Make Repo object safe. Create migration
2023-08-15 19:19:03 -05:00
09236d731d
Fix poll after completion, misused compare_and_swap
2023-08-14 21:17:57 -05:00
26ca3a7195
Remove associated Bytes type
2023-08-14 20:00:00 -05:00
da876fd553
Make it compile
2023-08-14 14:25:19 -05:00
3129f7844e
BROKEN: start work on hash discriminant
2023-08-13 22:06:42 -05:00
e4e93cddf9
Return JobId from push
2023-08-13 19:52:27 -05:00
5d4486fdf4
Finish queue implementation update for sled
2023-08-13 19:47:20 -05:00
35d0c065e9
Start work on 0.5 db format - initial queuerepo changes
2023-08-13 14:12:38 -05:00
0c4d632dda
Consider timeouts a client error
2023-08-05 16:18:06 -05:00
fee4ed1e3e
Add process timeout
2023-08-05 12:41:06 -05:00
5f12be0c6a
Clippy
2023-07-26 22:53:41 -05:00
2a59a6492a
Bring back hash_alias_key
2023-07-25 20:30:22 -05:00
bd3975f455
Simplify some repo functions, remove 0.3 migration
2023-07-25 20:08:18 -05:00
f221887599
Clippy
2023-07-23 16:04:17 -05:00
aac1bb7bc4
Finish media proxy implementation
2023-07-23 15:45:52 -05:00
4cca7d0f86
Prepare for proxying media
2023-07-23 12:10:36 -05:00
75bf425c6e
Update alias query, process query to accept proxy url
2023-07-23 11:45:10 -05:00
e59483c12c
Add ProxyRepo
2023-07-23 10:23:46 -05:00
d0c619bdf7
Clippy
2023-07-22 21:14:24 -05:00
77cdeab57e
Improve metrics, add job metrics
2023-07-22 21:11:28 -05:00
7dc8ede8eb
Gate cleanup behind read-only config
2023-07-22 20:26:28 -05:00
13d6e10886
Reduce log level on variants access
2023-07-22 20:20:58 -05:00
1f3298363f
Attempt to mark unmarked variants as accessed on launch
2023-07-22 19:55:50 -05:00
fe1f7c869f
Enable cleanup of variants
...
Still TODO: doing a first pass to mark variants as accessed on launch
2023-07-22 19:41:50 -05:00
47e13ec04e
Mark variants as accessed on fetch
2023-07-22 18:50:04 -05:00
b9e6d67d15
Add retention configuration
...
This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets
2023-07-22 18:37:25 -05:00
13387dec43
Fix specs
2023-07-22 17:58:03 -05:00
b786406ad0
Change IdentifierAccess to VariantAccess
2023-07-22 17:57:52 -05:00
ce0df080f4
Add prometheus metrics
2023-07-22 16:47:59 -05:00
d661732fb4
Tidy imports
2023-07-22 12:47:32 -05:00
a390b21df1
Make running pict-rs fluent
2023-07-22 12:39:39 -05:00
3ca994ee31
Replace global config with passed-around config
...
This will enable spinning up pict-rs multiple times in the same process with different configurations
2023-07-22 12:31:01 -05:00
e1262a5fda
Extract preprocess steps as urlencoded in config deserialization
2023-07-22 11:38:09 -05:00
ac48003f45
Extract ProcessMap out of static
2023-07-22 11:16:19 -05:00
8d35c2449d
Simplify the cancel-safe processor
2023-07-22 10:46:16 -05:00
c1127b4117
Set content-length in PutObject, CompleteMultipart
2023-07-21 19:45:22 -05:00
2f0a3618d8
Replace awc with reqwest
2023-07-21 16:58:31 -05:00
fc14ac2038
Clippy
2023-07-18 21:59:27 -05:00
3d93300ba9
Add access repos for alias & identifier
2023-07-18 21:56:13 -05:00
127dd6cdad
Make max file count configurable
2023-07-18 20:32:17 -05:00
eeac900d7e
Add quality value to process method, move -coalesce earlier in pipelines
2023-07-18 16:18:01 -05:00
558605381d
Fix dangling unprocessed uploads
...
Adds error boundary around backgrounded ingest
2023-07-17 22:32:08 -05:00
496bcc5dc1
Only include -b:v 0 for VP9
...
also simplify building arguments
2023-07-17 18:13:50 -05:00
3aecef3a9e
Remove println
2023-07-17 17:47:48 -05:00
b1bbc6b159
Attempt adding quality settings to pict-rs
2023-07-17 17:45:26 -05:00
c4c920191f
Add read-only mode
2023-07-17 14:24:49 -05:00
dd1d509bb1
Enable configuring download client timeout
2023-07-17 13:44:31 -05:00
5e8ab7856d
Extract Status errors into command-specific errors
2023-07-17 13:30:08 -05:00
2961aae6e3
Port migration changes from 0.4.1
2023-07-16 22:07:42 -05:00
a751d92436
Port error changes to 0.5
2023-07-16 21:51:14 -05:00
9871ad2ee0
Clippy
2023-07-14 23:16:57 -05:00
4d78ce7689
Fix store delegation for public_url
2023-07-14 15:23:07 -05:00
786f583d98
Add public_url to Store for optional redirects to public object store urls
...
Add details generation to file migration to set the content-type for uploads
2023-07-14 14:53:37 -05:00
4e58388b41
Clean alias before checking hash option
2023-07-13 19:58:31 -05:00
afeac8294a
Fix Alias cleanup ordering
...
Update dev.toml
2023-07-13 19:57:09 -05:00
ec6c9aac1e
Better discovery validation, proper invoking of imagemagick
2023-07-13 19:21:57 -05:00
9b1dd1f8c3
Copy process from 0.4.x branch
2023-07-13 19:21:28 -05:00
2c22f2ee3a
It compiles and runs, but doesn't work
2023-07-13 17:42:21 -05:00
27451971a6
Fix tests, add animated_webp test
2023-07-13 14:34:40 -05:00
33b83f97f2
It compiles
...
Need to reinstate media limits
2023-07-13 13:48:59 -05:00
ad1837f9dd
VERY BROKEN: start replacing parts of pict-rs
2023-07-12 22:12:21 -05:00
58d9765594
WIP: Rework format detection and conversions
...
- Add a bunch of types & methods that aren't used yet. The idea is
creating the output from the known input parameters is pure, and works
for any uploaded media
- Introduce a new kind of media "animation" which is neither an image
nor a video. It is likely that animations can be processed properly at
the /image/process.{ext} endpoint with a little massaging
TODO:
- Integrate this into exiftool, ffmpeg, magick commands
- Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a
hint)
- Infallible conversion from Details to OutputFile - this might be
impossible given that we don't currently store more than a mime type
and a frame count to give any sort of indication of real type
- Try not to break API
2023-07-11 23:11:25 -05:00
691bca286c
Make object storage timeouts configurable
2023-07-11 13:01:58 -05:00
f5c39f9be5
Typo, error formatting
2023-07-10 17:15:43 -05:00
4be2eb67d1
Increase object storage signature expiry to 15 seconds
2023-07-10 16:28:51 -05:00
9004ecaadf
Better classify process related errors
2023-07-10 15:29:41 -05:00
e10cb1401d
Bump version
2023-07-09 15:16:04 -05:00
7dadbcf3ed
Replace UnsupportedFormat error with more helpful errors
2023-07-09 15:07:49 -05:00
1694f49436
Use json for ffmpeg, imagemagick details parsing
2023-07-09 14:50:58 -05:00
686be1d93b
Turn export_path into flag for sled
2023-07-08 19:56:26 -05:00
7768d4e58e
Add database exports
...
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
2023-07-08 17:36:01 -05:00
4680f9c3aa
Instrument repo and store build methods
2023-07-08 10:42:03 -05:00
d51c348a3f
Run health checks on old and new stores before migrating
2023-07-07 14:05:13 -05:00
3ccb8ecd8c
Validate alias represents real file before setting not found
2023-07-07 13:33:27 -05:00
9e7376d411
Remove is_missing error, make Identifier and DeleteToken return Option on fetch
2023-07-07 13:17:26 -05:00
7274538b70
Try adding store healthcheck
2023-07-07 12:05:42 -05:00
c150903980
Expand store differentiation
2023-07-07 11:19:39 -05:00
75be876d3c
Differentiate missing files in old vs new store for migration
2023-07-07 11:03:10 -05:00
508dd4340b
Fix panic when migrating < 100 files, improve resumed migration messaging
2023-07-06 16:06:37 -05:00
3487cb0e30
Add ability to set 404 image
...
Fix imagemagick blur arguments
2023-07-05 19:07:03 -05:00
e7e4876908
Explicitly handle cases of invalid aliases
2023-07-05 16:46:44 -05:00
23e67b9697
Enhance migration handling of missing identifiers
...
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
2023-07-05 09:52:19 -05:00
2274dfecb4
Fix io-uring - read_to_end was polling Stream mapped through StreamReader after completion
2023-07-05 00:35:07 -05:00
c8e84b8914
Greatly improve error information during migration
2023-07-02 21:07:15 -05:00
e6f17cbac6
Log on successful migration
2023-06-29 11:45:06 -05:00
01d3610f46
Add basic progress indication for store migration
2023-06-29 11:39:47 -05:00
0d1df5ebc2
Avoid failed identifier cleanup on failed duplicate upload
2023-06-23 11:43:15 -05:00
7ba2fc9b26
Fix duplicate uploads potentially purging existing uploads on failure
2023-06-23 11:39:43 -05:00
58f0c328d9
Fix typo in error message
2023-06-23 11:20:55 -05:00
a1d48cbb42
Share client between object store and request extractor
...
Add config to limit connection pool size
2023-06-23 11:20:20 -05:00
071d788349
Include original error in object storage sendrequest error
2023-06-21 17:07:14 -05:00
a3ff072350
Remove unused cache duration parameter
2023-06-21 17:05:53 -05:00
6d2aef8cc0
Add jpegxl and avif support
2023-06-21 17:05:35 -05:00
554d852e68
Create unique errors for Repo and Store, separate from UploadError
...
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
2023-06-20 15:59:08 -05:00
1214b51ad7
Make use of FilesystemDefaults when constructing migrate commands
2023-06-19 15:04:36 -05:00
a3a986638d
Improve gif handling.
...
- Allow a fast-path exif cleaning if we'd re-encode a gif to a gif
- Use single-frame palettes to better map transparency from frame to frame
Unrelated:
- decrease ffmpeg logging when creating thumbnails
2023-06-19 14:25:31 -05:00
22e526fd2a
Allow type underscore (probably in instrument)
2023-06-01 17:33:43 -05:00
cc314c37a6
Use libopus for ffmpeg
2023-03-09 21:29:56 -06:00
dc27200a35
Add transparency to gifs, rustfmt
2023-02-25 14:31:57 -06:00
97ba4e806b
Clippy
2023-02-25 13:01:55 -06:00
9a2f846cd1
Detect input transparency for gif/webm
2023-02-25 12:59:47 -06:00
281ac43dff
Remove cache functionality
2023-02-25 11:34:48 -06:00
123488f617
clippy
2023-02-12 23:18:27 -06:00
788b87eceb
Remove obsolute pre-0.3 migrate check
2023-02-12 23:13:47 -06:00
138499d47f
Use transpose in a couple places
2023-02-05 20:50:59 -06:00
cc75d2d882
Clippy
2023-02-04 17:54:56 -06:00
4cc810d372
Add control over gif frame count
2023-02-04 17:52:23 -06:00
88ca9793e8
s/vrames/frames:v
2023-02-04 17:47:03 -06:00
40f57be0c7
Allow uploading small gifs
2023-02-04 17:32:36 -06:00
0aa3f574a5
Clippy
2023-01-29 11:57:59 -06:00
f6d6d54b88
Update deps
2023-01-29 11:47:28 -06:00
f0f40db8c3
Add healthcheck for db
2023-01-29 11:36:09 -06:00
5e5dd27a05
Retry migrate_file, remove variant before relating new one
2023-01-04 18:58:05 -06:00
28f7a139a0
Retry in-process, and rate-limit self
2023-01-04 17:26:05 -06:00
c55b1115d1
Increase client timeout, attempt to keep track of migration progress for resuming
2023-01-04 15:58:32 -06:00
dabcc24f98
Parse environment variables - main app
2023-01-04 15:05:34 -06:00
a489411c6c
Parse environment variables
2023-01-04 14:42:04 -06:00
3c844d86c5
Allow configuring pict-rs through serializable types
2022-12-26 16:35:25 -06:00
2c1a37fb62
Extract pict-rs as library
2022-12-23 12:56:15 -06:00
74cd3392fa
Borrow parent spans when creating children
2022-12-07 22:43:12 -06:00
3325b222a2
clippy
2022-11-26 11:18:34 -06:00
7037ee5791
Support auto-orienting photos
2022-10-15 11:13:24 -05:00
669b3fb86f
Fix webp metadata stripping, more tracing cleanup
2022-10-01 22:47:52 -05:00
718f09c43a
Clean tracing, simplify validation, rename InputFormat -> VideoFormat
2022-10-01 21:17:18 -05:00
e7cf21f862
Allow concurrency in small object uploads
2022-10-01 13:06:19 -05:00
fe2a499110
Fix sending upload parts (set size), optimize small file uploads
2022-10-01 13:00:07 -05:00
0307dc5a3b
Introduce BytesStream to simplify aggregating, forwarding bytes
2022-10-01 12:06:33 -05:00
c767804430
Default to vp9 for uploaded video
2022-10-01 10:17:32 -05:00
85c5ce955d
Clippy
2022-09-30 20:02:46 -05:00
d86d9a3228
Put proper file extension on videos
2022-09-30 20:00:14 -05:00
8eb2293808
Enable selecting video and audio codecs for uploaded media
2022-09-30 19:38:11 -05:00
f4542efcc1
Share ffprobe details code
2022-09-30 17:43:40 -05:00
3ceaaa391c
Update clap
2022-09-28 18:23:41 -05:00
656b0225af
Clippy
2022-09-27 23:23:06 -05:00
29cab025c6
Improve 0.3 migration code, repo traces
2022-09-27 23:19:52 -05:00
