fff4afe105
Start working on per-server temporary directory
2023-10-06 19:42:24 -05:00
914e21c043
Implement constant-time equality for delete tokens, inline alias cleanup
2023-10-04 12:11:29 -05:00
d28c1ac628
Upgrade Concurrency: Update docs, configs, limit maxiumum details operations
2023-10-03 17:04:40 -05:00
247902e600
Enable specifying migrate-store concurrency
2023-10-03 15:50:53 -05:00
81c6e73b5f
Simplify old_repo, remove client_pool_size
2023-09-30 22:46:15 -05:00
47751f3875
Downgrade some WARN logs to INFO, update docs for public methods
2023-09-30 17:52:58 -05:00
66e1711723
Enable proper draining of dropped request payloads
...
Doing this as the outermost middleware ensures all endpoints are covered.
Update request deadline to turn negative deadlines into immediate failures
2023-09-30 16:26:43 -05:00
745e990fa0
Remove direct dependency on actix-rt and actix-server
2023-09-24 15:45:10 -05:00
0926739d3c
Remove once_cell
2023-09-24 15:32:00 -05:00
e0968dcce3
Remove serde-cbor, num_cpus
2023-09-24 15:22:57 -05:00
6c921817e1
Enable thumbnailing animations, use dynamic dispatch for a number of async readers
2023-09-24 11:54:16 -05:00
1b97ac1c5a
Start replacing manual stream implementations with streem
2023-09-10 18:55:13 -04:00
a761daca34
clippy
2023-09-05 20:55:11 -05:00
e15a82c0c7
Add external validation check
2023-09-05 20:45:07 -05:00
3bd0f78e75
Use with_timeout for all timeouts
2023-09-04 21:58:57 -05:00
a4b1ab7dfb
Instrument postgres db calls
2023-09-04 21:51:27 -05:00
a43de122f9
postgres: add already-claimed case, general: tracing paranoia
2023-09-03 21:30:47 -05:00
a2933dbebc
Implement all the todos
2023-09-03 17:11:34 -05:00
8921f57a21
Remove Identifier trait, Replace with Arc<str>
2023-09-02 18:38:02 -05:00
8c532c97e6
Initial postgres work
2023-09-02 11:52:55 -05:00
ba3a23ed43
Add error codes
2023-09-01 20:51:01 -05:00
fd74161c61
Rename some repo methods, generate UploadId in repo
2023-09-01 18:41:04 -05:00
cbb66f1b75
Add internal endpoint for deleting alias without a delete token
2023-09-01 17:25:13 -05:00
8fb90a6f69
Give a meaningful distinction for format in details
2023-08-31 16:26:45 -05:00
c0f6b1e988
Enable searching hashes by date
2023-08-29 13:27:18 -05:00
64950bfe0e
Shorten pagination slugs
2023-08-29 12:53:29 -05:00
2b6ea97773
Enable setting timestamp on hash creation, improve paging api
2023-08-28 18:43:24 -05:00
24812afeba
Add paged hash access
2023-08-28 16:02:11 -05:00
95637fdfe5
Remove direct dep on futures-util
2023-08-23 19:10:10 -05:00
8f50a15b25
Reduce reliance on futures-util
2023-08-23 11:59:42 -05:00
0352ada9d8
Don't doulbe-open repo
2023-08-16 16:39:41 -05:00
88d4cc1a6a
Exit after repo migration
2023-08-16 16:34:42 -05:00
f95e61e402
Expose repo migrations
2023-08-16 16:32:19 -05:00
ba27a1a223
Build out repo-repo migration
2023-08-16 16:09:40 -05:00
521cdd5b9d
Enforce format on new details
2023-08-16 12:36:18 -05:00
4d909ba32a
Make migration concurrent
2023-08-16 11:47:36 -05:00
1ee77b19d8
Make Repo object safe. Create migration
2023-08-15 19:19:03 -05:00
da876fd553
Make it compile
2023-08-14 14:25:19 -05:00
5d4486fdf4
Finish queue implementation update for sled
2023-08-13 19:47:20 -05:00
35d0c065e9
Start work on 0.5 db format - initial queuerepo changes
2023-08-13 14:12:38 -05:00
fee4ed1e3e
Add process timeout
2023-08-05 12:41:06 -05:00
2a59a6492a
Bring back hash_alias_key
2023-07-25 20:30:22 -05:00
bd3975f455
Simplify some repo functions, remove 0.3 migration
2023-07-25 20:08:18 -05:00
f221887599
Clippy
2023-07-23 16:04:17 -05:00
aac1bb7bc4
Finish media proxy implementation
2023-07-23 15:45:52 -05:00
4cca7d0f86
Prepare for proxying media
2023-07-23 12:10:36 -05:00
75bf425c6e
Update alias query, process query to accept proxy url
2023-07-23 11:45:10 -05:00
7dc8ede8eb
Gate cleanup behind read-only config
2023-07-22 20:26:28 -05:00
1f3298363f
Attempt to mark unmarked variants as accessed on launch
2023-07-22 19:55:50 -05:00
fe1f7c869f
Enable cleanup of variants
...
Still TODO: doing a first pass to mark variants as accessed on launch
2023-07-22 19:41:50 -05:00
47e13ec04e
Mark variants as accessed on fetch
2023-07-22 18:50:04 -05:00
ce0df080f4
Add prometheus metrics
2023-07-22 16:47:59 -05:00
d661732fb4
Tidy imports
2023-07-22 12:47:32 -05:00
a390b21df1
Make running pict-rs fluent
2023-07-22 12:39:39 -05:00
3ca994ee31
Replace global config with passed-around config
...
This will enable spinning up pict-rs multiple times in the same process with different configurations
2023-07-22 12:31:01 -05:00
ac48003f45
Extract ProcessMap out of static
2023-07-22 11:16:19 -05:00
2f0a3618d8
Replace awc with reqwest
2023-07-21 16:58:31 -05:00
127dd6cdad
Make max file count configurable
2023-07-18 20:32:17 -05:00
eeac900d7e
Add quality value to process method, move -coalesce earlier in pipelines
2023-07-18 16:18:01 -05:00
c4c920191f
Add read-only mode
2023-07-17 14:24:49 -05:00
dd1d509bb1
Enable configuring download client timeout
2023-07-17 13:44:31 -05:00
2961aae6e3
Port migration changes from 0.4.1
2023-07-16 22:07:42 -05:00
9871ad2ee0
Clippy
2023-07-14 23:16:57 -05:00
786f583d98
Add public_url to Store for optional redirects to public object store urls
...
Add details generation to file migration to set the content-type for uploads
2023-07-14 14:53:37 -05:00
2c22f2ee3a
It compiles and runs, but doesn't work
2023-07-13 17:42:21 -05:00
33b83f97f2
It compiles
...
Need to reinstate media limits
2023-07-13 13:48:59 -05:00
ad1837f9dd
VERY BROKEN: start replacing parts of pict-rs
2023-07-12 22:12:21 -05:00
58d9765594
WIP: Rework format detection and conversions
...
- Add a bunch of types & methods that aren't used yet. The idea is
creating the output from the known input parameters is pure, and works
for any uploaded media
- Introduce a new kind of media "animation" which is neither an image
nor a video. It is likely that animations can be processed properly at
the /image/process.{ext} endpoint with a little massaging
TODO:
- Integrate this into exiftool, ffmpeg, magick commands
- Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a
hint)
- Infallible conversion from Details to OutputFile - this might be
impossible given that we don't currently store more than a mime type
and a frame count to give any sort of indication of real type
- Try not to break API
2023-07-11 23:11:25 -05:00
691bca286c
Make object storage timeouts configurable
2023-07-11 13:01:58 -05:00
e10cb1401d
Bump version
2023-07-09 15:16:04 -05:00
7dadbcf3ed
Replace UnsupportedFormat error with more helpful errors
2023-07-09 15:07:49 -05:00
7768d4e58e
Add database exports
...
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
2023-07-08 17:36:01 -05:00
d51c348a3f
Run health checks on old and new stores before migrating
2023-07-07 14:05:13 -05:00
3ccb8ecd8c
Validate alias represents real file before setting not found
2023-07-07 13:33:27 -05:00
9e7376d411
Remove is_missing error, make Identifier and DeleteToken return Option on fetch
2023-07-07 13:17:26 -05:00
7274538b70
Try adding store healthcheck
2023-07-07 12:05:42 -05:00
c150903980
Expand store differentiation
2023-07-07 11:19:39 -05:00
75be876d3c
Differentiate missing files in old vs new store for migration
2023-07-07 11:03:10 -05:00
508dd4340b
Fix panic when migrating < 100 files, improve resumed migration messaging
2023-07-06 16:06:37 -05:00
3487cb0e30
Add ability to set 404 image
...
Fix imagemagick blur arguments
2023-07-05 19:07:03 -05:00
e7e4876908
Explicitly handle cases of invalid aliases
2023-07-05 16:46:44 -05:00
23e67b9697
Enhance migration handling of missing identifiers
...
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
2023-07-05 09:52:19 -05:00
c8e84b8914
Greatly improve error information during migration
2023-07-02 21:07:15 -05:00
e6f17cbac6
Log on successful migration
2023-06-29 11:45:06 -05:00
01d3610f46
Add basic progress indication for store migration
2023-06-29 11:39:47 -05:00
a1d48cbb42
Share client between object store and request extractor
...
Add config to limit connection pool size
2023-06-23 11:20:20 -05:00
554d852e68
Create unique errors for Repo and Store, separate from UploadError
...
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
2023-06-20 15:59:08 -05:00
281ac43dff
Remove cache functionality
2023-02-25 11:34:48 -06:00
f0f40db8c3
Add healthcheck for db
2023-01-29 11:36:09 -06:00
5e5dd27a05
Retry migrate_file, remove variant before relating new one
2023-01-04 18:58:05 -06:00
28f7a139a0
Retry in-process, and rate-limit self
2023-01-04 17:26:05 -06:00
c55b1115d1
Increase client timeout, attempt to keep track of migration progress for resuming
2023-01-04 15:58:32 -06:00
3c844d86c5
Allow configuring pict-rs through serializable types
2022-12-26 16:35:25 -06:00
2c1a37fb62
Extract pict-rs as library
2022-12-23 12:56:15 -06:00
