9db5cc82f8
Add special handling for FileSize errors
2024-02-19 12:55:44 -06:00
4a7a51d147
Add more imagemagick security configuration
...
Remove default security policy, since it is overridden anyway
Update default value for max_area
Inherit animation frame count configuration for list-length
Add release document
2024-02-05 14:30:01 -06:00
71d81681d5
It compiles
2024-02-04 15:45:47 -06:00
d8c4a22ba9
Start describing metrics
2024-02-04 14:26:18 -06:00
8869b82065
Update rustls for actix-web
2024-02-03 21:58:08 -06:00
85f6356025
Fix images with trailing bytes failing to upload
2024-02-03 19:50:00 -06:00
835647d290
Unite launch-with-store fns
2024-02-03 18:42:34 -06:00
c176e4c686
clippy
2024-02-03 18:30:52 -06:00
880dfc20ee
Remove unneeded struct, import
2024-02-03 18:20:17 -06:00
7eba45f37e
cargo fix & cargo fmt
2024-02-03 18:19:14 -06:00
50e31f96da
It compiles again
2024-02-03 18:18:13 -06:00
f2410a9283
use State
2024-02-03 13:32:20 -06:00
cf7c753e65
Pass a State value around rather than a bunch of arguments
2024-02-03 13:31:54 -06:00
c3e322f2c9
typo
2024-02-02 14:00:54 -06:00
3a6faef5ef
Add logs for pict-rs binds
2024-02-01 21:55:31 -06:00
5805eb0aed
Enable configuring imagemagick security policy from pictrs
2024-01-31 21:32:01 -06:00
50fd3e6182
Enable serving over TLS
2024-01-31 17:47:42 -06:00
08472eabeb
Log console address, prometheus address with tracing
2024-01-30 15:21:15 -06:00
e7357cbdaf
Count spawned tasks in cleanup, debug log queue length
2024-01-30 14:57:48 -06:00
34cadb86b8
Don't propogate pop span into blocking closure
2024-01-30 14:30:24 -06:00
ee00fafee4
Yield more often in queue, simplify traces, log job_id in higher span
2024-01-30 14:18:07 -06:00
dc12f6f79d
replace remaining web::block with spawn_blocking
2024-01-30 14:16:05 -06:00
7282ee9312
Give background jobs more opportunity to yield to runtime
2024-01-25 16:59:46 -06:00
819b83bab7
Use a far smaller pool to listen for db notifications
2024-01-25 16:50:36 -06:00
c65b49339f
Take at most 2048 variants and proxies during each cleanup job
2024-01-25 16:26:29 -06:00
574b12b23f
Fix postgres commandline, add default for use_tls
2024-01-25 08:55:51 -06:00
2f588aa613
Ignore tokio-uring test in docs
2024-01-24 17:41:42 -06:00
fe1132aec1
Enable at-most-once queueing for some cleanup jobs
2024-01-24 17:14:31 -06:00
fee2294901
clippy
2024-01-15 18:21:36 -05:00
19147e2035
postgres: allow connecting to TLS-enabled databases
2024-01-15 18:11:08 -05:00
f6087d65be
Use async fn in trait for Store
2024-01-15 15:51:20 -05:00
9ed90efed4
Add release document, use .extend rather than .join
2024-01-11 16:48:18 -06:00
708c25f20a
Join identifier onto public_endpoint, rather than overwriting the existing path
2024-01-11 16:06:44 -06:00
9ceb26bb5c
Clippy
2024-01-07 10:09:06 -06:00
65e165af49
Set default audio codecs if none are specified
2024-01-07 10:03:40 -06:00
ca34a15cd3
Debug-log command arguments
2024-01-07 10:01:02 -06:00
004a54c4ae
Improve cooperation from background jobs
2024-01-06 17:47:44 -06:00
bcf73eb4e4
Improve cooperation from from_iterator to prevent task starvation
2024-01-05 19:50:10 -06:00
4145637a33
Add a few more metrics, increase repo disconnect sleep
2024-01-05 19:35:52 -06:00
688c797082
Remove LIMIT 10, single_value is already in use
2024-01-04 11:13:29 -06:00
6514487ea9
Index on queue_time for job queue, limit to 10 jobs in ID query
2024-01-04 11:01:25 -06:00
0fd6c459d0
Clippy
2024-01-04 10:42:34 -06:00
7d4f44a540
Set worker-id to None when marking a job as runnable after expired heartbeats
2024-01-04 10:29:37 -06:00
7e215c691b
Support RUST_LOG
2023-12-28 12:40:59 -06:00
bfc2410552
Add trace-level logs to every loop (paranoid)
2023-12-28 11:58:38 -06:00
dc90e00324
Update metrics to 0.22 (plus mechanical changes)
2023-12-26 18:06:38 -06:00
bd3cdc925e
Remove special-case for animated avif
2023-12-23 21:44:11 -06:00
6d259a0eea
Don't kill already-exited processes, don't log tmp_dir when fetching details
2023-12-23 13:29:30 -06:00
c2e53766c2
Don't -ping when extracting file details (workaround for imagemagick json bug)
2023-12-23 12:48:32 -06:00
df8fc00ad5
Try to clean up properly even if command errors
2023-12-23 12:11:15 -06:00
89a4e3995f
Add warn logs when performing blocking removes due to dropped files
2023-12-23 12:10:25 -06:00
6fa79b9188
Do more cleanup inline
2023-12-23 11:58:20 -06:00
e8380c31c1
Fix remaining warnings
2023-12-22 21:03:51 -06:00
c2b1b3db5d
Apply cargo fix
2023-12-22 21:01:19 -06:00
c544f167fb
Remove unused code
2023-12-22 21:00:37 -06:00
b938b09901
Apply cargo fix
2023-12-22 20:54:02 -06:00
5624671cbf
Rework ProcessRead to require a closure for using the internal AsyncRead
2023-12-22 20:52:58 -06:00
df88660b9e
Lower priority of some object storage spans
2023-12-22 13:58:05 -06:00
4369283534
More tracing cleaning
2023-12-22 13:30:17 -06:00
aa8fe7f3e8
Clean unused imports
2023-12-22 13:20:36 -06:00
0dbe722524
Don't poll handle if closed, prevents panic if child wait finishes before all bytes are read
2023-12-22 13:20:21 -06:00
b94ba5fcfc
Inline process background future, clean tracing a bit
2023-12-22 13:12:19 -06:00
db43392a3b
Simplify process future by removing explicit channel, don't lock mutex as often
2023-12-22 12:03:05 -06:00
c9155f7ce7
Include not_found key in repo migration, rework ProcessRead, add timeout to 0.4 migration
2023-12-17 23:15:27 -06:00
bd0b89b165
Clean spans a bit
2023-12-15 22:34:45 -06:00
2544cb478f
Add static bound to appease 1.74.1
2023-12-12 17:17:16 -06:00
f1c5a56353
Port prune_missing from 0.4.x
2023-12-12 16:54:41 -06:00
3ef4097ceb
Enable building without tokio_unstable
2023-12-12 16:23:02 -06:00
6daa745446
Add optional ability to log spans
2023-12-11 14:58:53 -06:00
cd80ab16c3
Downgrade retry warnings to debug
2023-12-11 13:08:10 -06:00
b452a577a7
Add object ID to not found error
2023-12-11 13:05:29 -06:00
7293628148
put timeout on collecting payloads
2023-12-10 19:26:16 -06:00
961de20a00
Add timeout and metrics to media processing
2023-12-10 19:11:36 -06:00
2523b6644b
Add AggregateTimeout, don't aggregate with permit
2023-12-04 15:39:34 -06:00
6b2ccbf975
Allow configuring pict-rs' temporary directory
2023-11-24 11:48:45 -06:00
a30044278c
Improve documentation for using pict-rs as a library
2023-11-24 11:38:14 -06:00
71f8f52b58
Tweak error reporting
2023-11-13 17:24:56 -06:00
1538389caf
Clippy
2023-11-11 14:28:05 -06:00
210af5d7d9
Add danger_dummy_mode
2023-11-11 14:22:12 -06:00
5fd59fc5b4
Change commandline flag to allow disabling video, since it is enabled by default
2023-11-11 13:22:12 -06:00
3a0546694f
Update opentelemetry dependencies
2023-11-10 15:19:54 -06:00
15189d293b
clippy
2023-11-09 18:26:57 -06:00
f61dac8187
Attempt to clean stray magick files
...
This change sets a unique temp directory for each invocation of imagemagick and spawns a task to remove that directory after the command terminates
2023-11-09 18:20:59 -06:00
20d07ea29b
Remove direct dependency on quick-xml, update dependencies (minor & point)
2023-11-04 13:18:22 -05:00
c795c1edfa
Replace most of actix-rt with tokio, give names to tasks
2023-10-20 19:08:11 -05:00
0bf8b6a2a8
Update rusty-s3 to 0.5.0
2023-10-08 11:29:59 -05:00
71fbc4c55b
Just have different mains
2023-10-07 12:31:05 -05:00
760b9d8c59
Install tracing & metrics from within runtime
2023-10-07 12:29:01 -05:00
3f5bc1d311
Build same pict-rs config for either runtime option
2023-10-07 12:22:17 -05:00
f97f76c01e
Fix io-uring initialization
2023-10-07 12:12:40 -05:00
4a45aa889c
Enable running on bare tokio
2023-10-07 11:48:58 -05:00
7f5cbc4571
Attempt clean-drop of tmp_dir
2023-10-07 11:36:49 -05:00
7b5a3020fa
Don't rely on global state for tmp_dir
2023-10-07 11:32:36 -05:00
fff4afe105
Start working on per-server temporary directory
2023-10-06 19:42:24 -05:00
c202484e9d
clippy
2023-10-04 12:46:38 -05:00
914e21c043
Implement constant-time equality for delete tokens, inline alias cleanup
2023-10-04 12:11:29 -05:00
d28c1ac628
Upgrade Concurrency: Update docs, configs, limit maxiumum details operations
2023-10-03 17:04:40 -05:00
f223d477c7
Enable specifying concurrency for 0.4 to 0.5 migration
2023-10-03 16:27:19 -05:00
75551dc3a1
Increase deadpool wait timeout, helps with extreme concurrency
2023-10-03 16:15:49 -05:00
247902e600
Enable specifying migrate-store concurrency
2023-10-03 15:50:53 -05:00
8568ae3364
Clippy
2023-09-30 22:47:49 -05:00
4c7067d4ca
Allow running without old_repo specified
2023-09-30 22:46:15 -05:00
81c6e73b5f
Simplify old_repo, remove client_pool_size
2023-09-30 22:46:15 -05:00
47751f3875
Downgrade some WARN logs to INFO, update docs for public methods
2023-09-30 17:52:58 -05:00
01cbe34ac3
Remove unneeded annotation
2023-09-30 17:34:05 -05:00
4bb9d81f51
Move internal middleware into own file
2023-09-30 17:33:01 -05:00
e3462f6664
payload middleware: switch to Rc, always inject if Payload isn't None
2023-09-30 17:24:48 -05:00
3267fb8301
Fail requests for malformed deadlines
2023-09-30 17:20:32 -05:00
65ea518542
Default log level to info
2023-09-30 16:28:04 -05:00
66e1711723
Enable proper draining of dropped request payloads
...
Doing this as the outermost middleware ensures all endpoints are covered.
Update request deadline to turn negative deadlines into immediate failures
2023-09-30 16:26:43 -05:00
6f44b05c0a
Remove references to Arbiter
2023-09-29 11:54:43 -05:00
735fcd1b49
Remove dependency on actix-web-codegen
2023-09-24 15:52:31 -05:00
ecdbaa9c92
Use diesel-async re-export of Hook
2023-09-24 15:48:19 -05:00
745e990fa0
Remove direct dependency on actix-rt and actix-server
2023-09-24 15:45:10 -05:00
0926739d3c
Remove once_cell
2023-09-24 15:32:00 -05:00
e0968dcce3
Remove serde-cbor, num_cpus
2023-09-24 15:22:57 -05:00
fd965bb1a5
Allow processing still images into animation formats
2023-09-24 13:20:33 -05:00
f3ef2d870b
Generate details during ingest
2023-09-24 12:58:16 -05:00
8f62b0c325
clippy
2023-09-24 12:42:23 -05:00
9ffdd0432f
Infer video thumbnail format from image format
2023-09-24 12:37:43 -05:00
6c921817e1
Enable thumbnailing animations, use dynamic dispatch for a number of async readers
2023-09-24 11:54:16 -05:00
cde4a72203
Add metrics to object storage body streams
2023-09-20 19:46:10 -05:00
49d3037358
Add metrics to object storage requests
2023-09-20 19:39:03 -05:00
7df6f3906e
Add timeout, metrics to external validation request
2023-09-20 19:29:19 -05:00
d0646dee25
Add config option for external validation timeout
2023-09-20 19:23:55 -05:00
4a5b9712aa
Combine map_ok and map_err underlying impl
2023-09-10 23:51:07 -04:00
2b10c48619
Update io-uring to use streem for file bytes
2023-09-10 23:45:24 -04:00
ea75ca24b5
Add track_caller to sync methods, update streem
2023-09-10 20:43:51 -04:00
b2674f06d0
More streme
2023-09-10 20:08:01 -04:00
1b97ac1c5a
Start replacing manual stream implementations with streem
2023-09-10 18:55:13 -04:00
858899b943
Add metrics & tracing to dropped uploads, turn pop into a single query
2023-09-07 19:20:41 -05:00
783df60f81
Fix on_conflict for postgres store migration path
2023-09-06 20:49:00 -05:00
a761daca34
clippy
2023-09-05 20:55:11 -05:00
e15a82c0c7
Add external validation check
2023-09-05 20:45:07 -05:00
bc49f8ca37
Migrate with timestamp
2023-09-04 22:25:11 -05:00
9c50bbe23a
match -> matches!
2023-09-04 22:06:28 -05:00
492b99922b
10 second worker delay after disconnect error
2023-09-04 22:04:15 -05:00
62ebc1c60a
Move timeout after metrics - timeouts count as incomplete
2023-09-04 22:02:59 -05:00
3bd0f78e75
Use with_timeout for all timeouts
2023-09-04 21:58:57 -05:00
a4b1ab7dfb
Instrument postgres db calls
2023-09-04 21:51:27 -05:00
37448722de
postgres: Rework job & upload notifications (more)
...
postgres: Add metrics to job push & pop, upload wait
sled: add upload wait metrics
2023-09-04 16:20:26 -05:00
00aa00c55c
postgres: Enable 'previous page' when less than full limit fit on previous page
2023-09-03 21:58:31 -05:00
a43de122f9
postgres: add already-claimed case, general: tracing paranoia
2023-09-03 21:30:47 -05:00
31caea438e
Fix slow connection pool access
2023-09-03 20:05:29 -05:00
8a892ba622
it might work
2023-09-03 18:21:46 -05:00
a2933dbebc
Implement all the todos
2023-09-03 17:11:34 -05:00
94cb2a9ef3
Pass clippy
2023-09-03 16:59:41 -05:00
443d327edf
Implement a couple more repo traits
2023-09-03 12:47:06 -05:00
eac4cd54a4
Initial work for pg notifications
2023-09-02 20:13:32 -05:00
e580e7701e
Finish implementing HashRepo
2023-09-02 19:13:00 -05:00
