d475723087
Finish writing migrations & generate diesel schema
2023-09-02 13:35:30 -05:00
8c532c97e6
Initial postgres work
2023-09-02 11:52:55 -05:00
ba3a23ed43
Add error codes
2023-09-01 20:51:01 -05:00
fd74161c61
Rename some repo methods, generate UploadId in repo
2023-09-01 18:41:04 -05:00
8fb90a6f69
Give a meaningful distinction for format in details
2023-08-31 16:26:45 -05:00
08fd96c2f7
clippy nit
2023-08-29 13:29:40 -05:00
c0f6b1e988
Enable searching hashes by date
2023-08-29 13:27:18 -05:00
0146202236
Fully commit to HumanDate
2023-08-29 12:59:36 -05:00
64950bfe0e
Shorten pagination slugs
2023-08-29 12:53:29 -05:00
2b6ea97773
Enable setting timestamp on hash creation, improve paging api
2023-08-28 18:43:24 -05:00
24812afeba
Add paged hash access
2023-08-28 16:02:11 -05:00
e302ab1f3d
Migrate not-found alias
2023-08-28 13:40:19 -05:00
ba27a1a223
Build out repo-repo migration
2023-08-16 16:09:40 -05:00
1559d57f0a
Don't overwrite existing variants
2023-08-16 15:12:16 -05:00
22cfbe979d
Note how many failures have occurred for a given hash
2023-08-16 13:29:22 -05:00
ee88823d41
clippy
2023-08-16 12:43:12 -05:00
521cdd5b9d
Enforce format on new details
2023-08-16 12:36:18 -05:00
f3c6239979
Allow skipping details generation if already generated
2023-08-16 11:57:09 -05:00
4d909ba32a
Make migration concurrent
2023-08-16 11:47:36 -05:00
ac996ff832
Remove unused error
2023-08-15 21:25:04 -05:00
1de257bb07
Ensure access values are unique
2023-08-15 21:18:25 -05:00
bd3b668d81
Don't error on duplicate - happens on retried migration
2023-08-15 20:28:42 -05:00
9fe7410dd6
Add logging to migration
2023-08-15 19:33:19 -05:00
1ee77b19d8
Make Repo object safe. Create migration
2023-08-15 19:19:03 -05:00
09236d731d
Fix poll after completion, misused compare_and_swap
2023-08-14 21:17:57 -05:00
26ca3a7195
Remove associated Bytes type
2023-08-14 20:00:00 -05:00
da876fd553
Make it compile
2023-08-14 14:25:19 -05:00
3129f7844e
BROKEN: start work on hash discriminant
2023-08-13 22:06:42 -05:00
e4e93cddf9
Return JobId from push
2023-08-13 19:52:27 -05:00
5d4486fdf4
Finish queue implementation update for sled
2023-08-13 19:47:20 -05:00
35d0c065e9
Start work on 0.5 db format - initial queuerepo changes
2023-08-13 14:12:38 -05:00
5f12be0c6a
Clippy
2023-07-26 22:53:41 -05:00
2a59a6492a
Bring back hash_alias_key
2023-07-25 20:30:22 -05:00
bd3975f455
Simplify some repo functions, remove 0.3 migration
2023-07-25 20:08:18 -05:00
e59483c12c
Add ProxyRepo
2023-07-23 10:23:46 -05:00
77cdeab57e
Improve metrics, add job metrics
2023-07-22 21:11:28 -05:00
13d6e10886
Reduce log level on variants access
2023-07-22 20:20:58 -05:00
1f3298363f
Attempt to mark unmarked variants as accessed on launch
2023-07-22 19:55:50 -05:00
fe1f7c869f
Enable cleanup of variants
...
Still TODO: doing a first pass to mark variants as accessed on launch
2023-07-22 19:41:50 -05:00
47e13ec04e
Mark variants as accessed on fetch
2023-07-22 18:50:04 -05:00
b786406ad0
Change IdentifierAccess to VariantAccess
2023-07-22 17:57:52 -05:00
fc14ac2038
Clippy
2023-07-18 21:59:27 -05:00
3d93300ba9
Add access repos for alias & identifier
2023-07-18 21:56:13 -05:00
2961aae6e3
Port migration changes from 0.4.1
2023-07-16 22:07:42 -05:00
7768d4e58e
Add database exports
...
This should help admins running pict-rs on traditional filesystems to
keep valid backups of pict-rs' sled repo
2023-07-08 17:36:01 -05:00
4680f9c3aa
Instrument repo and store build methods
2023-07-08 10:42:03 -05:00
9e7376d411
Remove is_missing error, make Identifier and DeleteToken return Option on fetch
2023-07-07 13:17:26 -05:00
e7e4876908
Explicitly handle cases of invalid aliases
2023-07-05 16:46:44 -05:00
23e67b9697
Enhance migration handling of missing identifiers
...
Queue a cleanup task for hashes that don't have original file identifiers
Be more specific about what field is missing when a field is missing
2023-07-05 09:52:19 -05:00
01d3610f46
Add basic progress indication for store migration
2023-06-29 11:39:47 -05:00
554d852e68
Create unique errors for Repo and Store, separate from UploadError
...
Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound
Allow skipping files that are not found during store migration
2023-06-20 15:59:08 -05:00
281ac43dff
Remove cache functionality
2023-02-25 11:34:48 -06:00
123488f617
clippy
2023-02-12 23:18:27 -06:00
788b87eceb
Remove obsolute pre-0.3 migrate check
2023-02-12 23:13:47 -06:00
138499d47f
Use transpose in a couple places
2023-02-05 20:50:59 -06:00
0aa3f574a5
Clippy
2023-01-29 11:57:59 -06:00
f0f40db8c3
Add healthcheck for db
2023-01-29 11:36:09 -06:00
5e5dd27a05
Retry migrate_file, remove variant before relating new one
2023-01-04 18:58:05 -06:00
669b3fb86f
Fix webp metadata stripping, more tracing cleanup
2022-10-01 22:47:52 -05:00
718f09c43a
Clean tracing, simplify validation, rename InputFormat -> VideoFormat
2022-10-01 21:17:18 -05:00
29cab025c6
Improve 0.3 migration code, repo traces
2022-09-27 23:19:52 -05:00
21e3e63ac3
Instrument upload store methods
2022-09-25 18:46:50 -05:00
e0bc95f785
Remove unneeded reborrow
2022-08-15 15:00:12 -05:00
547934071a
Don't migrate from the old database if it doesn't exist
2022-08-14 11:49:20 -05:00
dc73bdbad4
Make already-claimed a 400
2022-06-05 23:24:58 -05:00
b3cbda1337
Add internal endpoint for cleaning variants
2022-04-11 16:56:39 -05:00
05e2cf5e08
Cache on Alias rather than Hash
...
Make sled repo blocking tasks share parent span
2022-04-06 12:14:12 -05:00
865f4ee09c
Caching works:
...
- Insert new bucket for current access time before removing
previous access time record.
- Add instrument to more methods for more precise tracing
2022-04-05 21:47:35 -05:00
6cdae7b318
Add 'cache' endpoint for ingesting ephemeral media
...
By default, cached media should only stick around for 7 days, however
The timeout is reset every time media is accessed, so only obscure
cached media will be flushed from the cache. This '7 days' number is
configurable through the commandline run options as --media-cache-duration
and in the pict-rs.toml file as [media] cache_duration
2022-04-05 20:29:30 -05:00
b0670c2f66
Move migrate to old
2022-04-03 15:07:31 -05:00
8734dfbdc7
Add backgrounding routes
...
- Accept backgrounded uploads
- Allow backgrounded processing
Still TODO:
- Endpoint for waiting on/claiming an upload
2022-04-02 20:56:29 -05:00
c4d014597e
Better job recovery
2022-04-02 18:53:03 -05:00
77a400c7ca
Implement UploadRepo
2022-04-02 17:41:00 -05:00
09f53b9ce6
Move away from UploadManager to direct repo & store actions
2022-04-02 16:44:03 -05:00
c0d8e0e8e3
Support multiple queues for job processor
2022-04-01 11:51:46 -05:00
63d66050c8
Enable buffering iterator items for iterstream
2022-03-29 16:18:00 -05:00
5adb3fde89
Move some Stream implementations into 'stream' module
2022-03-29 15:59:17 -05:00
602d1ea935
Begin implementing queue, put cleanups in it
2022-03-29 12:51:16 -05:00
eb5e39c634
Use color-eyre
2022-03-28 20:47:46 -05:00
1662f153ce
Numerous changes:
...
- Improve error printing (display chain in addition to spantrace)
- Fix migration (read main identifier from identifier tree, not filename tree)
- Ensure uniqueness for processed images in ConcurrentProcessor (use source identifier in addition to thumbnail path, include extension in thumbnail path)
- Update default log levels (make pict-rs quieter)
- Add timeout for serving images from object storage (5 seconds)
2022-03-28 15:34:36 -05:00
37e6b21b55
Fix Range, consolidate errors, test object storage
2022-03-26 20:45:12 -05:00
15b52ba6ec
Get it compiling again
2022-03-26 16:49:23 -05:00
323016f994
Build out migration path
2022-03-25 18:47:50 -05:00
d6567fbbbd
Implement sled repo
2022-03-24 17:09:15 -05:00
