LemmyNet/pict-rs
2
0
Fork 0
mirror of https://git.asonix.dog/asonix/pict-rs synced 2024-12-21 19:01:25 +00:00
Code Releases Activity

Flesh out remaing feature descriptions, fixes, changes, add upgrade notes

Browse source
This commit is contained in:
asonix 2023-07-08 21:34:21 -05:00
parent 535ee33a97
commit fa74e3c82d
1 changed files with 376 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

381
releases/0.4.0.md
View file

@ -1,8 +1,8 @@
# pict-rs 0.4.0
I am excited to announce a new stable version of pict-rs, version 0.4.0. There's a few big things in
0.4 that might be exciting to developers and admins. Below you'll find a list of changes from 0.3 to
0.4.
0.4 that might be exciting to developers and admins. Below you'll find upgrade notes for
administrators, as well as a list of changes from 0.3 to 0.4.
## Overview
@ -15,6 +15,10 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's
- [Full Video](#full-video)
### Changes
- [Default video codec](#default-video-codec)
### Other Features:
- [Published Dependencies](#published-dependencies)
@ -24,10 +28,12 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's
- [GIF configuration](#gif-configuration)
- [JXL and AVIF](#jxl-and-avif)
- [H265, VP8, VP9, and AV1](#h265-vp8-vp9-and-av1)
- [Audio Codecs](#audio-codecs)
- [404 Image](#404-image)
- [DB Export](#db-export)
- [Variant cleanup endpoint](#variant-cleanup-endpoint)
- [Client pool configuration](#client-pool-configuration)
- [Identifier Endpoint](#identifier-endpoint)
- [Variant Cleanup Endpoint](#variant-cleanup-endpoint)
- [Client Pool Configuration](#client-pool-configuration)
### Fixes
@ -42,11 +48,22 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's
- [Purge by disk name](#purge-by-disk-name)
- [Aliases by disk name](#aliases-by-disk-name)
- [Filename endpoint](#filename-endpoint)
## Upgrade Notes
TODO
If you're running the provided docker container with default configuration values, the 0.4 container
can be pulled and launched with no changes. There is a migration from the 0.3 database format to the
0.4 database format that will occur automatically on first launch. After the migration is complete,
the server will start.
If you have any custom configuration, or are running outside of docker, see [Reworked
Configuration](#reworked-configuration) below. It is likely that you will need to change
configuration values for the migration to run properly.
More information can be found in the [pict-rs
readme](https://git.asonix.dog/asonix/pict-rs#user-content-0-3-to-0-4-migration-guide)
## Descriptions
@ -66,6 +83,38 @@ Notable changes:
- `PICTRS__PATH` has become `PICTRS__OLD_DB__PATH`, `PICTRS__REPO__PATH` and
`PICTRS__STORE__PATH`
The pict-rs.toml equivalents:
```toml
# pict-rs.toml
[server]
# same format as the previous PICTRS__ADDRESS
address = "0.0.0.0:8080"
[tracing.logging]
# same format as RUST_LOG
targets = "info"
[tracing.opentelemetry]
# same format as RUST_LOG
targets = "debug"
[old_db]
# same as the previous PICTRS__PATH
path = "/mnt"
[repo]
# Added the `sled-repo` subdirectory to the previous PICTRS__PATH
path = "/mnt/sled-repo"
[store]
# Added the `files` subdirectory to the previous PICTRS__PATH
path = "/mnt/files"
```
As mentioned above, every configuration option has moved. Check the linked example configuration
file for more information.
### Reworked Commandline
@ -178,6 +227,13 @@ It is important to note that while pict-rs is capable of storing videos, it's pr
functionality is limited to just the video's thumbnail.
### Default Video Codec
pict-rs 0.4 has changed the default video codec from h264 to vp9. This was done because mp4 is still
patent-encumbered, and I don't want to ship software the defaults to using it. This change may
result is slower transcoding times.
### Published Dependencies
pict-rs 0.3's experimental object storage support relied on a custom fork of the `rust-s3` library,
@ -192,3 +248,318 @@ enabling object storage. This library takes a sans-io approach, and was simple t
### HEAD Endpoints
For the `/image/original/{filename}` and `/image/process.{ext}?src={filename}` endpoints, there are
now analogous `HEAD` endpoints for fetching just the headers their `GET` counterparts would return.
### Healthcheck Endpoints
pict-rs now provides a `/healthz` endpoint that can be used to check if the running process is
healthy. It currently attempts to write to the sled database and fetch information from the
configured store, returning 200 if all operations succeeded.
### Image Preprocessing
pict-rs 0.4 introduces a new configuration option: `PICTRS__MEDIA__PREPROCESS_STEPS`. This option
accepts the same syntax as the `process` endpoint, and can be used to apply transformations to all
uploaded images.
Here are some examples:
```toml
# pict-rs.toml
[media]
# Crop all uploaded media to a 16x9 aspect ratio, and then shrink to fit within a 1200 by 1200 pixel box.
preprocess_steps = "crop=16x9&resize=1200"
```
```yaml
# docker-compose.yml
services:
pictrs:
image: asonix/pictrs:0.4.0
ports:
- "127.0.0.1:8080:8080"
restart: always
environment:
# Blur all uploaded images with a sigma value of 1.5
- PICTRS__MEDIA__PREPROCESS_STEPS=blur=1.5
volumes:
- ./volumes/pictrs:/mnt
```
### GIF configuration
pict-rs 0.3 would transcode all uploaded GIF files to video formats. This is no longer the case.
Some new configuration options have been introduced to determine when GIF files should and should
not be converted to silent videos. By default, GIF files will remain GIFs if their dimensions are
smaller than 128x128 px, and they don't have more than 100 frames. GIFs to that do fit these
restrictions will be transcoded to a video format.
To tweak these values, the following environment variables can be set:
```yaml
# docker-compose.yml
services:
pictrs:
# ...
environment:
- PICTRS__MEDIA__GIF__MAX_WIDTH=128
- PICTRS__MEDIA__GIF__MAX_HEIGHT=128
- PICTRS__MEDIA__GIF__MAX_AREA=16384
- PICTRS__MEDIA__GIF__MAX_FRAME_COUNT=100
```
Alternatively, these values can be set in pict-rs.toml
```toml
# pict-rs.toml
[media.gif]
max_width = 128
max_height = 128
max_area = 16384
max_frame_count = 100
```
### JXL and AVIF
pict-rs 0.4 introduces support for two new image formts: JPEG XL (JXL) and AVIF. These two image
formats offer better compression to quality ratios than other standard formats, although they may be
less well-supported by browsers or other clients. Not only can these formats be uploaded and handled
now, but they can also be used with the `PICTRS__MEDIA__FORMAT` variable for transcoding all
uploaded images.
Exmaples:
```yaml
# docker-compose.yml
services:
pictrs:
# ...
environment:
- PICTRS__MEDIA__FORMAT=avif
```
```toml
# pict-rs.toml
[media]
format = "jxl"
```
### H265, VP8, VP9, and AV1
As mentioned earlier, the default video codec has changed from `h264` to `vp9`, however, pict-rs 0.4
introduces support for a broader range of video codecs:
- in the mp4 container
- h264
- h265
- in the webm container
- vp8
- vp9
- av1
The default video codec can be changed with the following settings:
```yaml
# docker-compose.yml
services:
pictrs:
# ...
environment:
- PICTRS__MEDIA__VIDEO_CODEC=av1
```
```toml
# pict-rs.toml
[media]
video_codec = "h265"
```
Note that h264 and h265 in the mp4 container are still patent-encumbered, and you may wish to avoid
them.
### Audio Codecs
With the introduction of full video, pict-rs adds the ability to configure the audio codec used for
uploaded videos. By default, pict-rs will choose a reasonable codec for the provided video codec.
This means that there is no need to configure the audio codec yourself. The option is still
available, though.
Here is the existing mapping between video and audio codecs:
- av1, vp8, and vp9 use opus by default
- h264 and h265 use aac by default
Available codecs for configuration are
- opus
- vorbis
- aac
Setting a preferred codec can be done with the following configurations:
```yaml
# docker-compose.yml
services:
pictrs:
# ...
environment:
- PICTRS__MEDIA__AUDIO_CODEC=vorbis
```
```toml
# pict-rs.toml
[media]
audio_codec = "opus"
```
### 404 Image
pict-rs 0.4 adds support for setting a "Not Found" image. This image will be returned for any
request for an image that does not exist on the server. This is done through an internal endpoint,
which means it requires pict-rs' `PICTRS__SERVER__API_KEY` to be set.
After uploading an image to pict-rs, it's name can be submitted to the `/internal/set_not_found`
endpoint with the following JSON:
```json
{
"alias": "79087a4f-ea31-4878-9e8a-0498dada597b.webp"
}
```
If the alias exists on the server, it will mark that image for use as the "404 image"
### DB Export
Part of administering a server means taking regular backups. Many backup systems rely on simply
copying files around, which doesn't work well for pict-rs' sled database, which may change while
being read by the backup tool.
One solution to this is to use a snapshotting filesystem like btrfs, bcachefs, or zfs. A filesystem
snapshot is "atomic", and will represent the database at an exact moment in time.
For admins running on more common filesystems such as ext4 or xfs, there is now a configuration
option and internal endpoint for producing immutable copies of the current database state.
```toml
# pict-rs.toml
[repo]
export_path = "./data/exports"
```
```yaml
# docker-compose.yml
services:
pictrs:
# ...
environment:
- PICTRS__REPO__EXPORT_PATH=/opt/pict-rs/exports
```
When this value is set, the `/internal/export` endpoint becomes available. Hitting that endpoint
will result in pict-rs exporting its current state to a new timestamped directory inside the
`export_path`. This directory contains the same structure as the `sled-repo` directory, and in the
event of data loss, can be restored by simply moving it into place.
Here's an example process for restoring from an export:
1. Stop pict-rs
2. Move your current `sled-repo` directory to a safe location (e.g. `sled-repo.bak`)
```bash
$ mv sled-repo sled-repo.bak
```
3. Copy an exported database to `sled-repo`
```bash
$ cp -r exports/2023-07-08T22:26:21.194126713Z sled-repo
```
4. Start pict-rs
### Identifier Endpoint
pict-rs 0.4 introduces a new endpoint for retrieving an alias' "True Name", the Identifier. An
Identifier represents the real path to the file in filesystem storage or in object storage.
Currently these paths are generated using the same algorithm, but pict-rs retains the right to
optimize identifiers differently for filesystem and object storage in the future.
### Variant Cleanup Endpoint
This is one of the less exciting features, and won't be very commonly used. A new endpoint at
`/internal/variants` accepts the DELETE verb. This will spawn a background task that iterates over
all files in the database, removing any generated variants of an image, keeping only the Original.
All variants can be generated again on request.
### Client Pool Configuration
Another less exciting feature, the number of connections pict-rs' internal HTTP Client maintains in
its pool can now be configured. This value is 100 by default, and shouldn't need to be tweaked. If
you are experiencing heavy traffic and are using object storage, you could increase the value. If
you are constrained on allowed open file descriptors, you can decrease the value.
Note that this value is multiplied by the number of threads pict-rs spawns, which depends on the
number of cores allowed to the application. Limiting pict-rs to 2 CPUs with default pool settings
will result in an effective limit of 200 connections.
### GIF/webm Transparency
In 0.3, pict-rs removed transparency from uploaded gifs and videos. 0.4 supports preserving the
transparency of GIF and webm files on upload.
### Image Orientation
In 0.3, pict-rs would strip metadata from all images. This would remove metadata about the intended
orientation of an image, and lead to uploaded media being sideways or upside down. pict-rs 0.4 still
strips all metadata from uploaded images, but it now applies a rotation to images that require it as
well. Images uploaded in version 0.4 should no longer be sideways or upside down.
### io_uring File Write
Due to a conflict between the `AsyncReadExt` trait and `StreamReader`, actix-web's `Payload` stream
would be polled after completion when pict-rs' `io-uring` feature was enabled. This has been fixed
by using the `read_buf` API rather than the `read_to_end` API in the io_uring file implementation.
Note that io_uring is still considered an experimental feature, and probably should not be used in
production. YMMV.
### Blur Sigma
pict-rs' `blur` filter would incorrectly apply the provided `sigma` value as the `radius` for a
gaussian blur. This has been fixed in 0.4 to properly set the `sigma` value with a default `radius`
of 0. This tells imagemagick to calculate a reasonable radius for the given image and sigma.
### Purge by disk name
Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the
`/internal/purge` endpoint can no longer be provided with a `filename` query, and must be given an
`alias` query instead.
### Aliases by disk name
Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the
`/internal/aliases` endpoint can no longer be provided with a `filename` query, and must be given an
`alias` query instead.
### Filename Endpoint
This was replaced by the Identifier endpoint, since files are no longer required to be stored on the
filesystem.