diff --git a/releases/0.4.0.md b/releases/0.4.0.md index ab4f12d..fe86591 100644 --- a/releases/0.4.0.md +++ b/releases/0.4.0.md @@ -1,8 +1,8 @@ # pict-rs 0.4.0 I am excited to announce a new stable version of pict-rs, version 0.4.0. There's a few big things in -0.4 that might be exciting to developers and admins. Below you'll find a list of changes from 0.3 to -0.4. +0.4 that might be exciting to developers and admins. Below you'll find upgrade notes for +administrators, as well as a list of changes from 0.3 to 0.4. ## Overview @@ -15,6 +15,10 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's - [Full Video](#full-video) +### Changes + +- [Default video codec](#default-video-codec) + ### Other Features: - [Published Dependencies](#published-dependencies) @@ -24,10 +28,12 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's - [GIF configuration](#gif-configuration) - [JXL and AVIF](#jxl-and-avif) - [H265, VP8, VP9, and AV1](#h265-vp8-vp9-and-av1) +- [Audio Codecs](#audio-codecs) - [404 Image](#404-image) - [DB Export](#db-export) -- [Variant cleanup endpoint](#variant-cleanup-endpoint) -- [Client pool configuration](#client-pool-configuration) +- [Identifier Endpoint](#identifier-endpoint) +- [Variant Cleanup Endpoint](#variant-cleanup-endpoint) +- [Client Pool Configuration](#client-pool-configuration) ### Fixes @@ -42,11 +48,22 @@ I am excited to announce a new stable version of pict-rs, version 0.4.0. There's - [Purge by disk name](#purge-by-disk-name) - [Aliases by disk name](#aliases-by-disk-name) +- [Filename endpoint](#filename-endpoint) ## Upgrade Notes -TODO +If you're running the provided docker container with default configuration values, the 0.4 container +can be pulled and launched with no changes. There is a migration from the 0.3 database format to the +0.4 database format that will occur automatically on first launch. After the migration is complete, +the server will start. + +If you have any custom configuration, or are running outside of docker, see [Reworked +Configuration](#reworked-configuration) below. It is likely that you will need to change +configuration values for the migration to run properly. + +More information can be found in the [pict-rs +readme](https://git.asonix.dog/asonix/pict-rs#user-content-0-3-to-0-4-migration-guide) ## Descriptions @@ -66,6 +83,38 @@ Notable changes: - `PICTRS__PATH` has become `PICTRS__OLD_DB__PATH`, `PICTRS__REPO__PATH` and `PICTRS__STORE__PATH` +The pict-rs.toml equivalents: +```toml +# pict-rs.toml + +[server] +# same format as the previous PICTRS__ADDRESS +address = "0.0.0.0:8080" + +[tracing.logging] +# same format as RUST_LOG +targets = "info" + +[tracing.opentelemetry] +# same format as RUST_LOG +targets = "debug" + +[old_db] +# same as the previous PICTRS__PATH +path = "/mnt" + +[repo] +# Added the `sled-repo` subdirectory to the previous PICTRS__PATH +path = "/mnt/sled-repo" + +[store] +# Added the `files` subdirectory to the previous PICTRS__PATH +path = "/mnt/files" +``` + +As mentioned above, every configuration option has moved. Check the linked example configuration +file for more information. + ### Reworked Commandline @@ -178,6 +227,13 @@ It is important to note that while pict-rs is capable of storing videos, it's pr functionality is limited to just the video's thumbnail. +### Default Video Codec + +pict-rs 0.4 has changed the default video codec from h264 to vp9. This was done because mp4 is still +patent-encumbered, and I don't want to ship software the defaults to using it. This change may +result is slower transcoding times. + + ### Published Dependencies pict-rs 0.3's experimental object storage support relied on a custom fork of the `rust-s3` library, @@ -192,3 +248,318 @@ enabling object storage. This library takes a sans-io approach, and was simple t ### HEAD Endpoints + +For the `/image/original/{filename}` and `/image/process.{ext}?src={filename}` endpoints, there are +now analogous `HEAD` endpoints for fetching just the headers their `GET` counterparts would return. + + +### Healthcheck Endpoints + +pict-rs now provides a `/healthz` endpoint that can be used to check if the running process is +healthy. It currently attempts to write to the sled database and fetch information from the +configured store, returning 200 if all operations succeeded. + + +### Image Preprocessing + +pict-rs 0.4 introduces a new configuration option: `PICTRS__MEDIA__PREPROCESS_STEPS`. This option +accepts the same syntax as the `process` endpoint, and can be used to apply transformations to all +uploaded images. + +Here are some examples: + +```toml +# pict-rs.toml + +[media] +# Crop all uploaded media to a 16x9 aspect ratio, and then shrink to fit within a 1200 by 1200 pixel box. +preprocess_steps = "crop=16x9&resize=1200" +``` + +```yaml +# docker-compose.yml + +services: + pictrs: + image: asonix/pictrs:0.4.0 + ports: + - "127.0.0.1:8080:8080" + restart: always + environment: + # Blur all uploaded images with a sigma value of 1.5 + - PICTRS__MEDIA__PREPROCESS_STEPS=blur=1.5 + volumes: + - ./volumes/pictrs:/mnt +``` + + +### GIF configuration + +pict-rs 0.3 would transcode all uploaded GIF files to video formats. This is no longer the case. +Some new configuration options have been introduced to determine when GIF files should and should +not be converted to silent videos. By default, GIF files will remain GIFs if their dimensions are +smaller than 128x128 px, and they don't have more than 100 frames. GIFs to that do fit these +restrictions will be transcoded to a video format. + +To tweak these values, the following environment variables can be set: +```yaml +# docker-compose.yml + +services: + pictrs: + # ... + environment: + - PICTRS__MEDIA__GIF__MAX_WIDTH=128 + - PICTRS__MEDIA__GIF__MAX_HEIGHT=128 + - PICTRS__MEDIA__GIF__MAX_AREA=16384 + - PICTRS__MEDIA__GIF__MAX_FRAME_COUNT=100 +``` + +Alternatively, these values can be set in pict-rs.toml +```toml +# pict-rs.toml + +[media.gif] +max_width = 128 +max_height = 128 +max_area = 16384 +max_frame_count = 100 +``` + + +### JXL and AVIF + +pict-rs 0.4 introduces support for two new image formts: JPEG XL (JXL) and AVIF. These two image +formats offer better compression to quality ratios than other standard formats, although they may be +less well-supported by browsers or other clients. Not only can these formats be uploaded and handled +now, but they can also be used with the `PICTRS__MEDIA__FORMAT` variable for transcoding all +uploaded images. + +Exmaples: +```yaml +# docker-compose.yml + +services: + pictrs: + # ... + environment: + - PICTRS__MEDIA__FORMAT=avif +``` + +```toml +# pict-rs.toml +[media] +format = "jxl" +``` + + +### H265, VP8, VP9, and AV1 + +As mentioned earlier, the default video codec has changed from `h264` to `vp9`, however, pict-rs 0.4 +introduces support for a broader range of video codecs: +- in the mp4 container + - h264 + - h265 +- in the webm container + - vp8 + - vp9 + - av1 + +The default video codec can be changed with the following settings: +```yaml +# docker-compose.yml + +services: + pictrs: + # ... + environment: + - PICTRS__MEDIA__VIDEO_CODEC=av1 +``` + +```toml +# pict-rs.toml + +[media] +video_codec = "h265" +``` + +Note that h264 and h265 in the mp4 container are still patent-encumbered, and you may wish to avoid +them. + + +### Audio Codecs + +With the introduction of full video, pict-rs adds the ability to configure the audio codec used for +uploaded videos. By default, pict-rs will choose a reasonable codec for the provided video codec. +This means that there is no need to configure the audio codec yourself. The option is still +available, though. + +Here is the existing mapping between video and audio codecs: +- av1, vp8, and vp9 use opus by default +- h264 and h265 use aac by default + +Available codecs for configuration are +- opus +- vorbis +- aac + +Setting a preferred codec can be done with the following configurations: +```yaml +# docker-compose.yml + +services: + pictrs: + # ... + environment: + - PICTRS__MEDIA__AUDIO_CODEC=vorbis +``` + +```toml +# pict-rs.toml + +[media] +audio_codec = "opus" +``` + + +### 404 Image + +pict-rs 0.4 adds support for setting a "Not Found" image. This image will be returned for any +request for an image that does not exist on the server. This is done through an internal endpoint, +which means it requires pict-rs' `PICTRS__SERVER__API_KEY` to be set. + +After uploading an image to pict-rs, it's name can be submitted to the `/internal/set_not_found` +endpoint with the following JSON: +```json +{ + "alias": "79087a4f-ea31-4878-9e8a-0498dada597b.webp" +} +``` + +If the alias exists on the server, it will mark that image for use as the "404 image" + + +### DB Export + +Part of administering a server means taking regular backups. Many backup systems rely on simply +copying files around, which doesn't work well for pict-rs' sled database, which may change while +being read by the backup tool. + +One solution to this is to use a snapshotting filesystem like btrfs, bcachefs, or zfs. A filesystem +snapshot is "atomic", and will represent the database at an exact moment in time. + +For admins running on more common filesystems such as ext4 or xfs, there is now a configuration +option and internal endpoint for producing immutable copies of the current database state. + +```toml +# pict-rs.toml + +[repo] +export_path = "./data/exports" +``` + +```yaml +# docker-compose.yml + +services: + pictrs: + # ... + environment: + - PICTRS__REPO__EXPORT_PATH=/opt/pict-rs/exports +``` + +When this value is set, the `/internal/export` endpoint becomes available. Hitting that endpoint +will result in pict-rs exporting its current state to a new timestamped directory inside the +`export_path`. This directory contains the same structure as the `sled-repo` directory, and in the +event of data loss, can be restored by simply moving it into place. + +Here's an example process for restoring from an export: +1. Stop pict-rs +2. Move your current `sled-repo` directory to a safe location (e.g. `sled-repo.bak`) + ```bash + $ mv sled-repo sled-repo.bak + ``` +3. Copy an exported database to `sled-repo` + ```bash + $ cp -r exports/2023-07-08T22:26:21.194126713Z sled-repo + ``` +4. Start pict-rs + + +### Identifier Endpoint + +pict-rs 0.4 introduces a new endpoint for retrieving an alias' "True Name", the Identifier. An +Identifier represents the real path to the file in filesystem storage or in object storage. +Currently these paths are generated using the same algorithm, but pict-rs retains the right to +optimize identifiers differently for filesystem and object storage in the future. + + +### Variant Cleanup Endpoint + +This is one of the less exciting features, and won't be very commonly used. A new endpoint at +`/internal/variants` accepts the DELETE verb. This will spawn a background task that iterates over +all files in the database, removing any generated variants of an image, keeping only the Original. +All variants can be generated again on request. + + +### Client Pool Configuration + +Another less exciting feature, the number of connections pict-rs' internal HTTP Client maintains in +its pool can now be configured. This value is 100 by default, and shouldn't need to be tweaked. If +you are experiencing heavy traffic and are using object storage, you could increase the value. If +you are constrained on allowed open file descriptors, you can decrease the value. + +Note that this value is multiplied by the number of threads pict-rs spawns, which depends on the +number of cores allowed to the application. Limiting pict-rs to 2 CPUs with default pool settings +will result in an effective limit of 200 connections. + + +### GIF/webm Transparency + +In 0.3, pict-rs removed transparency from uploaded gifs and videos. 0.4 supports preserving the +transparency of GIF and webm files on upload. + + +### Image Orientation + +In 0.3, pict-rs would strip metadata from all images. This would remove metadata about the intended +orientation of an image, and lead to uploaded media being sideways or upside down. pict-rs 0.4 still +strips all metadata from uploaded images, but it now applies a rotation to images that require it as +well. Images uploaded in version 0.4 should no longer be sideways or upside down. + + +### io_uring File Write + +Due to a conflict between the `AsyncReadExt` trait and `StreamReader`, actix-web's `Payload` stream +would be polled after completion when pict-rs' `io-uring` feature was enabled. This has been fixed +by using the `read_buf` API rather than the `read_to_end` API in the io_uring file implementation. + +Note that io_uring is still considered an experimental feature, and probably should not be used in +production. YMMV. + + +### Blur Sigma + +pict-rs' `blur` filter would incorrectly apply the provided `sigma` value as the `radius` for a +gaussian blur. This has been fixed in 0.4 to properly set the `sigma` value with a default `radius` +of 0. This tells imagemagick to calculate a reasonable radius for the given image and sigma. + + +### Purge by disk name + +Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the +`/internal/purge` endpoint can no longer be provided with a `filename` query, and must be given an +`alias` query instead. + + +### Aliases by disk name + +Due to changes in how pict-rs stores files in 0.4 (introducing a pluggable store backend), the +`/internal/aliases` endpoint can no longer be provided with a `filename` query, and must be given an +`alias` query instead. + + +### Filename Endpoint + +This was replaced by the Identifier endpoint, since files are no longer required to be stored on the +filesystem.