pict-rs/releases/0.4.0.md

# pict-rs 0.4.0

I am excited to announce a new stable version of pict-rs, version 0.4.0. There's a few big things in
0.4 that might be exciting to developers and admins. Below you'll find a list of changes from 0.3 to
0.4.

## Overview

### Headline Features:

- [Reworked Configuration](#reworked-configuration)
- [Reworked Commandline](#reworked-commandline)
- [Object Storage](#object-storage)
- [Backgrounded Uploads](#backgrounded-uploads)
- [Full Video](#full-video)


### Other Features:

- [Published Dependencies](#published-dependencies)
- [HEAD Endpoints](#head-endpoints)
- [Healthcheck Endpoints](#healthcheck-endpoints)
- [Image Preprocessing](#image-preprocessing)
- [GIF configuration](#gif-configuration)
- [JXL and AVIF](#jxl-and-avif)
- [H265, VP8, VP9, and AV1](#h265-vp8-vp9-and-av1)
- [404 Image](#404-image)
- [DB Export](#db-export)
- [Variant cleanup endpoint](#variant-cleanup-endpoint)
- [Client pool configuration](#client-pool-configuration)


### Fixes

- [GIF/webm Transparency](#gif-webm-transparency)
- [Image Orientation](#image-orientation)
- [io_uring file write](#io_uring-file-write)
- [Blur Sigma](#blur-sigma)


### Removals

- [Purge by disk name](#purge-by-disk-name)
- [Aliases by disk name](#aliases-by-disk-name)


## Upgrade Notes

TODO


## Descriptions

### Reworked Configuration

Starting off with the most important information for server admins: The configuration format has
changed. The `pict-rs.toml` is now far better organized, and includes more configuration options
than before. Every field has been moved, so please take a look at the [example
pict-rs.toml](https://git.asonix.dog/asonix/pict-rs/src/branch/main/pict-rs.toml) file for
information on how configuration works in 0.4.

Notable changes:
- `RUST_LOG` no longer has an effect, see `PICTRS__TRACING__LOGGING__TARGETS` and
    `PICTRS__TRACING__OPENTELEMETRY__TARGETS` for configuring log levels.
- `PICTRS__ADDRESS` has become `PICTRS__SERVER__ADDRESS`
- `PICTRS__PATH` has become `PICTRS__OLD_DB__PATH`, `PICTRS__REPO__PATH` and
    `PICTRS__STORE__PATH`


### Reworked Commandline

Also notable for server admins: the commandline interface has changed. All configuration that can be
expressed via the pict-rs.toml file or related environment variables can be set via the commandline
as well.

Running the pict-rs server now requires invoking the `run` subcommand. Here are some examples:
```bash
$ pict-rs run
$ pict-rs -c /etc/pict-rs.toml run
$ pict-rs run filesystem -p /path/to/files sled -p /path/to/metadata
```

There is one flag that I will call out here: `--save-to <path>`. This can be incredibly helpful when
debugging pict-rs. It dumps the configuration that pict-rs is currently using to a file of your
choosing. Example:
```bash
$ PICTRS__SERVER__ADDRESS=127.0.0.1:1234 pict-rs --save-to debug.toml run
CTRL^C
$ cat debug.toml
```
It can also be useful to save your existing commandline arguments into a reusable configuration:
```bash
$ pict-rs --save-to config.toml \
    --log-targets warn \
    --log-format json \
    --console-address '127.0.0.1:6669' \
    --opentelemetry-url 'http://localhost:4317' \
    run \
        -a '127.0.0.1:1234' \
        --api-key "$API_KEY" \
        --client-pool-size 10 \
        --media-max-width 200 \
        --media-max-height 200 \
        --media-max-file-size 1 \
        --media-enable-full-video true \
        --media-video-codec av1 \
        --media-format webp \
    filesystem \
        -p /opt/pict-rs/files \
    sled \
        -p /opt/pict-rs/metadata \
        -e /opt/pict-rs/exports
CTRL^C
$ cat config.toml
$ pict-rs -c config.toml run
```


### Object Storage

This is the biggest feature for pict-rs 0.4, but it isn't new. pict-rs 0.3 had initial support for
uploading to object storage hidden behind the `object-storage` compile flag. My docker images and
provided binaries did not enable this feature, so only folks compiling from source could even have
tried using it. In 0.4 I am now much more confident in the object storage support, and use it myself
for my personal setup.

Although pict-rs now supports object storage as a backend for storing uploaded media, it doesn't
support generating URLs that bypass the pict-rs server for serving images. I've seen this
misconception a couple times during the release candidate process and I would like to put these
rumors to rest.

Using object storage when you're in a cloud environment is sensible. The cost for adding object
storage is far less than for adding block storage, and although there are egress fees, they are
typically very low. Admins currently running pict-rs on a VPS should consider moving to object
storage if they find their disk usage is quickly growing.

Using object storage does not fully remove pict-rs' dependence on a local disk. pict-rs stores its
metadata in an embedded key/value store called [sled](https://github.com/spacejam/sled/), which
means it still requires disk access to function.

For new admins, using object storage from the start might be worth considering, but is not
required. pict-rs provides a built-in migration path for moving from filesystem storage to object
storage, which is documented in the [pict-rs
readme](https://git.asonix.dog/asonix/pict-rs/src/branch/main#user-content-filesystem-to-object-storage-migration).


### Backgrounded Uploads

This feature is important for keeping pict-rs fast. Since media needs to be processed on upload,
upload times may grow long. This is especially true when considering pict-rs' other new features
like [full video](#full-video) and [image preprocessing](#image-preprocessing).

Backgrounded uploads work by deferring the image processing work to a background task in pict-rs,
and responding to the upload HTTP request as soon as the provided file has been stored
successfully. Instead of returning a file alias like the inline upload endpoint, it instead returns
a UUID that represents the current upload process. That UUID can be given to a new `claim` endpoint
to retrieve the uploaded media's alias, or an error in the event the media was rejected or failed to
process. The claim endpoint uses a technique called "long polling" in order to notify clients on
completion of the background processing. Clients can request to claim uploaded media as soon as the
backgrounded endpoint reponds with an upload ID, and the server will hold the connection open for at
most 10 seconds while it waits for processing to complete. If processing did not complete in time,
the client can request again to claim the upload.


### Full Video

pict-rs now supports uploading video with audio. In 0.3, pict-rs would always strip audio from
uploaded videos. This was intended as a gif-like feature, since pict-rs' primary use is for storing
pictures and not videos. By default, full video is disabled in pict-rs 0.4, but server admins can
opt into full video uploads by setting `PICTRS__MEDIA__ENABLE_FULL_VIDEO=true`.

Along with supporting full video, an additional configuration option has been added to help keep
down on file sizes and processing time: `PICTRS__MEDIA__MAX_FRAME_COUNT`. This option limits how
many frames an uploaded video is allowed to have, rejecting the upload if it exceeds the provided
value.

It is important to note that while pict-rs is capable of storing videos, it's processing
functionality is limited to just the video's thumbnail.


### Published Dependencies

pict-rs 0.3's experimental object storage support relied on a custom fork of the `rust-s3` library,
which did useful things like enable re-use of an HTTP Client between connections, and making the
Client a `trait`, so it could be implemented for any arbitrary HTTP Client. Since then, the PR that
was meant to upstream those changes was closed, and so pict-rs needed to find a replacement that
suited it's needs.

pict-rs now relies on the [`rusty-s3`](https://github.com/paolobarbolini/rusty-s3) library for
enabling object storage. This library takes a sans-io approach, and was simple to integrate with
`awc`, my HTTP Client of choice.


### HEAD Endpoints