From 6bc7e86099a3c6cf1f1a662d54e97f71d1a133b8 Mon Sep 17 00:00:00 2001 From: asonix Date: Sat, 8 Jul 2023 19:56:40 -0500 Subject: [PATCH] Start work on release announcement --- releases/0.4.0.md | 194 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 releases/0.4.0.md diff --git a/releases/0.4.0.md b/releases/0.4.0.md new file mode 100644 index 0000000..ab4f12d --- /dev/null +++ b/releases/0.4.0.md @@ -0,0 +1,194 @@ +# pict-rs 0.4.0 + +I am excited to announce a new stable version of pict-rs, version 0.4.0. There's a few big things in +0.4 that might be exciting to developers and admins. Below you'll find a list of changes from 0.3 to +0.4. + +## Overview + +### Headline Features: + +- [Reworked Configuration](#reworked-configuration) +- [Reworked Commandline](#reworked-commandline) +- [Object Storage](#object-storage) +- [Backgrounded Uploads](#backgrounded-uploads) +- [Full Video](#full-video) + + +### Other Features: + +- [Published Dependencies](#published-dependencies) +- [HEAD Endpoints](#head-endpoints) +- [Healthcheck Endpoints](#healthcheck-endpoints) +- [Image Preprocessing](#image-preprocessing) +- [GIF configuration](#gif-configuration) +- [JXL and AVIF](#jxl-and-avif) +- [H265, VP8, VP9, and AV1](#h265-vp8-vp9-and-av1) +- [404 Image](#404-image) +- [DB Export](#db-export) +- [Variant cleanup endpoint](#variant-cleanup-endpoint) +- [Client pool configuration](#client-pool-configuration) + + +### Fixes + +- [GIF/webm Transparency](#gif-webm-transparency) +- [Image Orientation](#image-orientation) +- [io_uring file write](#io_uring-file-write) +- [Blur Sigma](#blur-sigma) + + +### Removals + +- [Purge by disk name](#purge-by-disk-name) +- [Aliases by disk name](#aliases-by-disk-name) + + +## Upgrade Notes + +TODO + + +## Descriptions + +### Reworked Configuration + +Starting off with the most important information for server admins: The configuration format has +changed. The `pict-rs.toml` is now far better organized, and includes more configuration options +than before. Every field has been moved, so please take a look at the [example +pict-rs.toml](https://git.asonix.dog/asonix/pict-rs/src/branch/main/pict-rs.toml) file for +information on how configuration works in 0.4. + +Notable changes: +- `RUST_LOG` no longer has an effect, see `PICTRS__TRACING__LOGGING__TARGETS` and + `PICTRS__TRACING__OPENTELEMETRY__TARGETS` for configuring log levels. +- `PICTRS__ADDRESS` has become `PICTRS__SERVER__ADDRESS` +- `PICTRS__PATH` has become `PICTRS__OLD_DB__PATH`, `PICTRS__REPO__PATH` and + `PICTRS__STORE__PATH` + + +### Reworked Commandline + +Also notable for server admins: the commandline interface has changed. All configuration that can be +expressed via the pict-rs.toml file or related environment variables can be set via the commandline +as well. + +Running the pict-rs server now requires invoking the `run` subcommand. Here are some examples: +```bash +$ pict-rs run +$ pict-rs -c /etc/pict-rs.toml run +$ pict-rs run filesystem -p /path/to/files sled -p /path/to/metadata +``` + +There is one flag that I will call out here: `--save-to `. This can be incredibly helpful when +debugging pict-rs. It dumps the configuration that pict-rs is currently using to a file of your +choosing. Example: +```bash +$ PICTRS__SERVER__ADDRESS=127.0.0.1:1234 pict-rs --save-to debug.toml run +CTRL^C +$ cat debug.toml +``` +It can also be useful to save your existing commandline arguments into a reusable configuration: +```bash +$ pict-rs --save-to config.toml \ + --log-targets warn \ + --log-format json \ + --console-address '127.0.0.1:6669' \ + --opentelemetry-url 'http://localhost:4317' \ + run \ + -a '127.0.0.1:1234' \ + --api-key "$API_KEY" \ + --client-pool-size 10 \ + --media-max-width 200 \ + --media-max-height 200 \ + --media-max-file-size 1 \ + --media-enable-full-video true \ + --media-video-codec av1 \ + --media-format webp \ + filesystem \ + -p /opt/pict-rs/files \ + sled \ + -p /opt/pict-rs/metadata \ + -e /opt/pict-rs/exports +CTRL^C +$ cat config.toml +$ pict-rs -c config.toml run +``` + + +### Object Storage + +This is the biggest feature for pict-rs 0.4, but it isn't new. pict-rs 0.3 had initial support for +uploading to object storage hidden behind the `object-storage` compile flag. My docker images and +provided binaries did not enable this feature, so only folks compiling from source could even have +tried using it. In 0.4 I am now much more confident in the object storage support, and use it myself +for my personal setup. + +Although pict-rs now supports object storage as a backend for storing uploaded media, it doesn't +support generating URLs that bypass the pict-rs server for serving images. I've seen this +misconception a couple times during the release candidate process and I would like to put these +rumors to rest. + +Using object storage when you're in a cloud environment is sensible. The cost for adding object +storage is far less than for adding block storage, and although there are egress fees, they are +typically very low. Admins currently running pict-rs on a VPS should consider moving to object +storage if they find their disk usage is quickly growing. + +Using object storage does not fully remove pict-rs' dependence on a local disk. pict-rs stores its +metadata in an embedded key/value store called [sled](https://github.com/spacejam/sled/), which +means it still requires disk access to function. + +For new admins, using object storage from the start might be worth considering, but is not +required. pict-rs provides a built-in migration path for moving from filesystem storage to object +storage, which is documented in the [pict-rs +readme](https://git.asonix.dog/asonix/pict-rs/src/branch/main#user-content-filesystem-to-object-storage-migration). + + +### Backgrounded Uploads + +This feature is important for keeping pict-rs fast. Since media needs to be processed on upload, +upload times may grow long. This is especially true when considering pict-rs' other new features +like [full video](#full-video) and [image preprocessing](#image-preprocessing). + +Backgrounded uploads work by deferring the image processing work to a background task in pict-rs, +and responding to the upload HTTP request as soon as the provided file has been stored +successfully. Instead of returning a file alias like the inline upload endpoint, it instead returns +a UUID that represents the current upload process. That UUID can be given to a new `claim` endpoint +to retrieve the uploaded media's alias, or an error in the event the media was rejected or failed to +process. The claim endpoint uses a technique called "long polling" in order to notify clients on +completion of the background processing. Clients can request to claim uploaded media as soon as the +backgrounded endpoint reponds with an upload ID, and the server will hold the connection open for at +most 10 seconds while it waits for processing to complete. If processing did not complete in time, +the client can request again to claim the upload. + + +### Full Video + +pict-rs now supports uploading video with audio. In 0.3, pict-rs would always strip audio from +uploaded videos. This was intended as a gif-like feature, since pict-rs' primary use is for storing +pictures and not videos. By default, full video is disabled in pict-rs 0.4, but server admins can +opt into full video uploads by setting `PICTRS__MEDIA__ENABLE_FULL_VIDEO=true`. + +Along with supporting full video, an additional configuration option has been added to help keep +down on file sizes and processing time: `PICTRS__MEDIA__MAX_FRAME_COUNT`. This option limits how +many frames an uploaded video is allowed to have, rejecting the upload if it exceeds the provided +value. + +It is important to note that while pict-rs is capable of storing videos, it's processing +functionality is limited to just the video's thumbnail. + + +### Published Dependencies + +pict-rs 0.3's experimental object storage support relied on a custom fork of the `rust-s3` library, +which did useful things like enable re-use of an HTTP Client between connections, and making the +Client a `trait`, so it could be implemented for any arbitrary HTTP Client. Since then, the PR that +was meant to upstream those changes was closed, and so pict-rs needed to find a replacement that +suited it's needs. + +pict-rs now relies on the [`rusty-s3`](https://github.com/paolobarbolini/rusty-s3) library for +enabling object storage. This library takes a sans-io approach, and was simple to integrate with +`awc`, my HTTP Client of choice. + + +### HEAD Endpoints