LemmyNet/pict-rs

Fork 0

mirror of https://git.asonix.dog/asonix/pict-rs synced 2024-11-20 11:21:14 +00:00

asonix ed8fce817b Update documentation around log_spans

2023-12-12 17:55:40 -06:00

22 KiB

Raw Permalink Blame History

pict-rs 0.5.0

TODO: quick message about 0.5 release

Overview

Upgrade Notes

The upgrade from pict-rs 0.4 to 0.5 might take some time. Any uploads that have not had their metadata extracted will be processed during this migration. In order to reduce the time required the migration, pict-rs version 0.4.6 can be run first, and the /internal/prepare_upgrade endpoint can be hit. This will tell pict-rs to extract the required metadata for the 0.5 upgrade in the background while 0.4 is still running, enabling the upgrade to 0.5 to proceed much faster after it completes. More information about this endpoint can be found in the 0.4.6 release document

If you're running the provided docker container with default configuration values, the 0.5 container can be pulled and launched with no changes. There is a migration from the 0.4 database format to the 0.5 database format that will occur automatically on first launch. This is also the process that will extract metadata for any uploads that have not had their metadata extracted yet.

If you have any custom configuration, note that some media-related configuration values have been moved. See Media Configuration below.

pict-rs 0.5 now supports postgres for metadata storage. It is possible to upgrade directly from 0.4 to 0.5 with postgres, rather than upgrading to 0.5 with sled and then making the migration later. This process is documented in the pict-rs readme.

A notable addition in 0.5 is upgrade configuration. Since the migration accesses media from the configured store, increasing the concurrency can be beneficial. By default, pict-rs will attempt to migrate 32 records at a time, but this value can be increased.

In the configuration file

[upgrade]
concurrency = 32

With environment variables

PICTRS__UPGRADE__CONCURRENCY=32

On the commandline

pict-rs run --upgrade-concurrency 32

More information can be found in the pict-rs 0.4 to 0.5 migration guide.

Descriptions

Postgres Repo

One of the most anticipated features of pict-rs 0.5 is the support for postgres for metadata storage. This makes pict-rs (mostly) stateless, enabling it to be scaled horizontally across multiple physical or virtual hosts. It also simplifies administration and backup processes for folks who are already running a postgres server for other reasons.

Configuring postgres is simple. The repo section of the pict-rs configuration now supports more than just sled.

In the configuration file

[repo]
type = 'postgres'
url = 'postgres://user:password@host:5432/db'

With environment variables

PICTRS__REPO__TYPE=postgres
PICTRS__REPO__URL=postgres://user:password@host:5432/db

On the commandline

pict-rs run filesystem -p ./data postgres -u 'postgres://user:password@host:5432/db'

It is possible to update from 0.4 directly to 0.5 with postgres, this process is documented in the pict-rs readme.

It is possible to migrate to postgres after first upgrading to 0.5. This is also ducmented in the pict-rs readme.

Media Proxy

pict-rs 0.5 supports caching media, and purging cached media after it is no longer necessary to keep. This behavior is accessed with pict-rs' new proxy query parameter at the /image/original endpoint, and the /image/process.{ext} endpoint. By passing the URL to an image to the proxy parameter, you can instruct pict-rs to download the media from that URL and cache it locally before serving it, or a processed version of it. The proxied media is stored for a configurable time limit, which resets on each access to the proxied media.

This value can be configured as follows

In the configuration file

[media.retention]
proxy = "7d" # units available are "m" for minutes, "h" for hours, "d" for days, and "y" for years

With environment variables

PICTRS__MEDIA__RETENTION__PROXY=7d

On the commandline

pict-rs run --media-retention-proxy 7d

More documentation can be found in the pict-rs readme and in the example pict-rs.toml.

Improved Animation Support

pict-rs 0.4 only supported gif for animated images. This decision had been made due to the format's poor support for higher-resolution animations with higher framerates compared to silent videos for a given file size. In pict-rs 0.5, animated images are now supported to the same extent as still images. Available formats for animated images include apng, avif, gif, and webp (although not jxl yet).

With this support for more animated file types, pict-rs no longer transcodes larger animations into videos. Instead, animated images will be left in their original format unless an override is configured, and the default values for maximum animation dimensions, file size, and frame count have been increased.

Configuration related to animations has been moved from [media.gif] to [media.animation] to better reflect what it applies to.

For all configuration options regarding animations, see the example pict-rs.toml.

Media Configuration

pict-rs 0.5 splits media configuration further, allowing images, animations, and videos to each specify their own maximum file size, dimensions, frame counts, etc, in addition to a few global limits. Relevant sections of configuration are [media], [media.image], [media.animation], and [media.video].

A list of updated and moved configuration values can be found below.

Image Changes

Old Environment Variable	New Environment Variable
`PICTRS__MEDIA__FORMAT`	`PICTRS__MEDIA__IMAGE__FORMAT`
`PICTRS__MEDIA__MAX_WIDTH`	`PICTRS__MEDIA__IMAGE__MAX_WIDTH`
`PICTRS__MEDIA__MAX_HEIGHT`	`PICTRS__MEDIA__IMAGE__MAX_HEIGHT`
`PICTRS__MEDIA__MAX_AREA`	`PICTRS__MEDIA__IMAGE__MAX_AREA`
	`PICTRS__MEDIA__IMAGE__MAX_FILE_SIZE`

Old TOML Value	New TOML Value
`[media] format`	`[media.image] format`
`[media] max_width`	`[media.image] max_width`
`[media] max_height`	`[media.image] max_height`
`[media] max_area`	`[media.image] max_area`
	`[media.image] max_file_size`

Animation Changes

Old Environment Variable	New Environment Variable
`PICTRS__MEDIA__GIF__MAX_WIDTH`	`PICTRS__MEDIA__ANIMATION__MAX_WIDTH`
`PICTRS__MEDIA__GIF__MAX_HEIGHT`	`PICTRS__MEDIA__ANIMATION__MAX_HEIGHT`
`PICTRS__MEDIA__GIF__MAX_AREA`	`PICTRS__MEDIA__ANIMATION__MAX_AREA`
`PICTRS__MEDIA__GIF__MAX_FILE_SIZE`	`PICTRS__MEDIA__ANIMATION__MAX_FILE_SIZE`
`PICTRS__MEDIA__GIF__MAX_FRAME_COUNT`	`PICTRS__MEDIA__ANIMATION__MAX_FRAME_COUNT`
	`PICTRS__MEDIA__ANIMATION__FORMAT`
	`PICTRS__MEDIA__ANIMATION__MAX_FILE_SIZE`

Old TOML Value	New TOML Value
`[media.gif] max_width`	`[media.animation] max_width`
`[media.gif] max_height`	`[media.animation] max_height`
`[media.gif] max_area`	`[media.animation] max_area`
`[media.gif] max_file_size`	`[media.animation] max_file_size`
`[media.gif] max_frame_count`	`[media.animation] max_frame_count`
	`[media.animation] format`
	`[media.animation] max_file_size`

Video Changes

Old Environment Variable	New Environment Variable
`PICTRS__MEDIA__ENABLE_SILENT_VIDEO`	`PICTRS__MEDIA__VIDEO__ENABLE`
`PICTRS__MEDIA__ENABLE_FULL_VIDEO`	`PICTRS__MEDIA__VIDEO__ALLOW_AUDIO`
`PICTRS__MEDIA__VIDEO_CODEC`	`PICTRS__MEDIA__VIDEO__VIDEO_CODEC`
`PICTRS__MEDIA__AUDIO_CODEC`	`PICTRS__MEDIA__VIDEO__AUDIO_CODEC`
`PICTRS__MEDIA__MAX_FRAME_COUNT`	`PICTRS__MEDIA__VIDEO__MAX_FRAME_COUNT`
`PICTRS__MEDIA__ENABLE_FULL_VIDEO`	`PICTRS__MEDIA__VIDEO__ALLOW_AUDIO`
	`PICTRS__MEDIA__VIDEO__MAX_WIDTH`
	`PICTRS__MEDIA__VIDEO__MAX_HEIGHT`
	`PICTRS__MEDIA__VIDEO__MAX_AREA`
	`PICTRS__MEDIA__VIDEO__MAX_FILE_SIZE`

Old TOML Value	New TOML Value
`[media] enable_silent_video`	`[media.video] enable`
`[media] enable_full_video`	`[media.video] allow_audio`
`[media] video_codec`	`[media.video] video_codec`
`[media] audio_codec`	`[media.video] audio_codec`
`[media] max_frame_count`	`[media.video] max_frame_count`
`[media] enable_full_video`	`[media.video] allow_audio`
	`[media.video] max_width`
	`[media.video] max_height`
	`[media.video] max_area`
	`[media.video] max_file_size`

Note that although each media type now includes its own MAX_FILE_SIZE configuration, the PICTRS__MEDIA__MAX_FILE_SIZE value still exists as a global limit for any file type.

In addition to all the configuration options mentioned above, there are now individual quality settings that can be configured for each image and animation type, as well as for video files. Please see the pict-rs.toml file for more information.

For all configuration options regarding media, see the example pict-rs.toml.

Improved Collision Resistance

Previously, pict-rs relied on a sha256 hash of uploaded media's bytes in order to detect which files where the same, and which were unique. I have not heard of any existing problem with this behavior, however, as pict-rs becomes more widely deployed I want to ensure that deduplication does not happen erroneously. In the event of a hash collision between two different pieces of media, pict-rs would consider them duplicates and discard the newly uploaded file in favor of the one it already stored. This would lead to the wrong media being served for a user. In order to reduce the chances of a collission, pict-rs 0.5 now includes the media's file length and a marker indicating it's content type as part of it's unique identifier. This maintains the behavior of identical media being deduplicated, while making collisions far less likely to occur.

Optional Video Transcoding

In pict-rs 0.4, all videos that were uploaded were transcoded from their original format to either pict-rs' default format, or a format specified with configuration. pict-rs 0.5 removes the default "vp9" video codec, and allows uploaded videos to maintain their original codec and format, provided pict-rs supports it. This means uploaded h264, h265, vp8, vp9, and av1 videos no longer need to be decoded & encoded during the upload process aside from detecting their metadata.

This behavior is enabled by default, but for administrators who have configured a non-default video codec, this can be enabled by removing the video_codec and audio_codec configurations.

Library API Changes

The library API for pict-rs has changed from 0.4. It now follows a "fluent programming" pattern of chaining methods to configure and launch the service from within another application. Documentation for the new library API can be found on docs.rs

For most cases, using pict-rs' ConfigSource as the entrypoint is recommended.

As a quick example,

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    pict_rs::ConfigSource::memory(serde_json::json!({
        "server": {
            "address": "127.0.0.1:8080"
        },
        "repo": {
            "type": "sled",
            "path": "./sled-repo"
        },
        "store": {
            "type": "filesystem",
            "path": "./files"
        }
    }))
    .init::<&str>(None)?
    .run_on_localset()
    .await?;

    Ok(())
}

Reject Malformed Deadlines

pict-rs supports setting a request deadline with the X-Request-Deadline header, where the value is a unix timestamp in nanoseconds. In pict-rs 0.4, requests with deadlines that could not be parsed were treated as requests without deadlines. in pict-rs 0.5 a malformed deadline will cause the request to be rejected with a 400 Bad Request error.

Logging Verbosity

pict-rs 0.5 no longer logs the opening and closing of tracing spans. This means the logs will now be mostly empty aside from the occasional warning or error. Errors should still provide ample debug information, so the decrease in verbosity should not negatively affect debugging of the program.

These spans can be re-enabled by setting a configuration option or passing a commandline flag.

[tracing.logging]
log_spans = true

PICTRS__TRACING__LOGGING__LOG_SPANS=true

pict-rs --log-spans

Quality Configuration

pict-rs 0.5 now supports configuring the quality and compression of transcoded media. New configuration sections have been introduced to allow setting values for each image format individually, as well as individual quality settings for different video resolutions.

For the specifics of quality configuration, see the example pict-rs.toml.

Read-Only Mode

pict-rs 0.5 has gained the ability to run in a "read-only" mode, which will forbid the application from accepting new uploaded media, creating new media variants, or updating any metadata. This is useful for complex environments that are capable of running a copy of the pict-rs application while performing maintenance on the "real" deployment.