pict-rs 0.4.1

Overview

Today I am releasing pict-rs 0.4.1. This includes a couple bugfixes and a few requested features. Most notably is the object-storage migration now uploads files concurrently. It should be much faster than before.

Features

Add Concurrency to Migration
Improved Error Messages
Configurable Object Storage Timeouts
Set Content-Type in Object Storage

Upgrade Notes

There's no significant changes from 0.4, so upgrading should be as simple as pulling a newer version of pict-rs.

Descriptions

Add Concurrency to Migration

Adding concurrency means changing how pict-rs kept track of migration progress. Previously it was setting a single flag with the "newest" hash that it had migrated, so a resuming migration would iterate until it found that hash before continuing linearly. Now, pict-rs keeps track of each individual file. That way it does not matter the order in which files are migrated.

The migration is hardcoded to try moving 32 files at a time, so ideally this would mean a 32x speedup over the previous linear version. Whether it actually ends up being so fast is yet to be seen.

Improved Error Messages

Previously errors encountered when failing to run a process were pretty opaque. pict-rs relies on running the ffmpeg, ffprobe, magick, and exiftool commands, and if it could not run them it would print something vague like "File not found" or "Permission Denied", without much context for what wasn't found or what was denied.

Now, pict-rs properly informs the admin that "Required command magick not found" or "Cannot run command ffmpeg due to invalid permissions on binary".

Additionally, the error messages returned in the API were not very useful either. "Error in ffmpeg" or "Error in imagemagick" are not super useful. Now pict-rs will return the "root cause" of errors, which is generally more helpful. There's still more work to be done here, so this topic will be mentioned in future releases.

Configurable Object Storage Timeouts

pict-rs now provides the ability to increase the HTTP timeout and the Signature Expiration for requests to object storage. This may not be super helpful, but if your object storage is particularly slow you can now increase the values.

The relevant variables are PICTRS__SToRE__SIGNATURE_EXPIRATION and PICTRS__STORE__CLIENT_TIMEOUT.

In toml, that would be

[store]
signature_expiration = 15
client_timeout = 30

Set Content-Type in Object Storage

This is not immediately useful, but it means that the object storage will begin knowing about the content-type of uploaded files, so in the future when pict-rs supports serving directly from object-storage it will provide a meaningful content-type header to browsers.

Alias Cleanup

There was a bug introduced in 0.4 that made using the "delete" endpoint not actually remove the file from storage. It would un-link the alias so it couldn't be accessed, but full deletion never happened. This was a result of me reordering a couple lines in the alias cleanup. 0.4.1 fixes this issue by un-reordering them.

ImageMagick Security Policy

The security policy used in the pict-rs docker container has been updated to allow for more file types. This makes the new JSON-based imagemagick and ffmpeg output parsers work, and might make one or two rejected files now no longer be rejected. I don't know the full extent to which this was a real problem in 0.4, but regardless it's better now.

Broken Pipe on Upload Error

Previously if a file was uploaded that was too large, it would be rejected immediately with a file size error. This sometimes resulted in clients receiving a Broken Pipe error on the connection instead of properly receiving an HTTP Response. This has been fixed in the actix-form-data library (that I also maintain) by ensuring that we read the entire upload (dropping the bytes immediately) before replying.

4.2 KiB Raw Blame History