pict-rs 0.5.2

Overview

A quick release to avoid a bug leading to incredibly large cleanup queue sizes and resulting in high CPU use. Upgrading to 0.5.2 is recommended for any deployment that recently upgraded to 0.5.0 or 0.5.1.

Bugfixes

At-Most Once Cleanup Jobs

Changes

Support TLS in Postgres Connections
Compiler Version Requirements

Upgrade Notes

There's no significant changes from 0.5.1, so upgrading should be as simple as pulling a new version of pict-rs.

Descriptions

At-Most Once Cleanup Jobs

pict-rs 0.5.2 now is smarter about queueing certain cleanup jobs. Previously, it might queue an arbitrary number of the same cleanup job while under significant load, resulting in many redundant jobs consuming CPU. The reason for this is pict-rs' strategy for handling variants and proxied media that hadn't been accessed within the configured timeframe was to queue a job to check every 30 seconds. This is fine in normal conditions, but can be a problem after upgrading to 0.5 from 0.4.

During the upgrade, pict-rs marks all variants as having been accessed at the time of the upgrade, since this information was not tracked in 0.4. This results in many variants being queued for cleanup exactly 1 week (or otherwise configured cache duration) after upgrading. When the cleanup queue fills with these cleanup jobs, it takes longer to process them all than the 30 second window between queues of the general cleanup jobs, resulting in redundant cleanup jobs that spawn the same redundant removal jobs, increasing the queue size by the number of outdated variants each time.

By adding an option to the queueing system to mark certain jobs as unique, pict-rs can now ensure that only one copy of these general cleanup jobs can be queued or running at any given time, ensuring that all individual variant cleanup jobs will be queued after a general cleanup job runs before the next general cleanup job is queued.

Support TLS in Postgres Connections

The postgres repo now supports some new options to enable connecting via TLS. If the postgres server's certificate is not generally trusted, a CA certificate can be provided to pict-rs to allow it to trust the postgres connection.

If the postgres endpoint uses TLS and has a publicly trusted certificate, the configuration can be updated to simply add use_tls = true

[repo]
type = 'postgres'
url = 'postgres://pictrs:1234@localhost:5432/pictrs'
use_tls = true

or with environment variables

PICTRS__REPO__TYPE=postgres
PICTRS__REPO__URL=postgres://pictrs:1234@localhost:5432/pictrs
PICTRS__REPO__USE_TLS=true

If the server uses a self-signed certificate, the CA's certificate can be added to pict-rs' trusted CAs.

[repo]
type = 'postgres'
url = 'postgres://pictrs:1234@localhost:5432/pictrs'
use_tls = true
certificate_file = '/path/to/ca/certificate.crt'