pict-rs

a simple image hosting service

Links

• Find the code on gitea
• Join the discussion on matrix
• Hit me up on mastodon

Usage

Running

pict-rs 0.4.0-alpha.1
asonix <asonix@asonix.dog>
A simple image hosting service

USAGE:
    pict-rs [OPTIONS] <SUBCOMMAND>

OPTIONS:
    -c, --config-file <CONFIG_FILE>
            Path to the pict-rs configuration file

        --console-address <CONSOLE_ADDRESS>
            Address and port to expose tokio-console metrics

        --console-buffer-capacity <CONSOLE_BUFFER_CAPACITY>
            Capacity of the console-subscriber Event Buffer

    -h, --help
            Print help information

        --log-format <LOG_FORMAT>
            Format of logs printed to stdout

        --log-targets <LOG_TARGETS>
            Log levels to print to stdout, respects RUST_LOG formatting

        --old-db-path <OLD_DB_PATH>
            Path to the old pict-rs sled database

        --opentelemetry-service-name <OPENTELEMETRY_SERVICE_NAME>
            Service Name to use for OpenTelemetry

        --opentelemetry-targets <OPENTELEMETRY_TARGETS>
            Log levels to use for OpenTelemetry, respects RUST_LOG formatting

        --opentelemetry-url <OPENTELEMETRY_URL>
            URL to send OpenTelemetry metrics

        --save-to <SAVE_TO>
            File to save the current configuration for reproducible runs

    -V, --version
            Print version information

SUBCOMMANDS:
    filesystem        Migrate from the provided filesystem storage
    help              Print this message or the help of the given subcommand(s)
    object-storage    Migrate from the provided object storage
    run               Runs the pict-rs web server

pict-rs-run 
Runs the pict-rs web server

USAGE:
    pict-rs run [OPTIONS] [SUBCOMMAND]

OPTIONS:
    -a, --address <ADDRESS>
            The address and port to bind the pict-rs web server

        --api-key <API_KEY>
            The API KEY required to access restricted routes

    -h, --help
            Print help information

        --media-enable-silent-video <MEDIA_ENABLE_SILENT_VIDEO>
            Whether to enable GIF and silent MP4 uploads. Full videos are unsupported

        --media-filters <MEDIA_FILTERS>
            Which media filters should be enabled on the `process` endpoint

        --media-format <MEDIA_FORMAT>
            Enforce uploaded media is transcoded to the provided format

        --media-max-area <MEDIA_MAX_AREA>
            The maximum area, in pixels, for uploaded media

        --media-max-file-size <MEDIA_MAX_FILE_SIZE>
            The maximum size, in megabytes, for uploaded media

        --media-max-height <MEDIA_MAX_HEIGHT>
            The maximum height, in pixels, for uploaded media

        --media-max-width <MEDIA_MAX_WIDTH>
            The maximum width, in pixels, for uploaded media

        --media-skip-validate-imports <MEDIA_SKIP_VALIDATE_IMPORTS>
            Whether to validate media on the "import" endpoint

        --worker-id <WORKER_ID>
            

SUBCOMMANDS:
    filesystem        Run pict-rs with filesystem storage
    help              Print this message or the help of the given subcommand(s)
    object-storage    Run pict-rs with object storage

Try running help commands for more runtime configuration options

$ pict-rs run filesystem -h
$ pict-rs run object-storage -h
$ pict-rs run filesystem sled -h
$ pict-rs run object-storage sled -h

See pict-rs.toml for more configuration

Example:

Running on all interfaces, port 8080, storing data in /opt/data

$ ./pict-rs -a 0.0.0.0:8080 -p /opt/data run

Running locally, port 9000, storing data in data/, and converting all uploads to PNG

$ ./pict-rs -a 127.0.0.1:9000 -p data/ -f png run

Running locally, port 8080, storing data in data/, and only allowing the thumbnail and identity filters

$ ./pict-rs -a 127.0.0.1:8080 -p data/ -w thumbnail identity run

Running from a configuration file

$ ./pict-rs -c ./pict-rs.toml run

Migrating to object storage from filesystem storage (both storages must be configured in pict-rs.toml)

$ ./pict-rs -c ./pict-rs.toml --store filesystem migrate-store object-storage

Dumping commandline flags to a toml file

$ ./pict-rs -p data/ --store object-storage --object-storage-bucket-name pict-rs --object-storage-region us-east-1 dump pict-rs.toml

Docker

Run the following commands:

# Create a folder for the files (anywhere works)
$ mkdir ./pict-rs
$ cd ./pict-rs
$ mkdir -p volumes/pictrs
$ sudo chown -R 991:991 volumes/pictrs
$ wget https://git.asonix.dog/asonix/pict-rs/raw/branch/main/docker/prod/docker-compose.yml
$ sudo docker-compose up -d

Note

• pict-rs makes use of the system's temporary folder. This is generally /tmp on linux
• pict-rs makes use of an imagemagick security policy at /usr/lib/ImageMagick-$VERSION/config-Q16HDRI/policy.xml

Docker Development

The development system loads a rust environment inside a docker container with the neccessary dependencies already present

$ git clone https://git.asonix.dog/asonix/pict-rs
$ cd pict-rs/docker/dev
$ ./dev.sh
$ check # runs cargo check
$ build # runs cargo build

Development environments are provided for amd64, arm32v7, and arm64v8. By default dev.sh will load into the contianer targetting amd64, but arch arguments can be passed to change the target.

$ ./dev.sh arm32v7
$ build

Note

Since moving to calling out to ffmpeg, imagemagick, and exiftool's binaries instead of binding directly, the dev environment now only contains enough to build static binaries, but not run the pict-rs program. I have personally been using alpine and arch linux to test the results. Here's how I have been doing it:

With Arch

$ sudo docker run --rm -it -p 8080:8080 -v "$(pwd):/mnt" archlinux:latest
# pacman -Syu imagemagick ffmepg perl-image-exiftool
# ln -s /usr/bin/vendor_perl/exiftool /usr/bin/exiftool
# cp /mnt/docker/prod/root/usr/lib/ImageMagick-7.0.11/config-Q16HDRI/policy.xml /usr/lib/ImageMagick-7.1.0/config-Q16HDRI/
# RUST_LOG=debug /mnt/target/x86_64-unknown-linux-musl/debug/pict-rs -p /mnt/data

With Alpine

$ sudo docker run --rm -it -p 8080:8080 -v "$(pwd):/mnt alpine:3.14
# apk add imagemagick ffmpeg exiftool
# cp /mnt/docker/prod/root/usr/lib/ImageMagick-7.0.11/config-Q16HDRI/policy.xml /usr/lib/ImageMagick-7.0.11/config-Q16HDRI/
# RUST_LOG=debug /mnt/target/x86_64-unknown-linux-musl/debug/pict-rs -p /mnt/data

API

pict-rs offers the following endpoints:

• POST /image for uploading an image. Uploaded content must be valid multipart/form-data with an image array located within the images[] key

  This endpoint returns the following JSON structure on success with a 201 Created status

  {
      "files": [
          {
              "delete_token": "JFvFhqJA98",
              "file": "lkWZDRvugm.jpg"
          },
          {
              "delete_token": "kAYy9nk2WK",
              "file": "8qFS0QooAn.jpg"
          },
          {
              "delete_token": "OxRpM3sf0Y",
              "file": "1hJaYfGE01.jpg"
          }
      ],
      "msg": "ok"
  }
  

• GET /image/download?url=... Download an image from a remote server, returning the same JSON payload as the POST endpoint

• GET /image/original/{file} for getting a full-resolution image. file here is the file key from the /image endpoint's JSON

• GET /image/details/original/{file} for getting the details of a full-resolution image. The returned JSON is structured like so:

  {
      "width": 800,
      "height": 537,
      "content_type": "image/webp",
      "created_at": [
          2020,
          345,
          67376,
          394363487
      ]
  }
  

• GET /image/process.{ext}?src={file}&... get a file with transformations applied. existing transformations include

  • identity=true: apply no changes
  • blur={float}: apply a gaussian blur to the file
  • thumbnail={int}: produce a thumbnail of the image fitting inside an {int} by {int} square using raw pixel sampling
  • resize={int}: produce a thumbnail of the image fitting inside an {int} by {int} square using a Lanczos2 filter. This is slower than sampling but looks a bit better in some cases
  • crop={int-w}x{int-h}: produce a cropped version of the image with an {int-w} by {int-h} aspect ratio. The resulting crop will be centered on the image. Either the width or height of the image will remain full-size, depending on the image's aspect ratio and the requested aspect ratio. For example, a 1600x900 image cropped with a 1x1 aspect ratio will become 900x900. A 1600x1100 image cropped with a 16x9 aspect ratio will become 1600x900.

  Supported ext file extensions include png, jpg, and webp

  An example of usage could be

  GET /image/process.jpg?src=asdf.png&thumbnail=256&blur=3.0
  

  which would create a 256x256px JPEG thumbnail and blur it

• GET /image/details/process.{ext}?src={file}&... for getting the details of a processed image. The returned JSON is the same format as listed for the full-resolution details endpoint.

• DELETE /image/delete/{delete_token}/{file} or GET /image/delete/{delete_token}/{file} to delete a file, where delete_token and file are from the /image endpoint's JSON

The following endpoints are protected by an API key via the X-Api-Token header, and are disabled unless the --api-key option is passed to the binary or the PICTRS_API_KEY environment variable is set.

A secure API key can be generated by any password generator.

• POST /internal/import for uploading an image while preserving the filename. This should not be exposed to the public internet, as it can cause naming conflicts with saved files. The upload format and response format are the same as the POST /image endpoint.

• POST /internal/purge?... Purge a file by it's filename or alias. This removes all aliases and files associated with the query.

  • ?file=asdf.png purge by filename
  • ?alias=asdf.png purge by alias

  This endpoint returns the following JSON

  {
      "msg": "ok",
      "aliases": ["asdf.png"]
  }
  

• GET /internal/aliases?... Get the aliases for a file by it's filename or alias

  • ?file={filename} get aliases by filename
  • ?alias={alias} get aliases by alias

  This endpiont returns the same JSON as the purge endpoint

• GET /internal/filename?alias={alias} Get the filename for a file by it's alias This endpoint returns the following JSON

  {
      "msg": "ok",
      "filename": "asdf.png"
  }
  

Additionally, all endpoints support setting deadlines, after which the request will cease processing. To enable deadlines for your requests, you can set the X-Request-Deadline header to an i128 value representing the number of nanoseconds since the UNIX Epoch. A simple way to calculate this value is to use the time crate's OffsetDateTime::unix_timestamp_nanos method. For example,

// set deadline of 1ms
let deadline = time::OffsetDateTime::now_utc() + time::Duration::new(0, 1_000);

let request = client
    .get("http://pict-rs:8080/image/details/original/asdfghjkla.png")
    .insert_header(("X-Request-Deadline", deadline.unix_timestamp_nanos().to_string())))
    .send()
    .await;

Contributing

Feel free to open issues for anything you find an issue with. Please note that any contributed code will be licensed under the AGPLv3.

FAQ

Question: I want to configure it with yaml instead of toml

Answer: That's not a question, but you can configure pict-rs with json, hjson, yaml, ini, or toml. Writing configs in other formats is left as an exercise to the reader.

License

Copyright © 2022 Riley Trautman

pict-rs is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

pict-rs is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. This file is part of pict-rs.

You should have received a copy of the GNU General Public License along with pict-rs. If not, see http://www.gnu.org/licenses/.