From b3584a450bfdfaad9f90ade28447c270d33e47bb Mon Sep 17 00:00:00 2001 From: gribodyr <115513279+gribodyr@users.noreply.github.com> Date: Mon, 6 Nov 2023 10:57:13 +0000 Subject: [PATCH] Updated "Install from Scratch" guide (#279) * Updated "Install from Scratch" guide - PostgreSQL 16 installation - Pict-rs specific instructions (folders, ImageMagick AppImage problems in LXC containers, etc. - Update Lemmy version to 0.18.5 in the docs - Use /opt/lemmy for deployment (the Linux way (c)) - Update Nodejs and Yarn installation instructions per current updates from Nodesource - Bump Nginx version to 20 - Updated Nginx config file * Prettier * prettier once more * Fix typos * Address code review comments * Fix typos * Fix PostgreSQL installation snippet Update Nginx config * Remove obsolete paragraph * Fix annoying typo * Add actual port numbers Nix pictshare redirect from vhost config * More prettier! --------- Co-authored-by: gribodyr --- src/administration/from_scratch.md | 389 ++++++++++++++++++++++------- 1 file changed, 293 insertions(+), 96 deletions(-) diff --git a/src/administration/from_scratch.md b/src/administration/from_scratch.md index fff8e54..e8bb529 100644 --- a/src/administration/from_scratch.md +++ b/src/administration/from_scratch.md @@ -1,22 +1,76 @@ # From Scratch -These instructions are written for Ubuntu 20.04 / Ubuntu 22.04. +These instructions are written for Ubuntu 20.04 / Ubuntu 22.04. They are particularly useful when you'd like to setup a Lemmy container (e.g. LXC on Proxmox) and cannot use Docker. + +Lemmy is built from source in this guide, so this may take a while, especially on slow devices. For example, Lemmy v0.18.5 takes around 7 minutes to build on a quad core VPS. + +Installing and configuring Lemmy using this guide takes about 60-90 minutes. You might need to make yourself a fresh cup of coffee before you start. ## Installation -### Lemmy Backend +### Database -It is built from source, so this may take a while, especially on slow devices. For example, Lemmy v0.12.2 takes 17 minutes to build on a dual core VPS. If you prefer prebuilt binaries, use Docker or Ansible install methods. +For Ubuntu 20.04 the shipped PostgreSQL version is 12 which is not supported by Lemmy. So let's set up a newer one. +The most recent stable version of PostgreSQL is 16 at the time of writing this guide. -For the Rust compiles, it is ideal to use a non-privledged Linux account on your system. +#### Install dependencies -Install Rust by following the instructions on [Rustup](https://rustup.rs/) (using a non-privledged Linux account, it will install file in that user's home folder for rustup and cargo). +``` +sudo apt install -y wget ca-certificates pkg-config +wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - +sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" >> /etc/apt/sources.list.d/pgdg.list' +sudo apt update +sudo apt install libssl-dev libpq-dev postgresql +``` -Lemmy supports image hosting using [pict-rs](https://git.asonix.dog/asonix/pict-rs/). We need to install a couple of dependencies for this. You can also skip these steps if you don't require image hosting. **NOTE: Lemmy-ui will still allow users to attempt uploading images even if pict-rs is not configured, in this situation, the upload will fail and users will receive technical error messages.** +#### Setup Lemmy database -Depending on preference, pict-rs can be installed as a standalone application, or it can be embedded within Lemmy itself (see below). In both cases, pict-rs requires the `magick` command which comes with Imagemagick version 7, but Ubuntu 20.04 only comes with Imagemagick 6. So you need to install that command manually, eg from the [official website](https://imagemagick.org/script/download.php#linux). +Replace db-passwd with a unique password of your choice in the commands below. -```bash +``` +sudo -iu postgres psql -c "CREATE USER lemmy WITH PASSWORD 'db-passwd';" +sudo -iu postgres psql -c "CREATE DATABASE lemmy WITH OWNER lemmy;" +``` + +If you're migrating from an older version of Lemmy, the following might be required. + +``` +sudo -iu postgres psql -c "ALTER USER lemmy WITH SUPERUSER;" +``` + +Tune your PostgreSQL settings to match your hardware via [this guide](https://pgtune.leopard.in.ua/#/) + +#### Setup md5 auth + +Your Postgres config might need to be edited to allow password authentication instead of peer authentication. Simply add the following to your `pg_hba.conf`: + +``` +local lemmy lemmy md5 +``` + +### Install Rust + +For the Rust compiles, it is ideal to use a non-privileged Linux account on your system. Install Rust by following the instructions on [Rustup](https://rustup.rs/) (using a non-privileged Linux account, it will install file in that user's home folder for rustup and cargo). + +protobuf-compiler may be required for Ubuntu 20.04 or 22.04 installs, please report testing in lemmy-docs issues. + +``` +sudo apt install protobuf-compiler +``` + +### Setup pict-rs (Optional) + +You can skip this section if you don't require image hosting, but **NOTE that Lemmy-ui will still allow users to attempt uploading images even if pict-rs is not configured. In this situation, the upload will fail and users will receive technical error messages.** + +Lemmy supports image hosting using [pict-rs](https://git.asonix.dog/asonix/pict-rs/). We need to install a couple of dependencies for this. + +Depending on preference, pict-rs can be installed as a standalone application, or it can be embedded within Lemmy itself. In both cases, pict-rs requires the `magick` command which comes with Imagemagick version 7, but Ubuntu 20.04 only comes with Imagemagick 6. So you need to install that command manually, eg from the [official website](https://imagemagick.org/script/download.php#linux). + +**NOTE: on standard LXC containers an AppImage-based ImageMagick installation [will not work properly](https://github.com/LemmyNet/lemmy/issues/4112) with both embedded and standalone pict-rs. It uses FUSE which will emit "permission denied" errors when trying to upload an image through pict-rs. You must use alternative installation methods, such as [imei.sh](https://github.com/SoftCreatR/imei).** + +#### AppImage-based installation of ImageMagick + +``` sudo apt install ffmpeg exiftool libgexiv2-dev --no-install-recommends # save the file to a working folder it can be verified before copying to /usr/bin/ wget https://download.imagemagick.org/ImageMagick/download/binaries/magick @@ -26,52 +80,74 @@ sudo mv magick /usr/bin/ sudo chmod 755 /usr/bin/magick ``` -Install dependencies and setup database: +#### imei.sh-based installation of ImageMagick -```bash -sudo apt install pkg-config libssl-dev libpq-dev postgresql +Follow the instructions from the [official imei.sh page on GitHub](https://github.com/SoftCreatR/imei) -# replace db-passwd with a unique password of your choice -sudo -iu postgres psql -c "CREATE USER lemmy WITH PASSWORD 'db-passwd';" -sudo -iu postgres psql -c "CREATE DATABASE lemmy WITH OWNER lemmy;" -# NOTE: this may be required by migration, depending on version of Lemmy -# sudo -iu postgres psql -c "ALTER USER lemmy WITH SUPERUSER;" -# create user account on Linux for the lemmy_server application +#### Standalone pict-rs installation + +Since we're building stuff from source here, let's do the same for pict-rs. Follow the [instructions here](https://git.asonix.dog/asonix/pict-rs/#user-content-compile-from-source). + +However, the embedded pict-rs installation should work just fine for you. + +### Lemmy Backend + +#### Build the backend + +Create user account on Linux for the lemmy_server application + +``` sudo adduser lemmy --system --disabled-login --no-create-home --group ``` -Tune your PostgreSQL settings to match your hardware via https://pgtune.leopard.in.ua/#/ - Compile and install Lemmy, given the from-scratch intention, this will be done via GitHub checkout. This can be done by a normal unprivledged user (using the same Linux account you used for rustup). ```bash -# protobuf-compiler may be required for Ubuntu 22.04.2 installs, please report testing in lemmy-docs issues -sudo apt install protobuf-compiler git clone https://github.com/LemmyNet/lemmy.git lemmy cd lemmy -git checkout 0.18.2 +git checkout 0.18.5 git submodule init git submodule update --recursive --remote echo "pub const VERSION: &str = \"$(git describe --tag)\";" > "crates/utils/src/version.rs" -# These instructions assume you build pictrs independent, but it is -# OPTIONAL on next command: --features embed-pictrs -cargo build --release -# copy compiled binary to destination -sudo cp target/release/lemmy_server /usr/bin/lemmy_server ``` -Note: +When using the embedded pict-rs, use the following build command: -- Lemmy currently only supports non-SSL connections to databases. More info [here](https://github.com/LemmyNet/lemmy/issues/3007). -- Your postgres config might need to be edited to allow password authentication instead of peer authentication. Simply add: - ``` - local lemmy lemmy md5 - # Add the line above. - # Do not add the line below, it should already exist in your pg_hba.conf in some form. - local all all peer - ``` +``` +cargo build --release --features embed-pictrs +``` -Minimal Lemmy config, put this in `/etc/lemmy/lemmy.hjson` (see [here](https://github.com/LemmyNet/lemmy/blob/main/config/config.hjson) for more config options). Run `chown lemmy:lemmy /etc/lemmy/ -R` to set the correct owner. +Otherwise, just move forward with the following. + +``` +cargo build --release +``` + +#### Deployment + +Because we should [follow the Linux way](https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html), we should use the `/opt` directory to colocate the backend, frontend and pict-rs. + +``` +sudo mkdir /opt/lemmy +sudo mkdir /opt/lemmy/lemmy-server +sudo mkdir /opt/lemmy/pictrs +sudo mkdir /opt/lemmy/pictrs/files +sudo mkdir /opt/lemmy/pictrs/sled-repo +sudo mkdir /opt/lemmy/pictrs/old +sudo chown -R lemmy:lemmy /opt/lemmy +``` + +Note that it might not be the most obvious thing, but **creating the pictrs directories is not optional**. + +Then copy the binary. + +``` +sudo cp target/release/lemmy_server /opt/lemmy-server/lemmy_server +``` + +#### Configuration + +This is the minimal Lemmy config, put this in `/opt/lemmy/lemmy-server/lemmy.hjson` (see [here](https://github.com/LemmyNet/lemmy/blob/main/config/defaults.hjson) for more config options). ```hjson { @@ -92,22 +168,29 @@ Minimal Lemmy config, put this in `/etc/lemmy/lemmy.hjson` (see [here](https://g } ``` -Systemd unit file, so that Lemmy automatically starts and stops, logs are handled via journalctl etc. Put this file into /etc/systemd/system/lemmy.service, then run `systemctl enable lemmy` and `systemctl start lemmy`. +Set the correct owner + +``` +chown -R lemmy:lemmy /opt/lemmy/ +``` + +#### Server daemon + +Add a systemd unit file, so that Lemmy automatically starts and stops, logs are handled via journalctl etc. Put this file into /etc/systemd/system/lemmy.service. ``` [Unit] -Description=Lemmy - A link aggregator for the fediverse +Description=Lemmy Server After=network.target [Service] User=lemmy -ExecStart=/usr/bin/lemmy_server -Environment=LEMMY_CONFIG_LOCATION=/etc/lemmy/lemmy.hjson -# remove these two lines if you don't need pict-rs -Environment=PICTRS_PATH=/var/lib/pictrs +ExecStart=/opt/lemmy/lemmy-server/lemmy_server +Environment=LEMMY_CONFIG_LOCATION=/opt/lemmy/server/lemmy.hjson Environment=PICTRS_ADDR=127.0.0.1:8080 -Environment=RUST_LOG=info +Environment=RUST_LOG="info" Restart=on-failure +WorkingDirectory=/opt/lemmy # Hardening ProtectSystem=yes @@ -119,47 +202,72 @@ NoNewPrivileges=true WantedBy=multi-user.target ``` -If you did everything right, the Lemmy logs from `journalctl -u lemmy` should show "Starting http server at 127.0.0.1:8536". You can also run `curl localhost:8536/api/{version}/site` which should give a successful response, looking like `{"site_view":null,"admins":[],"banned":[],"online":0,"version":"unknown version","my_user":null,"federated_instances":null}`. For pict-rs, run `curl 127.0.0.1:8080` and ensure that it outputs nothing (particularly no errors). +If you need debug output in the logs, change the RUST_LOG line in the file above to -### Install lemmy-ui (web frontend) +``` +Environment=RUST_LOG="debug,lemmy_server=debug,lemmy_api=debug,lemmy_api_common=debug,lemmy_api_crud=debug,lemmy_apub=debug,lemmy_db_schema=debug,lemmy_db_views=debug,lemmy_db_views_actor=debug,lemmy_db_views_moderator=debug,lemmy_routes=debug,lemmy_utils=debug,lemmy_websocket=debug" +``` -Install dependencies (nodejs and yarn in Ubuntu 20.04 / Ubuntu 22.04 repos are too old) +Then run -```bash -# https://classic.yarnpkg.com/en/docs/install/#debian-stable +``` +sudo systemctl daemon-reload +sudo systemctl enable lemmy +sudo systemctl start lemmy +``` + +If you did everything right, the Lemmy logs from `sudo journalctl -u lemmy` should show "Starting http server at 127.0.0.1:8536". You can also run `curl localhost:8536/api/v3/site` which should give a successful response, looking like `{"site_view":null,"admins":[],"banned":[],"online":0,"version":"unknown version","my_user":null,"federated_instances":null}`. For pict-rs, run `curl 127.0.0.1:8080` and ensure that it outputs nothing (particularly no errors). + +### Lemmy Front-end (lemmy-ui) + +#### Install dependencies + +Nodejs and yarn in Ubuntu 20.04 / Ubuntu 22.04 repos are too old, so let's install Node 20. + +``` +# yarn curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list -# https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions -curl -fsSL https://deb.nodesource.com/setup_12.x | sudo -E bash - + +# nodejs +sudo apt install -y ca-certificates curl gnupg +sudo mkdir -p /etc/apt/keyrings +curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg +echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_20.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list + +sudo apt update sudo apt install nodejs yarn ``` -Clone the git repo, checkout the version you want (0.18.2 in this case), and compile it. +#### Build the front-end -```bash -mkdir /var/lib/lemmy-ui -cd /var/lib/lemmy-ui -chown lemmy:lemmy . +Clone the git repo, checkout the version you want (0.18.5 in this case), and compile it. + +``` # dont compile as admin +cd /opt/lemmy sudo -u lemmy bash -git clone https://github.com/LemmyNet/lemmy-ui.git --recursive . -git checkout 0.18.2 # replace with the version you want to install +git clone https://github.com/LemmyNet/lemmy-ui.git --recursive +cd lemmy-ui +git checkout 0.18.5 # replace with the version you want to install yarn install --pure-lockfile yarn build:prod exit ``` -Add another systemd unit file, this time for lemmy-ui. You need to replace example.com with your actual domain. Put the file in `/etc/systemd/system/lemmy-ui.service`, then run `systemctl enable lemmy-ui` and `systemctl start lemmy-ui`. More UI-related variables can be [found here](https://github.com/LemmyNet/lemmy-ui#configuration). +#### UI daemon + +Add another systemd unit file, this time for lemmy-ui. You need to replace `example.com` with your actual domain. Put the file in `/etc/systemd/system/lemmy-ui.service` ``` [Unit] -Description=Lemmy UI - Web frontend for Lemmy +Description=Lemmy UI After=lemmy.service Before=nginx.service [Service] User=lemmy -WorkingDirectory=/var/lib/lemmy-ui +WorkingDirectory=/opt/lemmy/lemmy-ui ExecStart=/usr/bin/node dist/js/server.js Environment=LEMMY_UI_LEMMY_INTERNAL_HOST=localhost:8536 Environment=LEMMY_UI_LEMMY_EXTERNAL_HOST=example.com @@ -175,6 +283,16 @@ NoNewPrivileges=true WantedBy=multi-user.target ``` +More UI-related variables can be [found here](https://github.com/LemmyNet/lemmy-ui#configuration). + +Then run. + +``` +sudo systemctl daemon-reload +sudo systemctl enable lemmy-ui +sudo systemctl start lemmy-ui +``` + If everything went right, the command `curl -I localhost:1234` should show `200 OK` at the top. ### Configure reverse proxy and TLS @@ -182,31 +300,125 @@ If everything went right, the command `curl -I localhost:1234` should show `200 Install dependencies ```bash -apt install nginx certbot python3-certbot-nginx +sudo apt install nginx certbot python3-certbot-nginx ``` Request Let's Encrypt TLS certificate (just follow the instructions) ```bash -certbot certonly --nginx +sudo certbot certonly --nginx ``` -Let's Encrypt certificates should be renewed automatically, so add the line below to your crontab, by running `sudo crontab -e`. Replace example.com with your actual domain. +Let's Encrypt certificates should be renewed automatically, so add the line below to your crontab, by running `sudo crontab -e`. Replace `example.com` with your actual domain. ``` @daily certbot certonly --nginx --cert-name example.com -d example.com --deploy-hook 'nginx -s reload' ``` -Finally, add the nginx config. After downloading, you need to replace some variables in the file. +Finally, add the Nginx virtual host config file. Copy paste the file below to `/etc/nginx/sites-enabled/lemmy.conf` -```bash -curl https://raw.githubusercontent.com/LemmyNet/lemmy-ansible/main/templates/nginx.conf \ - --output /etc/nginx/sites-enabled/lemmy.conf -# put your actual domain instead of example.com -sed -i -e 's/{{domain}}/example.com/g' /etc/nginx/sites-enabled/lemmy.conf -sed -i -e 's/{{lemmy_port}}/8536/g' /etc/nginx/sites-enabled/lemmy.conf -sed -i -e 's/{{lemmy_ui_port}}/1234/g' /etc/nginx/sites-enabled/lemmy.conf -nginx -s reload +``` +limit_req_zone $binary_remote_addr zone={{domain}}_ratelimit:10m rate=1r/s; + +server { + listen 80; + listen [::]:80; + server_name {{domain}}; + location /.well-known/acme-challenge/ { + root /var/www/certbot; + } + location / { + return 301 https://$host$request_uri; + } +} + +server { + listen 443 ssl http2; + listen [::]:443 ssl http2; + server_name {{domain}}; + + ssl_certificate /etc/letsencrypt/live/{{domain}}/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/{{domain}}/privkey.pem; + + # Various TLS hardening settings + # https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256'; + ssl_session_timeout 10m; + ssl_session_cache shared:SSL:10m; + ssl_session_tickets on; + ssl_stapling on; + ssl_stapling_verify on; + + # Hide nginx version + server_tokens off; + + # Enable compression for JS/CSS/HTML bundle, for improved client load times. + # It might be nice to compress JSON, but leaving that out to protect against potential + # compression+encryption information leak attacks like BREACH. + gzip on; + gzip_types text/css application/javascript image/svg+xml; + gzip_vary on; + + # Only connect to this site via HTTPS for the two years + add_header Strict-Transport-Security "max-age=63072000"; + + # Various content security headers + add_header Referrer-Policy "same-origin"; + add_header X-Content-Type-Options "nosniff"; + add_header X-Frame-Options "DENY"; + add_header X-XSS-Protection "1; mode=block"; + + # Upload limit for pictrs + client_max_body_size 20M; + + # frontend + location / { + # The default ports: + + set $proxpass "http://0.0.0.0:1234"; + if ($http_accept ~ "^application/.*$") { + set $proxpass "http://0.0.0.0:1234"; + } + if ($request_method = POST) { + set $proxpass "http://0.0.0.0:8536"; + } + proxy_pass $proxpass; + + rewrite ^(.+)/+$ $1 permanent; + + # Send actual client IP upstream + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } + + # backend + location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) { + proxy_pass http://0.0.0.0:8536; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Rate limit + limit_req zone={{domain}}_ratelimit burst=30 nodelay; + + # Add IP forwarding headers + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } +} + +access_log /var/log/nginx/access.log combined; +``` + +And then replace some variables in the file. Put your actual domain instead of example.com + +``` +sudo sed -i -e 's/{{domain}}/example.com/g' /etc/nginx/sites-enabled/lemmy.conf +sudo systemctl reload nginx ``` Now open your Lemmy domain in the browser, and it should show you a configuration screen. Use it to create the first admin user and the default community. @@ -217,12 +429,12 @@ Now open your Lemmy domain in the browser, and it should show you a configuratio Compile and install lemmy_server changes. This compile can be done by a normal unprivledged user (using the same Linux account you used for rustup and first install of Lemmy). -```bash +``` rustup update cd lemmy git checkout main git pull --tags -git checkout 0.18.2 # replace with version you are updating to +git checkout 0.18.5 # replace with version you are updating to git submodule update --recursive --remote echo "pub const VERSION: &str = \"$(git describe --tag)\";" > "crates/utils/src/version.rs" # These instructions assume you build pictrs independent, but it is @@ -230,39 +442,24 @@ echo "pub const VERSION: &str = \"$(git describe --tag)\";" > "crates/utils/src/ cargo build --release # copy compiled binary to destination # the old file will be locked by the already running application, so this sequence is recommended: -sudo -- sh -c 'systemctl stop lemmy && cp target/release/lemmy_server /usr/bin/lemmy_server && systemctl start lemmy' +sudo -- sh -c 'systemctl stop lemmy && cp target/release/lemmy_server /opt/lemmy/lemmy-server/lemmy_server && systemctl start lemmy' ``` ### Lemmy UI -```bash -cd /var/lib/lemmy-ui +``` +cd /opt/lemmy/lemmy-ui sudo -u lemmy bash git checkout main git pull --tags -git checkout 0.18.2 # replace with the version you are updating to +git checkout 0.18.5 # replace with the version you are updating to git submodule update yarn install --pure-lockfile yarn build:prod exit -systemctl restart lemmy-ui +sudo systemctl restart lemmy-ui ``` ### Pict-rs -If you did **not** use the `--features embed-pictrs` flag, then this script below is necessary for installing/updating Pict-rs as a standalone server. -Otherwise, pict-rs should update with lemmy_server. - -```bash -rustup update -cd /var/lib/pictrs-source -git checkout main -git pull --tags -# check docker-compose.yml for pict-rs version used by lemmy -# https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/docker-compose.yml#L43 -git checkout v0.2.6-r2 # replace with the version you want to install -# or simply add the bin folder to your $PATH -$HOME/.cargo/bin/cargo build --release -cp target/release/pict-rs /usr/bin/ -systemctl restart pictrs -``` +If you used the `--features embed-pictrs` flag, pict-rs should update with lemmy_server. Otherwise, refer to [pict-rs documentation](https://git.asonix.dog/asonix/pict-rs) for instructions on upgrading.