Commit graph

86 commits

Author SHA1 Message Date
Dull Bananas
4428c61bea Merge remote-tracking branch 'upstream/main' into migration-runner 2024-07-26 19:50:46 +00:00
phiresky
a08642f813
federation: parallel sending per instance (#4623)
* federation: parallel sending

* federation: some comments

* lint and set force_write true when a request fails

* inbox_urls return vec

* split inbox functions into separate file

* cleanup

* extract sending task code to separate file

* move federation concurrent config to config file

* off by one issue

* improve msg

* fix both permanent stopping of federation queues and multiple creation of the same federation queues

* fix after merge

* lint fix

* Update crates/federate/src/send.rs

Co-authored-by: dullbananas <dull.bananas0@gmail.com>

* comment about reverse ordering

* remove crashable, comment

* comment

* move comment

* run federation tests twice

* fix test run

* prettier

* fix config default

* upgrade rust to 1.78 to fix diesel cli

* fix clippy

* delay

* add debug to make localhost urls not valid in ap crate, add some debug logs

* federation tests: ensure server stop after test and random activity id

* ci fix

* add test to federate 100 events

* fix send 100 test

* different data every time so activities are distinguishable

* allow out of order receives in test

* lint

* comment about https://github.com/LemmyNet/lemmy/pull/4623#discussion_r1565437391

* move sender for clarity, add comment

* move more things to members

* update test todo comment, use same env var as worker test but default to 1

* remove else below continue

* some more cleanup

* handle todo about smooth exit

* add federate inboxes collector tests

* lint

* actor max length

* don't reset fail count if activity skipped

* fix some comments

* reuse vars

* format

* Update .woodpecker.yml

* fix recheck time

* fix inboxes tests under fast mode

* format

* make i32 and ugly casts

* clippy

---------

Co-authored-by: dullbananas <dull.bananas0@gmail.com>
2024-07-21 11:50:50 -04:00
Daniel Lo Nigro
8abbd56400
Switch PostgreSQL container to use pgautoupgrade (#4892)
* Switch PostgreSQL container to use pgautoupgrade

This handles automatically upgrading the data files to newer versions of PostgreSQL.

* Fixing other uses of postgres:16-alpine image.

* Simplifying upgrade scripts.

---------

Co-authored-by: Dessalines <tyhou13@gmx.com>
2024-07-17 09:59:21 +02:00
Dull Bananas
7734ba47a1 Merge remote-tracking branch 'upstream/main' into migration-runner 2024-06-25 19:42:41 +00:00
phiresky
96b7afc0b1
upgrade rust to 1.78 to fix diesel cli (#4761) 2024-05-31 08:39:45 -04:00
dullbananas
1a7189787d
Update .woodpecker.yml 2024-05-23 13:05:57 -07:00
dullbananas
1e56bdef4d
Update .woodpecker.yml 2024-05-23 12:23:24 -07:00
dullbananas
f9af3dec38
Update .woodpecker.yml 2024-05-23 11:58:37 -07:00
dullbananas
f252de1a6c
Update .woodpecker.yml 2024-05-23 11:57:28 -07:00
dullbananas
728b6d49d7
create separate database in ci 2024-05-23 11:52:15 -07:00
dullbananas
0f373563b2
Update .woodpecker.yml 2024-05-23 11:27:18 -07:00
dullbananas
de1c688ba6
Update .woodpecker.yml 2024-05-23 10:11:08 -07:00
dullbananas
132877c833
Update .woodpecker.yml 2024-05-22 19:15:27 -07:00
dullbananas
6f6a5c9070
Update .woodpecker.yml 2024-05-22 18:44:58 -07:00
dullbananas
d2f772a496
Update .woodpecker.yml 2024-05-22 14:57:55 -07:00
Dull Bananas
16dcbc7a99 fix merge 2024-05-17 20:31:40 +00:00
Dull Bananas
1af517f6c4 Merge remote-tracking branch 'upstream/main' into migration-runner 2024-05-17 20:28:03 +00:00
Dull Bananas
5d8af0d476 fix 2024-05-17 20:21:13 +00:00
Dessalines
1a4aa3eaba
Stop using a diesel_cli docker image, use cargo-install in woodpecker. (#4723)
* Stop using a diesel_cli docker image, use cargo-binstall in woodpecker.

- The diesel_cli image is 500MB, and rebuilt daily. Much easier to use
  binstall to install it.

* Trying out a multiline var.

* Try sequence merges 1

* Try removing features.

* Try path fix.

* Abstracting diesel cli install.

* Try installing mysql and postgres.

* Try installing mysql and postgres 2.

* Try installing mysql and postgres 3.

* Try installing mysql and postgres 4.

* Try installing mysql and postgres 5.

* Try installing mysql and postgres 6.

* Try installing mysql and postgres 7.

* Try installing mysql and postgres 8.

* Try installing mysql and postgres 9.

* Try installing mysql and postgres 10.

* Try installing mysql and postgres 11.

* Try installing mysql and postgres 12.

* Try installing mysql and postgres 13.

* Add logging line.

* Add logging line 2.

* Add logging line 3.

* Add logging line 4.

* Removing binstall.

* Extract taplo into its own image, ignore binstall.

* Use a smaller taplo.

* taplo is the same image.
2024-05-16 16:41:36 -04:00
Dull Bananas
d0d8139ff0 migriation run command 2024-05-11 20:45:27 +00:00
SleeplessOne1917
b05f221565
Remove login step from publish to crates.io (#4674) 2024-04-30 00:07:03 +00:00
dullbananas
4ba6221e04
Move SQL triggers from migrations into reusable sql file (#4333)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* Create down.sql

* Create up.sql

* Create replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update utils.rs

* Update .woodpecker.yml

* Update sql_format_check.sh

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Create dump_schema.sh

* Update start_dev_db.sh

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* stuff

* Update replaceable_schema.sql

* Update .pg_format

* fmt

* stuff

* stuff (#21)

* Update replaceable_schema.sql

* Update up.sql

* Update replaceable_schema.sql

* fmt

* update cargo.lock

* stuff

* Update replaceable_schema.sql

* Remove truncate trigger because truncate is already restricted by foreign keys

* Update replaceable_schema.sql

* fix some things

* Update replaceable_schema.sql

* Update replaceable_schema.sql

* Update .woodpecker.yml

* stuff

* fix TG_OP

* Psql env vars

* try to fix combine_transition_tables parse error

* Revert "try to fix combine_transition_tables parse error"

This reverts commit 75d00a4626.

* refactor combine_transition_tables

* try to fix create_triggers

* fix some things

* try to fix combined_transition_tables

* fix sql errors

* update comment count in post trigger

* fmt

* Revert "fmt"

This reverts commit a5bcd0834b.

* Revert "update comment count in post trigger"

This reverts commit 0066a4b42b.

* fix everything

* Update replaceable_schema.sql

* actually fix everything

* refactor create_triggers

* fix

* add semicolons

* add is_counted function and fix incorrect bool operator in update_comment_count_from_post

* refactor comment trigger

* refactor post trigger

* fix

* Delete crates/db_schema/src/utils/series.rs

* subscribers_local

* edit migrations

* move migrations

* remove utils::series module declaration

* fix everything

* stuff

* Move sql to schema_setup dir

* utils.sql

* delete .pg_format

* Update .woodpecker.yml

* Update sql_format_check.sh

* Update .woodpecker.yml

* Merge remote-tracking branch 'upstream/main' into bliss

* fmt

* Create main.rs

* Update lib.rs

* Update main.rs

* Update .woodpecker.yml

* Update main.rs

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update triggers.sql

* YAY

* Update mod.rs

* Update Cargo.toml

* a

* Update Cargo.toml

* Update Cargo.toml

* Delete crates/db_schema/src/main.rs

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update utils.sql

* Update utils.sql

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update down.sql

* Update up.sql

* Update triggers.sql

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update triggers.sql

* Update down.sql

* Update .woodpecker.yml

* Update Cargo.toml

* Update .woodpecker.yml

* Update Cargo.toml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update mod.rs

* Update Cargo.toml

* Update mod.rs

* make dump_schema.sh executable

* fix dump_schema.sh

* defer

* diff dumps

* fmt

* Update utils.sql

* Update .woodpecker.yml

* use correct version for pg_dump

* Update .woodpecker.yml

* Update .woodpecker.yml

* change migration date

* atomic site_aggregates insert

* temporarily repeat tests in CI

* drop r schema in CI migration check

* show ReceivedActivity::create error

* move check_diesel_migration CI step

* Update .woodpecker.yml

* Update scheduled_tasks.rs

* Update scheduled_tasks.rs

* update cargo.lock

* move sql files

* move rank functions

* filter post_aggregates update

* fmt

* cargo fmt

* replace post_id with id

* update cargo.lock

* avoid locking rows that need no change in up.sql

* only run replaceable_schema if migrations were run

* debug ci test failure

* make replaceable_schema work in CI

* Update .woodpecker.yml

* remove println

* Use migration revert and git checkout

* Update schema_setup.rs

* Fix

* Update schema_setup.rs

* Update schema_setup.rs

* Update .woodpecker.yml

---------

Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-04-17 20:58:44 -04:00
Dessalines
705e86eb4c
Fixing docker release (#4592)
* Re-add notif on tag failure.

* Upping rust version.

* Version 0.19.4-beta.1a

* Try again.

* Version 0.19.4-beta.1b

* Removing unstable inspect.

* Version 0.19.4-beta.1c

* Remove use release cache.

* Trying to fix cargo publish 1.

* Version 0.19.4-beta.1d

* Re-adding publish release

* Version 0.19.4-beta.2

* Fixing workspace for lemmy_federate
2024-04-04 16:14:59 +02:00
Dessalines
a1d632e582
Re-add notif on tag failure. (#4591) 2024-04-04 10:21:31 +02:00
Dessalines
aaaa362b98
Remove latest tag for pgformatter. (#4589) 2024-04-03 17:29:24 -04:00
Dessalines
067332553d
Fixing woodpecker. (#4575) 2024-03-27 11:09:44 -04:00
Dessalines
95069d7648
Fixing some clippy and woodpecker lints. (#4565)
* Fixing some clippy and woodpecker lints.

* Try fixing woodpecker 1.

* Revert "Try fixing woodpecker 1."

This reverts commit 7c2020a08d.
2024-03-26 10:17:42 +01:00
Richard Schwab
80bfd23b4d
Ensure rustfmt is installed in cargo fmt CI job (#4466) 2024-02-26 11:22:22 +01:00
Dessalines
890565ca14
Deleting denied local_users older than a week. Fixes #4434 (#4448)
* Deleting denied local_users older than a week. Fixes #4434

* Addressing PR comments.

* Upping rust to 1.76

* Delete the person rows also.
2024-02-15 07:50:53 -05:00
Dessalines
9367cbdb00
Upgrading from postgres 15 -> 16-alpine. (#4426)
- Includes an upgrade script.
- Fixes #4406
2024-02-07 11:21:02 +01:00
Dessalines
eb0dc2fda4
Moving from yarn to pnpm. (#4414)
* Moving from yarn to pnpm.

* Prettier check.
2024-01-29 11:38:39 +01:00
dullbananas
f481a607d0
Show server output if federation tests fail in CI (#4389)
* Show server output if federation tests fail in CI

* dummy failure

* Revert dummy failure

* Update private_message.rs

* Fix errors

* Update private_message.rs

* Update private_message.rs

* Update .woodpecker.yml

* correct exit code

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update .woodpecker.yml

* Update prepare-drone-federation-test.sh

* Update .woodpecker.yml

* Update prepare-drone-federation-test.sh

* Update .woodpecker.yml

* Update prepare-drone-federation-test.sh

* Update private_message.rs

* Update lib.rs

* Update lib.rs

* Update session_middleware.rs

* Update session_middleware.rs

* Update .woodpecker.yml
2024-01-26 10:38:15 +01:00
dullbananas
ade1cb1495
Remove outdated comment from .woodpecker.yml (#4404) 2024-01-26 10:33:03 +01:00
dullbananas
759f6d8a9a
Better query plan viewing experience (#4285)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* replace get(0) with first()

* as_slice

* Update series.rs

* Update db_perf.sh

* Update and rename crates/db_schema/src/utils/series.rs to crates/db_perf/src/series.rs

* Update utils.rs

* Update main.rs

* Update main.rs

* Update .woodpecker.yml

* fmt main.rs

* Update .woodpecker.yml

* Instance::delete at end

* Update main.rs

* Update Cargo.toml

---------

Co-authored-by: Nutomic <me@nutomic.com>
2024-01-24 10:22:33 -05:00
Dessalines
e78fe5a34c
Removing group from woodpecker, as its deprecated. (#4387)
* Removing group from woodpecker, as its deprecated.

* Removing meltwater drone cache.
2024-01-22 10:57:07 +01:00
92b49dea3a
Increase CI timeout for restore cache (#4358) 2024-01-09 12:37:48 -05:00
Dessalines
3d9dda4677
Woodpecker rust 1.75 (#4355)
* Upgrading woodpecker rust to 1.75

* Trying to get publish working, try 1.

* Version 0.19.2-rc.3.publish1

* Temporarily disable arm builds.

* Version 0.19.2-rc.3.publish2

* Comment out arm blocks.

* Version 0.19.2-rc.3.publish3

* Moving back publish block to below.

* Upgrade to rasky lemmy-arm v0.2.0

* Version 0.19.2-rc.3.publish4

* Test bump for upgraded woodpecker.

* Revert "Test bump for upgraded woodpecker."

This reverts commit e2a5a99f33.

* Adding link to raskys repo.
2024-01-09 11:48:29 -05:00
952c162dac
Fix CI cache (#4276)
* Minor CI improvements (second attempt)

* test slow check condition

* remove rebuild cache condition for testing

* trigger quick ci check

* mkdir

* ls, remove steps

* exclude cargo home from prettier

* ci

* increase timeout

* more timeout

* even higher just for testing

* disable compression

* also increase restore timeout

* cleanup

* try again

* Update .woodpecker.yml

* Update .woodpecker.yml

* cleanup

* rerun ci

* cleanup
2024-01-04 11:40:41 -05:00
2b9d89057d
Reenable crates.io releases (fixes #4291) (#4294)
* Reenable crates.io releases (fixes #4291)

* disable docker release for now

* fixes

* fix indent

* move to top

* install libssl

* -y

* apt update

* debian version

* install cargo, git

* use cargo install

* move to orig position
2023-12-20 08:18:54 -05:00
Dessalines
e4b97ad8a9
Revert "Minor CI improvements (#4266)" (#4271)
This reverts commit af4d008ce4.
2023-12-14 18:36:33 -05:00
af4d008ce4
Minor CI improvements (#4266)
* Speed up SQL formatting in CI with parallel processing

* mess up formatting

* check permissions

* make repo world writable

* need git

* use debian image

* apt update

* perl image for faster install

* fix format

* run restore cache in parallel to format

* add drone-cache exit code

* revert sql format changes
2023-12-14 11:18:15 -05:00
4a55d4f871
Quick fix for Peertube federation (fixes #4261) (#4264)
* Quick fix for Peertube federation (fixes #4261)

* prettier

* apub assets should be included in slow check paths
2023-12-14 07:25:47 -05:00
95130370f0
Add CI check for ignored files (#4252)
* Add CI check for ignored files

* remove ignored files
2023-12-12 08:54:06 -05:00
Kroese
fc07ba2d3b
Fix entrypoint in Dockerfile (#4202)
* Fix entrypoint

* Delete docker/builders/lemmy-builder-arm64/docker-build.sh

* Delete docker/builders directory

* Remove exception for builder

* Remove publish_builder_arm64
2023-11-26 16:50:31 -05:00
Enzo Nocera
8a05c8f8be
fix: Cross-compilation to ARM64 (#4142)
* feat(docker/docs): explain how building lemmy works

Signed-off-by: Enzo NOCERA <enzo@nocera.eu>

* feat: add arm build

* review: rename script & fix typo

* feat(ci): allow cross platform compilation

* feat(ci): prettier

* fix(docker): fix base image name

* fix: add dockerfile in CI path

Signed-off-by: Enzo Nocera <enzo@nocera.eu>

* fix(docker): fix runner name

* fix(docker): fix builder base image

* fix(docker): fix builder base image platform

* fix(docker): avoid using the wrapper adduser/addgroup

* feat: avoid adding the whole docker directory in the build context

---------

Signed-off-by: Enzo NOCERA <enzo@nocera.eu>
Signed-off-by: Enzo Nocera <enzo@nocera.eu>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Nutomic <me@nutomic.com>
2023-11-24 12:52:19 -05:00
934fe7d1cb
Remove pictrs config section old_db (#4199)
* Remove pictrs config section `old_db` (ref #4194)

* ci

* remove cargo config comment

* ci

* Revert "remove cargo config comment"

This reverts commit 9389e93649.

* rename .cargo to .cargo_home
2023-11-24 11:26:21 -05:00
dullbananas
30f7135f32
Use rust 1.74.0 in CI and move clippy lints to Cargo.toml (#4170)
* ci rust 1.74.0

* do it

* Update .woodpecker.yml

* Update Cargo.toml

* Update lint.sh

* Update lib.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-11-21 08:51:22 -05:00
9007d49324
Remove unused deps (#4111)
* Remove unused deps

* Adding cargo-machete to woodpecker.

---------

Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-10-31 11:11:12 +01:00
Dessalines
645bf21d54
Moving to debian-based docker image. Fixes #3972 (#4004)
* Moving to debian-based docker image. Fixes #3972

* Fix cargo.toml format.

* Try nightly rust for cargo_fmt.

* Trying to fix cargo build.

* Trying to fix cargo build 2.

* Trying to fix federation tests.

* Trying to fix federation tests 2.

* Trying to fix federation tests 3.

* Try to fix aarch 1.

* Try to fix aarch 2.

* Try to fix aarch 3.

* Removing aarch

* Adding tests back in.

* Adding ca-certs, and debian non-root user

* Commenting arm image building.

* Fix platform on runner.

* Version 0.19.0-beta.6
2023-10-11 11:57:05 -04:00
5ef4807d09
Dont create :latest docker tag (fixes #3996) (#4016)
* Dont create :latest docker tag (fixes #3996)

* ci

* Update .woodpecker.yml

Co-authored-by: Richard Schwab <mail@w.tf-w.tf>

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Richard Schwab <mail@w.tf-w.tf>
2023-10-04 18:34:56 -04:00