Commit graph

42 commits

Author SHA1 Message Date
phiresky
a08642f813
federation: parallel sending per instance (#4623)
* federation: parallel sending

* federation: some comments

* lint and set force_write true when a request fails

* inbox_urls return vec

* split inbox functions into separate file

* cleanup

* extract sending task code to separate file

* move federation concurrent config to config file

* off by one issue

* improve msg

* fix both permanent stopping of federation queues and multiple creation of the same federation queues

* fix after merge

* lint fix

* Update crates/federate/src/send.rs

Co-authored-by: dullbananas <dull.bananas0@gmail.com>

* comment about reverse ordering

* remove crashable, comment

* comment

* move comment

* run federation tests twice

* fix test run

* prettier

* fix config default

* upgrade rust to 1.78 to fix diesel cli

* fix clippy

* delay

* add debug to make localhost urls not valid in ap crate, add some debug logs

* federation tests: ensure server stop after test and random activity id

* ci fix

* add test to federate 100 events

* fix send 100 test

* different data every time so activities are distinguishable

* allow out of order receives in test

* lint

* comment about https://github.com/LemmyNet/lemmy/pull/4623#discussion_r1565437391

* move sender for clarity, add comment

* move more things to members

* update test todo comment, use same env var as worker test but default to 1

* remove else below continue

* some more cleanup

* handle todo about smooth exit

* add federate inboxes collector tests

* lint

* actor max length

* don't reset fail count if activity skipped

* fix some comments

* reuse vars

* format

* Update .woodpecker.yml

* fix recheck time

* fix inboxes tests under fast mode

* format

* make i32 and ugly casts

* clippy

---------

Co-authored-by: dullbananas <dull.bananas0@gmail.com>
2024-07-21 11:50:50 -04:00
a3c8761bed
Revert "Remove unneeded error "last successful id is higher than latest id" (fixes #4363) (#4486)" (#4806)
This reverts commit c895e57086.
2024-06-14 08:15:12 -04:00
phiresky
e8a7bb07a3
fix both permanent stopping of federation queues and multiple creation of the same federation queues (#4754)
Co-authored-by: Nutomic <me@nutomic.com>
2024-05-30 05:08:27 -04:00
phiresky
7d80a3c7d6
replace instanceid with domain (#4753) 2024-05-29 23:10:25 +02:00
Dessalines
abcfa266af
Fixing slowness in saved post fetching. #4756 (#4758)
* Fixing slowness in saved post fetching. #4756

* Also fix comment_view.rs
2024-05-29 17:03:42 -04:00
af034f3b5e
Unit tests and cleanup for outgoing federation code (#4733)
* test setup

* code cleanup

* cleanup

* move stats to own file

* basic test working

* cleanup

* processes test

* more test cases

* fmt

* add file

* add assert

* error handling

* fmt

* use instance id instead of domain for stats channel
2024-05-27 09:34:58 -04:00
dullbananas
ec77c00ef8
Fix lost separation caused by comment width change (#4739)
* Update post_view.rs

* Update structs.rs

* Update worker.rs

* Update worker.rs
2024-05-23 14:05:35 -04:00
d8dc38eb06
Upgrade dependencies (#4740) 2024-05-23 10:55:20 -04:00
c96017c009
Configure max comment width in clippy (#4738)
* Configure max comment width in clippy

* update default config
2024-05-23 08:46:26 -04:00
6b46a70535
Extra logging to debug duplicate activities (ref #4609) (#4726)
* Extra logging to debug duplicate activities (ref #4609)

* Fix logging for api tests

* fmt
2024-05-21 14:47:06 -04:00
Dessalines
d075acce43
Make all single-fetch database calls return an Option. (#4617)
- Diesel ordinarily throws an error when no results are returned for a
  single fetch, which is a bit confusing. This PR ensures that the
  missing value cases are all caught, and wrapped with new LemmyErrors,
  rather than diesel errors.
- Fixes #4601
2024-04-16 14:48:15 +02:00
1d0a6ac08f
Avoid breaking api change, reduce api cache duration (#4610)
* Dont mark site.public_key as `serde(skip)` to avoid breaking change (fixes #4605)

* Reduce cache duration for api
2024-04-09 10:10:20 -04:00
087684658a
Cache result of LocalSite::read to avoid unnecessary db calls (#4585)
* Cache result of LocalSite::read to avoid unnecessary db calls

* single const for cache duration

* clippy

* revert apub send changes

* clippy

* fmt
2024-04-03 17:38:31 -04:00
Dessalines
0f77951e05
Upgrading deps. (#4537)
* Upgrading deps.

* Addressing PR comments
2024-03-18 10:36:49 +01:00
c895e57086
Remove unneeded error "last successful id is higher than latest id" (fixes #4363) (#4486) 2024-03-01 11:31:37 -05:00
SleeplessOne1917
4b4b99aa78
Allow community mods to see votes in addition to admins (#4392)
* Allow community mods to see votes in addition to admins

* Use Post instead of PostView

---------

Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
2024-01-23 18:47:28 -05:00
b58da11fb7
Mark instance as alive after successful activity send (fixes #4039) (#4377)
* Mark instance as alive after successful activity send (fixes #4039)

* clippy

* Instance::update

* domain

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-19 09:40:12 -05:00
4ca63c5641
Move apub context to join-lemmy.org (fixes #4234) (#4302)
* Move apub context to join-lemmy.org (fixes #4234)

* Dont store federation context in sent_activity table

* include basic activitypub context

* update lib

* ci
2024-01-05 15:42:46 +01:00
phiresky
0e6669f617
no endless loop if queue too recent (#4349) 2024-01-04 13:28:26 -05:00
phiresky
024ab7d530
Fix federate loop (#4330)
* make activity channel infallible

* clippy

* federate: make cancellabletask loop itself
2024-01-03 13:30:06 -05:00
SleeplessOne1917
4ef00e068f
Prevent crates that shouldn't be published from being published (#4309)
* Prevent crates that shouldn't be published from being published

* Make dependent crates publishable

* Make dependent crates publishable

* fix toml formatting

* Removing publish=true

* Removing versioned deps.

---------

Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-12-20 09:11:58 -05:00
phiresky
6790b54d4d
make activity channel infallible (#4295) 2023-12-18 13:17:10 -05:00
phiresky
7353be5b68
federate-less-noisy (#4263) 2023-12-14 07:25:04 -05:00
c7ee53026f
Upgrade dependencies (#4248)
* Upgrade dependencies

* clippy
2023-12-12 08:56:33 -05:00
Dessalines
2b5a31b411
Upgrading deps. (#4196)
* Upgrading deps.

* Try to use native node-fetch function.

* Fixing image upload tests.
2023-11-24 10:29:41 +01:00
dullbananas
30f7135f32
Use rust 1.74.0 in CI and move clippy lints to Cargo.toml (#4170)
* ci rust 1.74.0

* do it

* Update .woodpecker.yml

* Update Cargo.toml

* Update lint.sh

* Update lib.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-11-21 08:51:22 -05:00
cetra3
1c03a86c17
Upgrade activitypub to 0.5.0-beta.5 (#4179)
* Remove anyhow test

* Update lock

* Fix lints hopefully!
2023-11-21 10:26:12 +01:00
phiresky
b9b65c9c18
add federation queue state to get_federated_instances api (#4104)
* add federation queue state to get_federated_instances api

* feature gate

* move retry sleep function

* move stuff around
2023-11-06 16:07:04 -05:00
9007d49324
Remove unused deps (#4111)
* Remove unused deps

* Adding cargo-machete to woodpecker.

---------

Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-10-31 11:11:12 +01:00
Dessalines
bd9739b5b7
Upgrading deps. (#4103) 2023-10-25 05:32:21 -04:00
phiresky
2c283d90ca Revert "Revert "re-remove debug comments""
This reverts commit 6d519ef376.
2023-09-21 11:45:33 +00:00
phiresky
dec25f9ee8 ensure overlap 2023-09-21 11:11:54 +00:00
phiresky
6d519ef376 Revert "re-remove debug comments"
This reverts commit ae37ca0137.
2023-09-21 11:07:36 +00:00
phiresky
ae37ca0137 re-remove debug comments 2023-09-20 15:33:09 +00:00
phiresky
a808d3208a use max(id) instead of seq max value to prevent uncommitted transactions from causing skipped activities 2023-09-20 14:57:18 +00:00
phiresky
1a0c866b51 logging temporary 2023-09-20 13:37:18 +00:00
phiresky
a59a94492d more debug log 2023-09-20 12:30:38 +00:00
phiresky
9e886fba4a delay shorter 2023-09-18 16:49:55 +00:00
phiresky
b09ffa7197 instead of changing fed delays in debug mode, change them via env variable 2023-09-13 11:23:13 +00:00
phiresky
dca43dcfd9 fix a bug where after an hour community follows would be overwritten completely by an incremental upgrade 2023-09-13 10:54:10 +00:00
4f1240487e Fixing high CPU usage on federation worker recheck. Fixes #3958 2023-09-12 15:54:49 -04:00
phiresky
375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
2023-09-09 12:25:03 -04:00