Commit graph

247 commits

Author SHA1 Message Date
phiresky
375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
2023-09-09 12:25:03 -04:00
SorteKanin
722027ce2e
Bump ts_rs version (#3947)
Co-authored-by: Nutomic <me@nutomic.com>
2023-09-06 13:04:24 -04:00
71d61138bc
Replace ammonia lib with manual html escaping (fixes #3774) (#3938)
* Replace ammonia lib with manual html escaping (fixes #3774)

* prettier

* clippy

* remove sanitize unit test

* fix tests

* fix schema
2023-09-06 10:56:26 -04:00
fed6542055
Implement remote follow (#3738)
* Add remote follow url to webfinger response

* update apub lib
2023-08-31 09:00:41 -04:00
b2aee565f3
Allow passing auth via header or cookie (#3725)
* Allow passing auth via header or cookie

* revert submodule

* taplo

* fix build

* working

* convert apub api methods

* also set cache-control header

* opt

* clippy

* deduplicate code, ignore invalid auth

* clippy

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-08-29 10:47:57 -04:00
phiresky
514f2222e0
Fix time zone handling (#3496)
* convert naive time to utc time

* compounding fixes

* cargo fmt

* fix the rest

* fix down migration

* fix migrations

* fix after merge

* clippy fix

* ap-fed 0.5.0

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-08-24 11:27:00 -04:00
Lukas Trombach
28324ad2c8
Sitemap (#3808)
* generate sitemap.xml file

* set up endpoint for sitemap

* Update sitemap generation

- remove sitemap generation from scheduled tasks
- add posts query for sitemap
- create sitemap module in API crate

* remove priority and change freq from sitemap

* add configuration option for number of posts for sitemap

* fix default config

* rate limit sitemap endpoint

* update sitemap query

* update sitemap generation

- remove config value for query limit
- adjust sitemap generation to query changes
- tidy up error handling

* refactor sitemap generation loop

* remove `limit` argument

* refactor `generate_urlset` and add unit test

* change query to only fetch local posts of past 24h

* fix outdated comment and log

* cargo fmt
2023-08-22 10:30:15 -04:00
91c024fd98 Revert "Version 0.18.4-rc.2"
This reverts commit 48466b6826.
2023-08-07 17:12:09 +02:00
48466b6826 Version 0.18.4-rc.2 2023-08-07 17:09:10 +02:00
2d0f77af59
Dont use sha hash for password reset token (fixes #3491) (#3795) 2023-08-02 13:02:53 -04:00
37998b3398
Rewrite some API handlers to remove Perform trait (#3735)
* Rewrite some API handlers to remove Perform trait

* Convert CreateComment

* ci
2023-07-28 10:39:38 -04:00
RocketDerp
21a87ebaf2
Federation tests replication round1 - demonstrate absent replication of comment deletes (#3657)
* more robust test of unlike a comment, confirm replication to instance downstream from community home

* more robust 'delete a comment' test, confirm replication

* Far more robust "Report a comment" test. Many comments about situation, this is currently failing because gamma does not get the report

* typo and actually have Gamma comment check use gamma, not alpha

* prepare-drone-federation-test.sh has some more echo output and note about the LEMMY_DATABASE_URL format (#3651)

* Add http cache for webfingers (#3317)

* Add http cache for webfingers

* Remove the outgoing cache middleware & adjust the cache headers directive

* Use 1h & 3day cache header

* Update routes and adjust the cache headers location

* revert apub caching

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Felix Ableitner <me@nutomic.com>

* Rewrite activity lists to fix delete federation (fixes #3625)

* Revert "typo and actually have Gamma comment check use gamma, not alpha"

This reverts commit 7dfb6ee0f4.

* Revert "Far more robust "Report a comment" test. Many comments about situation, this is currently failing because gamma does not get the report"

This reverts commit 7bd3b20ae0.

* prettier TypeScript

* revised comments, as ResolveObject isn't using routine replication

* fmt

* fix api tests

* remove comment

---------

Co-authored-by: cetra3 <cetra3@hotmail.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Felix Ableitner <me@nutomic.com>
2023-07-27 06:17:40 -04:00
3471f3533c
Sanitize html (#3708)
* HTML sanitization in apub code

* Sanitize API inputs

* fmt

* Dont allow html a, img tags

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-07-26 14:01:15 -04:00
Ben Wyatt
95e758e707
Bump version of dependency "webmention" (#3711)
Webmention .4->.5 bumps a few internal dependencies that needed to be upgraded
2023-07-25 13:01:13 -04:00
3753a3bf54
Upgrade activitypub library to 0.4.6 (fixes #3222) (#3675) 2023-07-20 11:33:19 -04:00
5d23ef960e
Remove SendActivity and Perform traits, rely on channel (#3596)
* Remove SendActivity and Perform traits, rely on channel

These traits arent necessary anymore now that websocket is removed.
Removing them allows us to use normal actix http handler methods
which are much more flexible, and allow using different middlewares
as well as setting response attributes.

* compiling and create post federating

* clippy

* rename methods, join outgoing activities task

* fix api tests

* no unwrap

* conditional compile

* add back getrandom

* make crates optional

* fmt
2023-07-19 09:49:41 -04:00
dullbananas
88215bfbc9
Remove TypedBuilder from db_views and db_views_actor (#3637)
* change pool fields to parameters for list

* remove my_person_id and admin fields

* Change recipient id to list param

* Remove TypedBuilder from db_views and db_views_actor
2023-07-17 12:20:25 +02:00
Dessalines
164f4b93d9
Make lemmy_api_common wasm-compatible (#3587)
* Fixing lemmy_api_common optionals

* Adding taplo for .toml files.

* Versioning taplo in woodpecker.

* Addressing PR comments.
2023-07-14 10:45:18 +02:00
93225e5ddf
Error enum fixed (#3487)
* Create error type enum

* Replace magic string slices with LemmyErrorTypes

* Remove unused enum

* Add rename snake case to error enum

* Rename functions

* clippy

* Fix merge errors

* Serialize in PascalCase instead of snake_case

* Revert src/lib

* Add serialization tests

* Update translations

* Fix compilation error in test

* Fix another compilation error

* Add code for generating typescript types

* Various fixes to avoid breaking api

* impl From<LemmyErrorType> for LemmyError

* with_lemmy_type

* trigger ci

---------

Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>
2023-07-10 16:50:07 +02:00
ff26bc21af
Upgrade all dependencies (#3526)
* Upgrade all dependencies

* as base64
2023-07-10 15:00:55 +02:00
dullbananas
934f72511e
Use async email sender (#3554) 2023-07-10 14:04:39 +02:00
0c82f4e660 Version 0.18.1 2023-07-07 09:24:49 -04:00
b3dfc3721f
Upgrade crossbeam-channel (#3512) 2023-07-06 14:21:11 -04:00
22cdcc6b60 Version 0.18.1-rc.10 2023-07-06 14:46:46 +02:00
ce1ffebeb0
Use fixed prettier version for CI (#3507)
3.0.0 was pushed to docker hub 2 hours ago, thats probably why
builds are breaking now.
2023-07-06 12:23:51 +02:00
phiresky
2158621bda
upgrade markdown-it to fix panic (#3490)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-07-05 13:08:53 -04:00
ebaf69bd70
Cache federation blocklist (#3486)
* Cache federation blocklist

* revert submodule change
2023-07-05 11:08:02 -04:00
Andrew Fields
1e99e8b9dc
Add Prometheus endpoint (#3456)
Add a server for serving Prometheus metrics. Include a configuration
block in the config file. Provide HTTP metrics on the API, along with
process-level metrics and DB pool metrics.
2023-07-05 13:25:19 +02:00
Piotr Juszczyk
bf62fbe644
Remove not needed anymore http-signature-normalization-actix dependency (#3458) 2023-07-04 12:14:37 +02:00
7a04971ecd Version 0.18.1-rc.9 2023-07-03 17:16:48 -04:00
phiresky
cb28af508d
don't strip, log trace if requested (#3425) 2023-06-30 09:32:43 -04:00
3159eedd99 Version 0.18.1-rc.4 2023-06-29 10:45:59 -04:00
Dessalines
6c64cb5233
Fixing release script. (#3398)
* Fixing release script.

* Version 0.18.1-rc.2

* Removing cargo update from release script.

* Fixing topdir location.
2023-06-29 16:17:59 +02:00
c216153dfb
Update activitypub-federation crate to 0.4.5 (#3379)
https://github.com/LemmyNet/activitypub-federation-rust/releases/tag/0.4.5
2023-06-28 12:58:23 -04:00
ad6f244b61 Version 0.18.1-rc.1 2023-06-27 08:14:46 -04:00
TKilFree
2aef6a5a33
feat: re-added captcha checks (#3289) 2023-06-27 06:38:53 -04:00
Sander Saarend
211e76dc27
Batch hot rank updates (#3175) 2023-06-27 10:13:51 +02:00
dullbananas
b2a9d4a335
Use compression middleware (#3343) 2023-06-26 12:54:41 +02:00
Sander Saarend
6d67f88603
Add support for sslmode=require for diesel-async DB connections (#3189) 2023-06-26 10:25:38 +02:00
cetra3
d7da911a48
Remove actix_rt & use standard tokio spawn (#3158)
* Remove `actix_rt` & use standard tokio spawn

* Adjust rust log back down

* Format correctly

* Update cargo lock

* Add DB settings

* Change name and update to latest rev

* Clean up formatting changes

* Move `worker_count` and `worker_retry_count` to settings

* Update defaults

* Use `0.4.4` instead of git branch
2023-06-26 10:24:11 +02:00
63d3759c48 Version 0.18.0 2023-06-23 08:41:20 -04:00
cb3b9b1a66 Version 0.18.0-rc.8 2023-06-23 07:07:51 -04:00
Dessalines
1f1c0ad17a
Revert "feat: re-added captcha checks (#3249)" (#3288)
This reverts commit 8a086c8240.
2023-06-23 07:02:05 -04:00
TKilFree
8a086c8240
feat: re-added captcha checks (#3249) 2023-06-23 11:47:12 +02:00
60baebcb14 Version 0.18.0-rc.6 2023-06-22 08:55:08 -04:00
ffce66abb8
Update apub library to 0.4.4 (#3258)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-06-22 08:35:12 -04:00
289052f261 Version 0.18.0-rc.5 2023-06-21 16:02:42 -04:00
3775418952 Version 0.18.0-rc.4 2023-06-21 08:05:15 -04:00
dullbananas
45818fb4c5
Reduce memory usage of rate limiting (#3111)
* Reduce Vec allocations

* Optimize stuff

* Move embedded migrations to separate crate

* Revert "Move embedded migrations to separate crate"

This reverts commit 44b1049970.

* clippy, fmt

* Shrink rate limit allowance to f32

* Initialize rate limit allowance directly

* Add removal of old rate limit buckets

* Improve readability

* Remove usage of is_okay_and for Rust 1.67 compatibility

* Add dhat-heap feature

* Fix api_benchmark.sh and add run_and_benchmark.sh

* Revert "Fix api_benchmark.sh and add run_and_benchmark.sh"

This reverts commit b4528e5b85.

* Revert "Add dhat-heap feature"

This reverts commit 08e835d487.

* Manually revert remaining stuff

* Use Ipv6Addr in RateLimitStorage

* Shrink last_checked in RateLimitBucket to 32 bits

* Fix rate_limit::get_ip

* Stuff (#1)

* Update rate_limiter.rs

* Update mod.rs

* Update scheduled_tasks.rs

* Fix rate_limiter.rs

* Dullbananas patch 1 (#2)

* Update rate_limiter.rs

* Update mod.rs

* Update scheduled_tasks.rs

* Fix rate_limiter.rs

* Rate limit IPv6 addresses in groups

* Fmt lib.rs

* woodpicker trigger

* Refactor and comment `check_rate_limit_full`

* Add `test_split_ipv6`

* Replace -2.0 with UNINITIALIZED_TOKEN_AMOUNT

* Add `test_rate_limiter`

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-06-21 10:28:20 +02:00
b5dba17426
Update all dependencies (#3091) 2023-06-15 11:37:05 +02:00