Commit graph

99 commits

Author SHA1 Message Date
phiresky
375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
2023-09-09 12:25:03 -04:00
phiresky
514f2222e0
Fix time zone handling (#3496)
* convert naive time to utc time

* compounding fixes

* cargo fmt

* fix the rest

* fix down migration

* fix migrations

* fix after merge

* clippy fix

* ap-fed 0.5.0

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-08-24 11:27:00 -04:00
Freakazoid182
51ccf318e8
Lowercase domain on db query filters (#3849) (#3873)
* Lowercase domain on db query filters (#3849)

* Add test to get a community on different cased domain (#3849)

* Lowercase the identity for webfinger (#3849)

* Lowercase both sides of the domain comparison (#3849)

* Format api_tests (#3849)

* Lowercase domain lookup on Instance and Person (#3849)

---------

Co-authored-by: Freek van Zee <freek.van.zee@mediamonks.com>
Co-authored-by: Freakazoid182 <>
2023-08-22 11:10:21 -04:00
marsara9
dc4572460e
Make resolve_object not require auth #3685 (#3716)
* Resolves issue #3685

If user isn't authenticated with resolve_object, only allow a local search instead of possibly making an http request.

* Making sure to validate auth before doing a potential remote lookup.
2023-07-26 12:17:42 -04:00
dullbananas
1d38aad9d3
Make functions work with both connection and pool (#3420)
* a lot

* merge

* Fix stuff broken by merge

* Get rid of repetitive `&mut *context.conn().await?`

* Add blank lines under each line with `conn =`

* Fix style mistakes (partial)

* Revert "Fix style mistakes (partial)"

This reverts commit 48a033b87f4fdc1ce14ff86cc019e1c703cd2741.

* Revert "Add blank lines under each line with `conn =`"

This reverts commit 773a6d3beba2cf89eac75913078b40c4f5190dd4.

* Revert "Get rid of repetitive `&mut *context.conn().await?`"

This reverts commit d2c6263ea13710177d49b2791278db5ad115fca5.

* Use DbConn for CaptchaAnswer methods

* DbConn trait

* Remove more `&mut *`

* Fix stuff

* Re-run CI

* try to make ci start

* fix

* fix

* Fix api_common::utils

* Fix apub::activities::block

* Fix apub::api::resolve_object

* Fix some things

* Revert "Fix some things"

This reverts commit 2bf8574bc8333d8d34ca542d61a0a5b50039c24d.

* Revert "Fix apub::api::resolve_object"

This reverts commit 3e4059aabbe485b2ff060bdeced8ef958ff62832.

* Revert "Fix apub::activities::block"

This reverts commit 3b02389abd780a7b1b8a2c89e26febdaa6a12159.

* Revert "Fix api_common::utils"

This reverts commit 7dc73de613a5618fa57eb06450f3699bbcb41254.

* Revert "Revert "Fix api_common::utils""

This reverts commit f740f115e5457e83e53cc223e48196a2c47a9975.

* Revert "Revert "Fix apub::activities::block""

This reverts commit 2ee206af7c885c10092cf209bf4a5b1d60327866.

* Revert "Revert "Fix apub::api::resolve_object""

This reverts commit 96ed8bf2e9dcadae760743929498312334e23d2e.

* Fix fetch_local_site_data

* Fix get_comment_parent_creator

* Remove unused perma deleted text

* Fix routes::feeds

* Fix lib.rs

* Update lib.rs

* rerun ci

* Attempt to create custom GetConn and RunQueryDsl traits

* Start over

* Add GetConn trait

* aaaa

* Revert "aaaa"

This reverts commit acc9ca1aed10c39efdd91cefece066e035a1fe80.

* Revert "Revert "aaaa""

This reverts commit 443a2a00a56d152bb7eb429efd0d29a78e21b163.

* still aaaaaaaaaaaaa

* Return to earlier thing

Revert "Add GetConn trait"

This reverts commit ab4e94aea5bd9d34cbcddf017339131047e75344.

* Try to use DbPool enum

* Revert "Try to use DbPool enum"

This reverts commit e4d1712646a52006b865a1fbe0dcf79976fdb027.

* DbConn and DbPool enums (db_schema only fails to compile for tests)

* fmt

* Make functions take `&mut DbPool<'_>` and make db_schema tests compile

* Add try_join_with_pool macro and run fix-clippy on more crates

* Fix some errors

* I did it

* Remove function variants that take connection

* rerun ci

* rerun ci

* rerun ci
2023-07-11 09:09:59 -04:00
93225e5ddf
Error enum fixed (#3487)
* Create error type enum

* Replace magic string slices with LemmyErrorTypes

* Remove unused enum

* Add rename snake case to error enum

* Rename functions

* clippy

* Fix merge errors

* Serialize in PascalCase instead of snake_case

* Revert src/lib

* Add serialization tests

* Update translations

* Fix compilation error in test

* Fix another compilation error

* Add code for generating typescript types

* Various fixes to avoid breaking api

* impl From<LemmyErrorType> for LemmyError

* with_lemmy_type

* trigger ci

---------

Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>
2023-07-10 16:50:07 +02:00
Dessalines
1b5437cbe3
Addressing slow profile queries. #2777 (#2830)
* Addressing slow profile queries. #2777

* Addressing PR comments.
2023-04-25 19:28:06 -04:00
6f513793cb
Activitypub crate rewrite (#2782)
* update activitypub-federation crate to 0.4.0

* fixes

* apub compiles!

* everything compiling!

* almost done, federated follow failing

* some test fixes

* use release

* add code back in
2023-03-21 16:03:05 +01:00
62663a9f2e
Implement separate mod activities for feature, lock post (#2716)
* Implement separate mod activities for feature, lock post

Also includes collection for featured posts. Later we also need
to do the same for Comment.distinguished

* some changes

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-02-18 09:50:28 -05:00
Dessalines
a610211557
Fixing .drone.yml (#2677)
* Try to fix docker/drone plugin

* Trying to use one rust image.

* Fixing drone 1.

* Fixing drone 2.

* Add drone notify.

* Fixing drone 3.

* Fixing drone 4.

* Fix clippy.

* Remove uninlined format lint.

* Combine all cargo tasks

* Fixing drone 5.

* Fixing drone 6.

* Fixing drone 7.

* Fixing drone 8.

* Fixing drone 9.

* Fixing drone 10.

* Fixing drone 12.

* Fixing drone 13.

* Fixing drone 14.

* Fixing drone 15.

* Fixing drone 16.

* Fixing drone 17.

* Fixing drone 18.

* Fixing drone 19.

* Fixing drone 20.

* Fixing drone 21.

* Fixing drone 22.

* Fixing drone 23.

* Fixing drone 24.

* Fixing drone 25.

* Fixing drone 26.

* Fixing drone 27.

* Fixing drone 28.

* Fixing drone 29.

* Fixing drone 30.

* Fixing drone 31.

* Fixing drone 32.

* Fixing drone 33.

* Fixing drone 34.

* Fixing drone 35.

* Fixing drone 36.

* Fixing drone 37.

* Fixing drone 38.

* Fixing drone 39.

* Fixing drone 40.

* Fixing drone 41.

* Fixing drone 43.

* Fixing drone 44.

* Fixing drone 45.

* Last cleanup.

* Fixing drone 46.

* Separate ci steps (#2679)

* separate ci steps

* fix 1

* add comments

* dont add rustfmt explicitly

* Revert "dont add rustfmt explicitly"

This reverts commit 358ce3302a134b7ac88d90a854079356995e9725.

* dont use all features for tests

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-01-30 14:17:24 -05:00
0ecf256ce3
Remove federation backwards compatibility with 0.16.x (#2183)
* Breaking: remove compatiblity with page.url field (ref #2182)

* Breaking: change type of Instance to `Application` (ref #2200)

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2022-12-09 11:21:17 -05:00
c6c52ab9cc Add SendActivity trait so that api crates compile in parallel with lemmy_apub 2022-12-02 10:46:49 +01:00
904d7bec2f Builds lemmy_routes in parallel with lemmy_apub 2022-12-02 10:46:49 +01:00
030afbc2e7 Merge websocket crate into api_common 2022-12-02 10:46:49 +01:00
bcf5c91f81
Use audience field to federate items in groups (fixes #2464) (#2584) 2022-12-01 15:52:49 -05:00
d20d2b9218
Implement federated user following (fixes #752) (#2577)
* Implement federated user following (fixes #752)

* rewrite send_activity_in_community and add docs, remove default for column pending

* improve migration

* replace null values in db migration
2022-11-23 18:40:47 -05:00
872c60a013
Remove federation settings, rely on sensible defaults instead (#2574)
This affects federation_strict_allowlist and federation_http_fetch_retry_limit
2022-11-21 11:44:34 -05:00
Dessalines
5d837780f5
Add diesel_async, get rid of blocking function (#2510)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Testing out async

* Testing out async 2

* Fixing federation tests.

* Trying to fix check features 1.

* Starting on adding diesel async. 1/4th done.

* Added async to views and schema.

* Adding some more async

* Compiling now.

* Added diesel async. Fixes #2465

* Running clippy --fix

* Trying to fix cargo test on drone.

* Trying new muslrust.

* Trying a custom dns

* Trying a custom dns 2

* Trying a custom dns 3

* Trying a custom dns 4

* Trying a custom dns 5

* Trying a custom dns 6

* Trying a custom dns 7

* Addressing PR comments.

* Adding check_apub to all verify functions.

* Reverting back drone.

* Fixing merge

* Fix docker images.

* Adding missing discussion_languages.

* Trying to fix federation tests.

* Fix site setup user creation.

* Fix clippy

* Fix clippy 2

* Test api faster

* Try to fix 1

* Try to fix 2

* What are these lines about

* Trying to fix 3

* Moving federation test back to top.

* Remove logging cat.
2022-11-09 10:05:00 +00:00
Dessalines
235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2022-10-27 09:24:07 +00:00
6c3e984ad1
Only allow authenticated users to fetch remote objects (#2493)
* Only allow authenticated users to fetch remote objects

* try to fix api tests
2022-10-13 16:30:31 +00:00
Dessalines
ee41654394
Dont show deleted users or communities on profile page. (#2450)
* Dont show deleted users or communities on profile page.

- Fixes #2448

* Fix missing communities

* Add include_deleted to resolve_actor_identifier.
2022-09-28 20:54:32 +00:00
Dessalines
c9f1407429
Diesel 2.0.0 upgrade (#2452)
* Initial commit to bump diesel to 2.0.0-rc.0 and see what happens

* Add chrono feature from diesel

* db_schema crate is close to building?

* Upgrade diesel-derive-newtype

* Mostly modifying references to connections to be mutable ones; also used
new way to do migrations as suggested by the migration guide; a lot more
compiles now, though I can't figure out this tricky ToSql issue at the
moment

* Running clippy --fix

* Trying to fix drone clippy 1

* Fix clippy

* Upgrade clux-musl

* Trying to fix drone clippy 2

* Trying to fix drone clippy 3

* Trying to fix drone clippy 5

* Adding diesel table aliases, removing sql view hack. Fixes #2101

Co-authored-by: Steven Chu <stevenc1@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
2022-09-26 14:09:32 +00:00
5e8a37ac86 clippy fixes 2022-09-02 10:49:54 -04:00
b9f1fc0518
Fix panics in search_by_apub_id() (fixes #2371) (#2373) 2022-07-27 17:03:44 -04:00
b7a2677b4d
Be more explicit about returning deleted actors or not (#2335)
* Be more explicit about returning deleted actors or not

* simplify db queries
2022-07-05 17:40:44 -04:00
Dessalines
a745fa6f43
Remove update and read site config. Fixes #2306 (#2329)
* Remove update and read site config. Fixes #2306

* Removing lazy_static, removing Settings::get()
2022-06-22 20:24:54 +00:00
cd4d235171
Move setting http_fetch_retry_limit into federation block (#2314) 2022-06-14 01:20:30 -04:00
fcaf7a084c
Upgrade activitypub_federation to 0.2.0, add setting federation.debug (#2300) 2022-06-08 11:45:39 -04:00
a2a594b763
Extract Activitypub logic into separate library (#2288)
* Create example for apub lib

* some rewriting of apub lib

* Add LocalInstance struct for apub lib to avoid using Lemmy Settings

* Move ActorType trait to lemmy_apub, because its not needed in library

* Use reqwest_retry instead of custom impl, dont specify timeout on every send()

* Some improvements to example

* Moved inbox handling to library

* bug fixes

* Move context and serde helpers into library

* wip: example changes

* Add lemmy_utils feature to build only LemmyError

* Rename to activitypub_federation

* Remove lemmy_utils dep from activitypub_federation using generic error type

* Finish activitypub example

* Cleanup and fix tests

* Reorganize library files

* Remove ApubObject.to_tombstone()

* Extract activitypub library into separate git repository
2022-06-02 16:33:41 +02:00
7b86441bab
Federate with Peertube (#2244) 2022-05-06 23:53:33 +00:00
3aa3d75a1e
Add cargo feature for building lemmy_api_common with mininum deps (#2243) 2022-05-03 17:44:13 +00:00
bfae246734
Migrate towards using page.attachment field for url (ref #2144) (#2182) 2022-04-01 18:25:19 +00:00
4cf0da7b60 Clippy fixes. 2022-03-30 10:58:03 -04:00
8112816e99
If viewed actor isnt in db, fetch it from other instance (#2145) 2022-03-23 21:27:51 +00:00
166ec196b0
Rework error handling (fixes #1714) (#2135) 2022-03-16 20:11:49 +00:00
dd865c5af5
Implement instance actor (#1798)
* Implement instance actor

* wip: make site bans federate

* finish implementation and unit tests for federated bans

* start adding api tests

* fix api test

* remve site from GetCommunityResponse

* only federate site bans originating from user's home instance

* dont expose site.private_key in api
2022-02-07 19:23:12 +00:00
4a23ee4d8b
Dont make webfinger request when viewing community/user profile (fixes #1896) (#2049) 2022-01-27 16:39:22 +00:00
5b3d707d8f Use from_message for static error string 2021-12-07 17:17:19 -06:00
97ebf2f6f3 Consolidate reqwest clients, use reqwest-middleware for tracing 2021-12-06 16:54:34 -06:00
35cbae61bc
Don't drop error context when adding a message to errors (#1958)
* Respond directly with LemmyError

Instrument Perform implementations for more precise traces
Use ApiError to format JSON errors when messages are present
Keep SpanTrace output in LemmyError Display impl

* Hide SpanTrace debug output from LemmyError

* Don't log when entering spans, only when leaving

* Update actix-web

* Update actix-rt

* Add newline after error info in LemmyError Display impl

* Propogate span information to blocking operations

* Instrument apub functions

* Use skip_all for more instrument attributes, don't skip 'self' in some api actions

* Make message a static string

* Send proper JSON over websocket

* Add 'message' to LemmyError display if present

* Use a quieter root span builder, don't pretty-print logs

* Keep passwords and emails out of logs

* Re-enable logging Login

* Instrument feeds

* Emit our own errors

* Move error log after status code recording

* Make Sensitive generic over the inner type

* Remove line that logged secrets
2021-12-06 09:54:47 -05:00
Dessalines
040770d7ba
Fix retry infinite loops. Fixes #1964 (#1967)
* Fix retry infinite loops. Fixes #1964

* Moving retry_limit to settings
2021-12-05 10:03:13 -05:00
3fea5645f8
Add tracing (#1942) 2021-11-23 12:16:47 +00:00
05d43150bc Making public key required. Fixes #1934 2021-11-22 10:10:18 -05:00
afccd5cf69 Replace activitystreams crate with activitystreams-kinds 2021-11-22 14:28:57 +01:00
da89ea22fb Fix Smithereen webfinger, remove duplicate webfinger impl (fixes #1916) 2021-11-18 19:28:53 +01:00
579ca37ddf Make webfinger standard compliant 2021-11-16 18:03:09 +01:00
Dessalines
76220a4523
Fixing clippy lints. (#1885)
* Fixing clippy lints.

* Revert object id display

* Trying to fix clippy again
2021-11-09 18:16:37 +00:00
a5a674a270 Add method ApubObject.verify() 2021-11-06 18:35:14 +01:00
02ce7bdc7d Remove unused ActorType methods 2021-11-06 15:01:00 +01:00
8ea21c39b7 Reduce stack memory usage in apub code
- use our own, smaller Endpoints struct
- wrap ObjectId.url in Box
- adjust usage of Box in different places
2021-11-06 14:53:39 +01:00