Commit graph

214 commits

Author SHA1 Message Date
cetra3
9256895635
Cache & Optimize Woodpecker CI (#3450)
* Try using drone cache plugin

* Try another path

* Include volume

* Fix formatting

* Include fmt

* Exclude cargo dir from prettier

* Don't override cargo

* Just do check

* Add cache key

* Use different cache plugin

* Add clippy

* Try minio

* Add quotes

* Try adding secrets

* Try again

* Again

* Use correct secret formation

* Add back clippy

* Use secret for the root bucket name

* Try drone cache instead

* Add region

* Add path-style option

* Include cargo clippy

* Include everything again

* Fix formatting

* Don't run clippy twice

* Add `allow` statements for tests to pass

* Adjust endpoint to be a secret

* Fix prettier

* Merge & fix tests

* Try to restart the woodpecker test

* Change the ENV var name

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-07-17 11:04:14 -04:00
Sander Saarend
6688a8a5d4
Optimize hot rank updates (#3617) 2023-07-17 11:05:55 +02:00
e9e76549a8
Split activity table into sent and received parts (fixes #3103) (#3583)
* Split activity table into sent and received parts (fixes #3103)

The received activities are only stored in order to avoid processing
the same incoming activity multiple times. For this purpose it is
completely unnecessary to store the data. So we can split the
table into sent_activity and received_activity parts, where
only sent_activity table needs to store activity data. This should
reduce storage use significantly.

Also reduces activity storage duration to three months, we can reduce
this further if necessary.

Additionally the id columns of activity tables are removed because
they are completely unused and risk overflowing (fixes #3560).

* address review

* move insert_received_activity() methods to verify handlers

* remove unnecessary conflict line

* clippy

* use on conflict, add tests
2023-07-14 11:17:06 -04:00
7d8cb93b53
Check for dead federated instances (fixes #2221) (#3427)
* Check for dead federated instances (fixes #2221)

* move to apub crate, use timestamp

* make it compile

* clippy

* use moka to cache blocklists, dead instances, restore orig scheduled tasks

* remove leftover last_alive var

* error handling

* wip

* fix alive check for instances without nodeinfo, add coalesce

* clippy

* move federation blocklist cache to #3486

* unused deps
2023-07-13 16:12:01 +02:00
dullbananas
1d38aad9d3
Make functions work with both connection and pool (#3420)
* a lot

* merge

* Fix stuff broken by merge

* Get rid of repetitive `&mut *context.conn().await?`

* Add blank lines under each line with `conn =`

* Fix style mistakes (partial)

* Revert "Fix style mistakes (partial)"

This reverts commit 48a033b87f.

* Revert "Add blank lines under each line with `conn =`"

This reverts commit 773a6d3beb.

* Revert "Get rid of repetitive `&mut *context.conn().await?`"

This reverts commit d2c6263ea1.

* Use DbConn for CaptchaAnswer methods

* DbConn trait

* Remove more `&mut *`

* Fix stuff

* Re-run CI

* try to make ci start

* fix

* fix

* Fix api_common::utils

* Fix apub::activities::block

* Fix apub::api::resolve_object

* Fix some things

* Revert "Fix some things"

This reverts commit 2bf8574bc8.

* Revert "Fix apub::api::resolve_object"

This reverts commit 3e4059aabb.

* Revert "Fix apub::activities::block"

This reverts commit 3b02389abd.

* Revert "Fix api_common::utils"

This reverts commit 7dc73de613.

* Revert "Revert "Fix api_common::utils""

This reverts commit f740f115e5.

* Revert "Revert "Fix apub::activities::block""

This reverts commit 2ee206af7c.

* Revert "Revert "Fix apub::api::resolve_object""

This reverts commit 96ed8bf2e9.

* Fix fetch_local_site_data

* Fix get_comment_parent_creator

* Remove unused perma deleted text

* Fix routes::feeds

* Fix lib.rs

* Update lib.rs

* rerun ci

* Attempt to create custom GetConn and RunQueryDsl traits

* Start over

* Add GetConn trait

* aaaa

* Revert "aaaa"

This reverts commit acc9ca1aed.

* Revert "Revert "aaaa""

This reverts commit 443a2a00a5.

* still aaaaaaaaaaaaa

* Return to earlier thing

Revert "Add GetConn trait"

This reverts commit ab4e94aea5.

* Try to use DbPool enum

* Revert "Try to use DbPool enum"

This reverts commit e4d1712646.

* DbConn and DbPool enums (db_schema only fails to compile for tests)

* fmt

* Make functions take `&mut DbPool<'_>` and make db_schema tests compile

* Add try_join_with_pool macro and run fix-clippy on more crates

* Fix some errors

* I did it

* Remove function variants that take connection

* rerun ci

* rerun ci

* rerun ci
2023-07-11 09:09:59 -04:00
Pawan Hegde
ef9dc5d0b6
Fix #3366: Wrap plain-text error responses from the API in JSON (#3559)
* Fix #3366: API does return plain HTML errors

* Fix Clippy errors

* Improve api response times by doing send_activity asynchronously (#3493)

* do send_activity after http response

* move to util function

* format

* fix prometheus

* make synchronous federation configurable

* cargo fmt

* empty

* empty

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>

* Updating `login.rs` with generic `incorrect_login` response. (#3549)

* Adding v0.18.1 and v0.18.0 release notes. (#3530)

* Update RELEASES.md (#3556)

added instruction to find the location of your docker directory (especially useful for those who used ansible since they never had to setup docker manually)

* Use async email sender (#3554)

* Upgrade all dependencies (#3526)

* Upgrade all dependencies

* as base64

* Adding phiresky to codeowners. (#3576)

* Error enum fixed (#3487)

* Create error type enum

* Replace magic string slices with LemmyErrorTypes

* Remove unused enum

* Add rename snake case to error enum

* Rename functions

* clippy

* Fix merge errors

* Serialize in PascalCase instead of snake_case

* Revert src/lib

* Add serialization tests

* Update translations

* Fix compilation error in test

* Fix another compilation error

* Add code for generating typescript types

* Various fixes to avoid breaking api

* impl From<LemmyErrorType> for LemmyError

* with_lemmy_type

* trigger ci

---------

Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>

* Only update site_aggregates for local site (#3516)

* Fix #3501 - Fix aggregation counts for elements removed and deleted (#3543)

Two bugs were found and fixed:
- previously elements removal and deletion were counted as two separate disappearances
- removing comments did not affect post aggregations

* Use LemmyErrorType also make error_type compulsory

* Add missing import for jsonify_plain_text_errors

* Fix formatting

* Trying to make woodpecker run again

---------

Co-authored-by: phiresky <phireskyde+git@gmail.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: rosenjcb <rosenjcb@gmail.com>
Co-authored-by: nixoye <12674582+nixoye@users.noreply.github.com>
Co-authored-by: dullbananas <dull.bananas0@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>
Co-authored-by: Sander Saarend <sander@saarend.com>
Co-authored-by: Piotr Juszczyk <74842304+pijuszczyk@users.noreply.github.com>
2023-07-10 22:44:14 +02:00
Sander Saarend
651f2747ee
Only update site_aggregates for local site (#3516) 2023-07-10 11:20:39 -04:00
dullbananas
934f72511e
Use async email sender (#3554) 2023-07-10 14:04:39 +02:00
phiresky
b35757b429
Improve api response times by doing send_activity asynchronously (#3493)
* do send_activity after http response

* move to util function

* format

* fix prometheus

* make synchronous federation configurable

* cargo fmt

* empty

* empty

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-07-10 12:27:49 +02:00
Diamond
084f603745
Allow cross-origin requests (#3421)
Co-authored-by: pfg <pfg@pfg.pw>
2023-07-06 07:25:19 -04:00
Andrew Fields
1e99e8b9dc
Add Prometheus endpoint (#3456)
Add a server for serving Prometheus metrics. Include a configuration
block in the config file. Provide HTTP metrics on the API, along with
process-level metrics and DB pool metrics.
2023-07-05 13:25:19 +02:00
Lemmus.org
ff47d97bd3
Add feature for enabling json logging (#3462) 2023-07-04 07:11:47 -04:00
3578dab67f
Remove PerformApub trait (#3423)
* Remove PerformApub trait

This is completely useless now that websocket is gone. In the future
I also plan to remove Perform and PerformCrud traits, but it will be
difficult to do that while still compiling crates in parallel.

* params need to use query
2023-07-03 11:01:41 +02:00
TKilFree
2aef6a5a33
feat: re-added captcha checks (#3289) 2023-06-27 06:38:53 -04:00
Sander Saarend
211e76dc27
Batch hot rank updates (#3175) 2023-06-27 10:13:51 +02:00
dullbananas
b2a9d4a335
Use compression middleware (#3343) 2023-06-26 12:54:41 +02:00
cetra3
9406c3ad2b
Adjust the config check to be a separate faster to compile binary (#3313) 2023-06-26 06:23:21 -04:00
cetra3
d6b580a530 Remove actix_rt & use standard tokio spawn (#3158)
* Remove `actix_rt` & use standard tokio spawn

* Adjust rust log back down

* Format correctly

* Update cargo lock

* Add DB settings

* Change name and update to latest rev

* Clean up formatting changes

* Move `worker_count` and `worker_retry_count` to settings

* Update defaults

* Use `0.4.4` instead of git branch
2023-06-26 10:45:22 +02:00
Dessalines
37a47de3a8 Revert "feat: re-added captcha checks (#3249)" (#3288)
This reverts commit 8a086c8240.
2023-06-26 10:45:22 +02:00
TKilFree
9537878519 feat: re-added captcha checks (#3249) 2023-06-26 10:45:22 +02:00
Vijay Ramesh
418bca78e1
Merge branch 'main' into clear_deleted_posts_comments 2023-06-22 09:59:43 -07:00
ffce66abb8
Update apub library to 0.4.4 (#3258)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-06-22 08:35:12 -04:00
Óliver García Albertos
985359918f
logger: Use forwarded instead of peer IP address (#3223) 2023-06-22 09:34:51 +02:00
Vijay Ramesh
0d576a8f23
Merge branch 'main' into clear_deleted_posts_comments 2023-06-21 13:35:37 -07:00
kartikynwa
ef3544bac4
Put /admin/purge routes before /admin routes (#3244) 2023-06-21 22:32:06 +02:00
Vijay Ramesh
4db65c191c after 30 days post deletion, replace comment.content and post.body with 'Permanently Deleted' 2023-06-21 12:46:56 -07:00
dullbananas
45818fb4c5
Reduce memory usage of rate limiting (#3111)
* Reduce Vec allocations

* Optimize stuff

* Move embedded migrations to separate crate

* Revert "Move embedded migrations to separate crate"

This reverts commit 44b1049970.

* clippy, fmt

* Shrink rate limit allowance to f32

* Initialize rate limit allowance directly

* Add removal of old rate limit buckets

* Improve readability

* Remove usage of is_okay_and for Rust 1.67 compatibility

* Add dhat-heap feature

* Fix api_benchmark.sh and add run_and_benchmark.sh

* Revert "Fix api_benchmark.sh and add run_and_benchmark.sh"

This reverts commit b4528e5b85.

* Revert "Add dhat-heap feature"

This reverts commit 08e835d487.

* Manually revert remaining stuff

* Use Ipv6Addr in RateLimitStorage

* Shrink last_checked in RateLimitBucket to 32 bits

* Fix rate_limit::get_ip

* Stuff (#1)

* Update rate_limiter.rs

* Update mod.rs

* Update scheduled_tasks.rs

* Fix rate_limiter.rs

* Dullbananas patch 1 (#2)

* Update rate_limiter.rs

* Update mod.rs

* Update scheduled_tasks.rs

* Fix rate_limiter.rs

* Rate limit IPv6 addresses in groups

* Fmt lib.rs

* woodpicker trigger

* Refactor and comment `check_rate_limit_full`

* Add `test_split_ipv6`

* Replace -2.0 with UNINITIALIZED_TOKEN_AMOUNT

* Add `test_rate_limiter`

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-06-21 10:28:20 +02:00
Dessalines
b214d3dc00
Adding current domain from settings to CORs. (#3231) 2023-06-20 20:29:48 -04:00
Joel Beckmeyer
8179b0daf2
remove debugging println (#3227) 2023-06-20 14:20:20 -04:00
b37c30ffae
Move cors default to source code (#3219) 2023-06-20 11:19:03 -04:00
Andrew DeLisa
bfc0c0e7d3
fix: add CORS origin environment variable (#3191)
* fix: add CORS origin environment variable

* chore: formatting

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-06-20 08:29:38 -04:00
Joel Beckmeyer
06a3f9c5aa
make embedded pict-rs port configurable (#3201) 2023-06-20 13:07:18 +02:00
Dessalines
981c111567
Move connection creation into scheduler. (#3120)
* Move connection creation into scheduler.

- #3076

* Fix clippy.
2023-06-20 11:33:03 +02:00
Sander Saarend
68d814b9b1
Handle scheduled task errors (#3090)
* Add --disable-scheduled-tasks CLI flag

* Add error handling for scheduled tasks
2023-06-15 11:29:12 +02:00
Dessalines
8cb5939f50
Adding hot_rank columns in place of function sorting. (#2952)
* Adding hot_rank columns in place of function sorting.

- Creates hot_rank columns for post, comment, and community.
- Fixes #2932

* Updating all hot ranks on startup.

* Fixing post.url migration.

* Removing update_instance_software from startup.

* Adding post_rank query
2023-06-08 16:15:15 -04:00
3565ad984a
Remove chatserver (#2919)
* Remove chatserver

* fix clippy

* Remove captchas (fixes #2922)

* fix prettier

* fix api_common build

* cargo fmt
2023-06-06 12:27:22 -04:00
Dessalines
af3eb6e27b
Add CORS support for debug mode. (#2884) 2023-05-29 17:14:00 -04:00
4240af86ca
Remove websocket code (#2878)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-05-29 10:44:20 -04:00
df57795595 Dont retry outgoing HTTP requests (ref #2865) 2023-05-18 16:34:52 +02:00
Dessalines
1b5437cbe3
Addressing slow profile queries. #2777 (#2830)
* Addressing slow profile queries. #2777

* Addressing PR comments.
2023-04-25 19:28:06 -04:00
Dessalines
33ba1fc623
Separate federated_instances into its own endpoint. Fixes #1931 (#2813) 2023-04-16 17:00:31 -04:00
Dessalines
63f54a3103
Making the chat server an actor. (#2793)
* Making the chat server an actor.

- Fixes #2778
- #2787

* Forgot to add handlers folder.

* Some cleanup.

* Forgot to remove a comment.

* Address PR comments.

* Using ToString for enum operations.
2023-04-13 06:53:55 -04:00
6f513793cb
Activitypub crate rewrite (#2782)
* update activitypub-federation crate to 0.4.0

* fixes

* apub compiles!

* everything compiling!

* almost done, federated follow failing

* some test fixes

* use release

* add code back in
2023-03-21 16:03:05 +01:00
Anon
6bc49bdd70
Add Custom Emojis Support (#2616)
* Add Custom Emojis

* Modify index
2023-03-20 17:32:31 -04:00
d9e7f0100a
Dont upsert Instance row every apub fetch (#2771)
This is not necessary because the domain cant change, so we only
need to insert if no row exists for this domain.

Also fetch instance actor when parsing person, not only community

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-02-28 21:36:57 -05:00
Dessalines
3844ac76c3
Fixing broken websockets. (#2770)
* Fixing broken websockets.

* Use cloned() for serde::Value
2023-02-28 21:27:33 -05:00
Dessalines
209c8a9185
Switch from bb8 to deadpool. Fixes #2765 (#2768)
* Switch from bb8 to deadpool. Fixes #2765

* Remove unecessary deadpool dependency.

* Ignoring nodeinfo test.
2023-02-28 22:45:37 +01:00
Dessalines
8c0c1628e0
Adding a vector indexing check to prevent panics. Fixes #2753 (#2754)
* Adding a vector indexing check to prevent panics. Fixes #2753

* Fixing for new clippy lint.

* Externalizing get_top_mod check. Adding get_first clippy lint

* Removing unit tests get(0)s

* Fixing some firsts manually
2023-02-28 12:34:50 +01:00
Dessalines
1917e3d495
Separate comment distinguish (#2740)
* Combine prod and dev docker setups using build-arg

- Fixes #2603

* Dont use cache for release build.

* Separate comment distinguish into its own action.

- Fixes #2708

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-02-18 09:46:34 -05:00
Dessalines
3735c6fabf
Adding instance software and version. Fixes #2222 (#2733)
* Adding instance software and version. Fixes #2222

* Fix clippy.

* Fix clippy 2

* Fixing some more issues.
2023-02-18 09:36:12 -05:00