* generate sitemap.xml file
* set up endpoint for sitemap
* Update sitemap generation
- remove sitemap generation from scheduled tasks
- add posts query for sitemap
- create sitemap module in API crate
* remove priority and change freq from sitemap
* add configuration option for number of posts for sitemap
* fix default config
* rate limit sitemap endpoint
* update sitemap query
* update sitemap generation
- remove config value for query limit
- adjust sitemap generation to query changes
- tidy up error handling
* refactor sitemap generation loop
* remove `limit` argument
* refactor `generate_urlset` and add unit test
* change query to only fetch local posts of past 24h
* fix outdated comment and log
* cargo fmt
* remove n^2 part of person triggers, improve community aggregate trigger
* comment out comment_score tests since previously they only accidentally succeeded
* empty
* HTML sanitization in apub code
* Sanitize API inputs
* fmt
* Dont allow html a, img tags
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Added controversy rank property to posts and comments, and ability to sort by it
* Triggers instead of schedules tasks, integer -> double, TODO: comments don't seem to get updated with floats, divide SortTypes
* Created PersonSortType
* PersonSortType::MostComments case
* Removed unused PartialOrd trait
* Added new person sort type mappings
* SortType -> PersonSortType
* fixes
* cargo fmt
* fixes after merge with main
* Fixed bug in controversy rank trigger, removed TopX sorts from PersonSortType and added CommentScore instead
* Uncovered enum case
* clippy
* reset translation changes
* translations
* translations
* Added additional hot ordering on controversial posts and comments
* featured local and featured community added to controversy rank index, additional order_by removed (?), added post_score and post_count to PersonSortType
* Woodpecker rerun
* cargo fmt
* woodpecker rerun
* fixed controversy_rank order
* fix
* Readded migration as latest, removed second update statement for setting controversy rank
* add new flag to api
* add new ui settings for local user
* remove extraneous def
* add props to application reg.
* fix clippy updated these
* re-order db schema entries
* remove dupe
* update lemmy sdk
* update lemmy js client
---------
Co-authored-by: Nutomic <me@nutomic.com>
* Denormalize community_id into post_aggregates for a 1000x speed-up when loading posts
* Remove unused index
* Add creator_id to post_aggregates
* Use post_aggregates as main table for PostQuery
* Make post_aggregates the main table for PostView
* Reformat SQL
* Try using drone cache plugin
* Try another path
* Include volume
* Fix formatting
* Include fmt
* Exclude cargo dir from prettier
* Don't override cargo
* Just do check
* Add cache key
* Use different cache plugin
* Add clippy
* Try minio
* Add quotes
* Try adding secrets
* Try again
* Again
* Use correct secret formation
* Add back clippy
* Use secret for the root bucket name
* Try drone cache instead
* Add region
* Add path-style option
* Include cargo clippy
* Include everything again
* Fix formatting
* Don't run clippy twice
* Add `allow` statements for tests to pass
* Adjust endpoint to be a secret
* Fix prettier
* Merge & fix tests
* Try to restart the woodpecker test
* Change the ENV var name
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Split activity table into sent and received parts (fixes#3103)
The received activities are only stored in order to avoid processing
the same incoming activity multiple times. For this purpose it is
completely unnecessary to store the data. So we can split the
table into sent_activity and received_activity parts, where
only sent_activity table needs to store activity data. This should
reduce storage use significantly.
Also reduces activity storage duration to three months, we can reduce
this further if necessary.
Additionally the id columns of activity tables are removed because
they are completely unused and risk overflowing (fixes#3560).
* address review
* move insert_received_activity() methods to verify handlers
* remove unnecessary conflict line
* clippy
* use on conflict, add tests
* Check for dead federated instances (fixes#2221)
* move to apub crate, use timestamp
* make it compile
* clippy
* use moka to cache blocklists, dead instances, restore orig scheduled tasks
* remove leftover last_alive var
* error handling
* wip
* fix alive check for instances without nodeinfo, add coalesce
* clippy
* move federation blocklist cache to #3486
* unused deps
* a lot
* merge
* Fix stuff broken by merge
* Get rid of repetitive `&mut *context.conn().await?`
* Add blank lines under each line with `conn =`
* Fix style mistakes (partial)
* Revert "Fix style mistakes (partial)"
This reverts commit 48a033b87f.
* Revert "Add blank lines under each line with `conn =`"
This reverts commit 773a6d3beb.
* Revert "Get rid of repetitive `&mut *context.conn().await?`"
This reverts commit d2c6263ea1.
* Use DbConn for CaptchaAnswer methods
* DbConn trait
* Remove more `&mut *`
* Fix stuff
* Re-run CI
* try to make ci start
* fix
* fix
* Fix api_common::utils
* Fix apub::activities::block
* Fix apub::api::resolve_object
* Fix some things
* Revert "Fix some things"
This reverts commit 2bf8574bc8.
* Revert "Fix apub::api::resolve_object"
This reverts commit 3e4059aabb.
* Revert "Fix apub::activities::block"
This reverts commit 3b02389abd.
* Revert "Fix api_common::utils"
This reverts commit 7dc73de613.
* Revert "Revert "Fix api_common::utils""
This reverts commit f740f115e5.
* Revert "Revert "Fix apub::activities::block""
This reverts commit 2ee206af7c.
* Revert "Revert "Fix apub::api::resolve_object""
This reverts commit 96ed8bf2e9.
* Fix fetch_local_site_data
* Fix get_comment_parent_creator
* Remove unused perma deleted text
* Fix routes::feeds
* Fix lib.rs
* Update lib.rs
* rerun ci
* Attempt to create custom GetConn and RunQueryDsl traits
* Start over
* Add GetConn trait
* aaaa
* Revert "aaaa"
This reverts commit acc9ca1aed.
* Revert "Revert "aaaa""
This reverts commit 443a2a00a5.
* still aaaaaaaaaaaaa
* Return to earlier thing
Revert "Add GetConn trait"
This reverts commit ab4e94aea5.
* Try to use DbPool enum
* Revert "Try to use DbPool enum"
This reverts commit e4d1712646.
* DbConn and DbPool enums (db_schema only fails to compile for tests)
* fmt
* Make functions take `&mut DbPool<'_>` and make db_schema tests compile
* Add try_join_with_pool macro and run fix-clippy on more crates
* Fix some errors
* I did it
* Remove function variants that take connection
* rerun ci
* rerun ci
* rerun ci
Two bugs were found and fixed:
- previously elements removal and deletion were counted as two separate disappearances
- removing comments did not affect post aggregations
* Use serde(skip) instead of skip_serializing
The latter breaks lemmy_crawler as the field is not included in
the Lemmy API, but is required when attempting to parse API responses.
Should only use serde(skip) to avoid this problem
* use option
* add placeholders
* no unwrap
* Remove `actix_rt` & use standard tokio spawn
* Adjust rust log back down
* Format correctly
* Update cargo lock
* Add DB settings
* Change name and update to latest rev
* Clean up formatting changes
* Move `worker_count` and `worker_retry_count` to settings
* Update defaults
* Use `0.4.4` instead of git branch
* Adding hot_rank columns in place of function sorting.
- Creates hot_rank columns for post, comment, and community.
- Fixes#2932
* Updating all hot ranks on startup.
* Fixing post.url migration.
* Removing update_instance_software from startup.
* Adding post_rank query
* update diesel to 2.1.0 and diesel-async to 0.3.0 (issue #2882)
* get rid of deprecated RawValue type alias usage, explicitly import diesel types instead of using wildcard import
* Update diesel-derive-enum and diesel-derive-newtype libraries
* Revert manual changes to schema.rs
* Update to newer diesel-cli image in CI, run fix-clippy before schema comparison, update schema to one generated with diesel-cli 2.1.0 and clippyfixed
* Remove wildcard import lint, fix diesel schema pipeline
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Adding typescript generation for API. Fixes#2824
* Try to fix Ltree issue 1.
* Forgot a few types.
* Fixing api tests.
* Removing url_serde line.
* Manually deriving TS for some types.
* Adding diesel enums for SortType and ListingType
- Uses diesel-derive-enum.
- Adds diesel.toml , so we can again use the auto-generated schema.rs
- Fixes a lot of DB null issues and column ordering issues.
- Fixes#1136
- Also replaces RegistrationMode boilerplate.
* Fixing unit tests 1.
* Remove comment line.
* Before patch.
* Before again.
* Using patch file to fix diesel_ltree issue with diesel.toml
* Adding some yalc ignores
* Fixing RegistrationMode enums
* Adding woodpecker diesel schema check.
* Try adding openssl 1.
* Try using diesel-cli image 1
* Try using diesel-cli image 2
* Try using diesel-cli image 3
* Try using diesel-cli image 4
* Try using diesel-cli image 5
* Try using diesel-cli image 6
* Try using diesel-cli image 7
* Try using diesel-cli image 8
* Try using diesel-cli image 9
* Try using diesel-cli image 10
* Try using diesel-cli image 11
* Try using diesel-cli image 12
* Try using diesel-cli image 13
