* fix: Run extract_opengraph_data only on first 64kB of data and if data is not binary.
* use mime type for determination
* chore: simplify collect function
* pass local user to send local notif
* pass local user to all crud calls
* execution of command "cargo +nightly fmt"
* Formatting and mode fixes.
---------
Co-authored-by: Dessalines <tyhou13@gmx.com>
* federation: parallel sending
* federation: some comments
* lint and set force_write true when a request fails
* inbox_urls return vec
* split inbox functions into separate file
* cleanup
* extract sending task code to separate file
* move federation concurrent config to config file
* off by one issue
* improve msg
* fix both permanent stopping of federation queues and multiple creation of the same federation queues
* fix after merge
* lint fix
* Update crates/federate/src/send.rs
Co-authored-by: dullbananas <dull.bananas0@gmail.com>
* comment about reverse ordering
* remove crashable, comment
* comment
* move comment
* run federation tests twice
* fix test run
* prettier
* fix config default
* upgrade rust to 1.78 to fix diesel cli
* fix clippy
* delay
* add debug to make localhost urls not valid in ap crate, add some debug logs
* federation tests: ensure server stop after test and random activity id
* ci fix
* add test to federate 100 events
* fix send 100 test
* different data every time so activities are distinguishable
* allow out of order receives in test
* lint
* comment about https://github.com/LemmyNet/lemmy/pull/4623#discussion_r1565437391
* move sender for clarity, add comment
* move more things to members
* update test todo comment, use same env var as worker test but default to 1
* remove else below continue
* some more cleanup
* handle todo about smooth exit
* add federate inboxes collector tests
* lint
* actor max length
* don't reset fail count if activity skipped
* fix some comments
* reuse vars
* format
* Update .woodpecker.yml
* fix recheck time
* fix inboxes tests under fast mode
* format
* make i32 and ugly casts
* clippy
---------
Co-authored-by: dullbananas <dull.bananas0@gmail.com>
* Adding checks for higher admin and mod.
* Adding admin/mod checks for ban and comment removal.
* Combining mod or admin check with an SQL union.
* Making community ban or add mod also allow higher admins.
* Making sure remove post also checks higher mods or admins.
* Add unit test for is_higher_mod_or_admin_check
* Fixing comment.
* Addressing PR comments.
* Get rid of pointless wrapper functions, return lemmyresult directly.
* Fixing not being able to create comments on local community posts.
- This was caused by not passing my_person_id into various
`CommentView::read` functions.
- Fixes#4853
* Refactoring views to use local_user, rather than person
* Addressing PR comments.
* Fixing API tests.
* Adding a show_read override to GetPosts.
- If show_read is true, it overrides the local user show_read
setting.
- Fixes#4124
* Addressing PR comments.
* Update crates/db_views/src/post_view.rs
Co-authored-by: dullbananas <dull.bananas0@gmail.com>
* Fixing formatting.
---------
Co-authored-by: dullbananas <dull.bananas0@gmail.com>
* Adding an image_details table to store image dimensions.
- Adds an image_details table, which stores the height,
width, and content_type for local and remote images.
- For LocalImages, this information already comes back with
the upload.
- For RemoteImages, it calls the pictrs details endpoint.
- Fixed some issues with proxying non-image urls.
- Fixes#3328
- Also fixes#4703
* Running sql format.
* Running fmt.
* Don't fetch metadata in background for local API requests.
* Dont export remote_image table to typescript.
* Cleaning up validate.
* Dont proxy url.
* Fixing tests, fixing issue with federated thumbnails.
* Fix tests.
* Updating corepack, fixing issue.
* Refactoring image inserts to use transactions.
* Use select exists again.
* Fixing imports.
* Fix test.
* Removing pointless backgrounded metadata generation version.
* Removing public pictrs details route.
* Fixing clippy.
* Running prettier.
* A few more fixes.
* Moving diesel schema check back down.
* Addressing PR comments.
* Changing back request head to get.
* Fixing lockfile.
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Allow empty string to clear URL-type DB fields.
- To address difficulties with clearing URL-type fields like
avatars, banners, site icons, this PR turns the URL type form
fields into strings.
- This allows an empty string to be used as a "clear data", as
in the case with the regular text form fields.
- Also includes various cleanups.
- Fixes#4777
- Context: #2287
* Fixing comment.
* Use Option<&str> and deref.
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* After creating a comment, update the unread comments for the post.
- Fixes#3863
* Addressing PR comments.
* Add comment.
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Adding an image_details table to store image dimensions.
- Adds an image_details table, which stores the height,
width, and content_type for local and remote images.
- For LocalImages, this information already comes back with
the upload.
- For RemoteImages, it calls the pictrs details endpoint.
- Fixed some issues with proxying non-image urls.
- Fixes#3328
- Also fixes#4703
* Running sql format.
* Running fmt.
* Don't fetch metadata in background for local API requests.
* Dont export remote_image table to typescript.
* Cleaning up validate.
* Dont proxy url.
* Fixing tests, fixing issue with federated thumbnails.
* Fix tests.
* Updating corepack, fixing issue.
* Refactoring image inserts to use transactions.
* Use select exists again.
* Fixing imports.
* Fix test.
* Removing pointless backgrounded metadata generation version.
* Removing public pictrs details route.
* Fixing clippy.
* Fixing proxy image fetching. Fixes#4703
- This extracts only the proxy image fixes from #4704, leaving off
thumbnails.
* Fix test.
* Addressing PR comments.
* Address PR comments 2.
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Mark database fields as sensitive so they dont show up in logs
* add file
* fix test
* Update crates/apub/src/objects/person.rs
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Update crates/apub/src/objects/community.rs
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Update crates/apub/src/objects/instance.rs
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Replace dependency on unmaintained encoding crate with dep on maintained encoding_rs crate
* Update lockfile
* Taplo format Cargo.toml
* Use better variable name
* Replace into_owned with into
- Diesel ordinarily throws an error when no results are returned for a
single fetch, which is a bit confusing. This PR ensures that the
missing value cases are all caught, and wrapped with new LemmyErrors,
rather than diesel errors.
- Fixes#4601
* Fixing bug where comment replies wouldn't be sent to blocked instances.
- Instance blocks should only affect communities, not comments.
- Fixes#4590
* Revert "Fixing bug where comment replies wouldn't be sent to blocked instances."
This reverts commit 1349aa351a.
* Only block replies from the community's instance id.
- Also refactor send_local_notifs slightly, since it has to fetch the
community now.
- Fixes#4590
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Generate post thumbnail/metadata in background (ref #4529)
* fix api test
* Apply suggestions from code review
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* fix test
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* Delete a person's local images on delete account.
* Rename purge function to delete.
* Use purge_user_account instead of Person::delete_account in purge person.
* Fixing clippy
* Adding listMedia endpoint, to view all your local image uploads.
- Fixes#4445
* Fix ts import.
* Forgot to order by published desc
* Adding an endpoint to list all images, for admins only.
* Forgot to add file.
* Add additional test.
* Use better logic for no-limit version.
* Better call sites.
* Adding another test.
* Fix tests.
* Moving list_media to /account action.
* Addressing PR comments.
* Removing pointless comment.
---------
Co-authored-by: SleeplessOne1917 <28871516+SleeplessOne1917@users.noreply.github.com>
* On registration, automatically set content languages from accept header
* no need to set site language or default language for new user anymore
* fix test
* fix langs
* avoid duplicate writing of new user languages
* Add a blocklist for URLs.
* Fix SQL format
* Make clippy happy.
* Use regex for URL matching.
* Escape regex chars in URLs.
* Use post for modification.
* Make URL block regex static and remove API routes.
* Add date fields to table and use transaction.
* Use Cache for blocklist.
* Rename check_links + move list to parameters of process_markdown.
* SQL format.
* Format, again.
* Remove println.
* Add API test.
* Set a shorter lifetime for regex in debug mode.
* Add missing macro.
* Update lemmy-js-client
* Update api_test/pnpm-lock.yaml
* Don't break other tests
* Use different URL for test
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Nutomic <me@nutomic.com>
* Adding ability to hide posts.
- Adds an post/hide API route.
- Adds a `show_hidden` (default false) to `GetPosts`.
- Adds a `hidden` field to `PostView`.
- Removes the single `post_id` from MarkPostAsRead.
- Fixes#1403
* Add a check to make sure hidden field is true.
* Fixing test.
* Add back semicolon
* Expose LemmyErrorType in lemmy_api_common
* Make conditional compilation gates for utils
* Make it so api_common doesn't pull in unnecessary deps
* Make error type non exhaustive
* Fix formatting
* Format toml
* Add some convenience derives to LemmyError
* Simplify features
* Fix CI compile error
---------
Co-authored-by: SleeplessOne1917 <insomnia-void@protonmail.com>
* Add support for local only community (fixes#1576)
* add filters and tests to db views
* dont federate local only community
* test get apub community http
* tests
* more checks
* wip
* api test
* fix tests
* change community.local_only column to visibility enum
(for private communities)
* sql fmt
* rename vars
* clippy
* fix tests
* update lib
* review
* fix js client version
* update client
* Add markdown rule to add rel=nofollow for all links
* Add markdown image rule to add local image proxy (fixes#1036)
* comments
* rewrite markdown image links working
* add comment
* perform markdown image processing in api/apub receivers
* clippy
* add db table to validate proxied links
* rewrite link fields for avatar, banner etc
* sql fmt
* proxy links received over federation
* add config option
* undo post.url rewriting, move http route definition
* add tests
* proxy images through pictrs
* testing
* cleanup request.rs file
* more cleanup (fixes#2611)
* include url content type when sending post over apub (fixes#2611)
* store post url content type in db
* should be media_type
* get rid of cache_remote_thumbnails setting, instead automatically
take thumbnail from federation data if available.
* fix tests
* add setting disable_external_link_previews
* federate post url as image depending on mime type
* change setting again
* machete
* invert
* support custom emoji
* clippy
* update defaults
* add image proxy test, fix test
* fix test
* clippy
* revert accidental changes
* address review
* clippy
* Markdown link rule-dess (#4356)
* Extracting opengraph_data to its own type.
* A few additions for markdown-link-rule.
---------
Co-authored-by: Nutomic <me@nutomic.com>
* fix setting
* use enum for image proxy setting
* fix test configs
* add config backwards compat
* clippy
* machete
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Merge /site_inbox into /inbox (fixes#4137)
Get rid of different inboxes, only use /inbox
Remove shared_inbox_url db columns
add code migration
move to db migration, fixes
machete
fix sql
drop inbox url unique constraints
Dont create auth cookie in backend (#4136)
dont change individual inboxes to shared inbox
Dont send comment reply to user who has community blocked. Fixes#3684 (#4096)
* Dont send comment reply to user who has community blocked. Fixes#3684
* Adding source instance block check.
* Adding api test.
* Addressing PR comments.
* move site inbox rewrite to db
* fix test
* clippy
* clippy 2
* fix test
* Dont send comment reply to user who has community blocked. Fixes#3684
* Adding source instance block check.
* Adding api test.
* Addressing PR comments.
* Revert "Convert more responses to SuccessResponse. Fixes#2860 (#4058)"
This reverts commit 8deb4e5752.
* Removing purgeitem and deletecustomemoji response.
* Adding back in utils building.
* Handle invalid ban expires values (fixes#4045)
* Adding a few missing expire time checks. Fixing up time conversions. (#4051)
* Adding a few missing expire time checks. Fixing up time conversions.
* Increase settings export wait time.
* get rid of RemoveCommunity.expires
* fmt
* tests
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* feat: Add user setting for toggling avatar gif animations
* Add forgotten post listing mode option to settings form
* Fix compilation error
* Fix formatting
* Change name of column to be more general purpose
* Fix sql format.
* Fix comments
* Running format.
---------
Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* wip
* stuff
* fmt
* fmt 2
* fmt 3
* fix default feature
* use Authorization header
* store ip and user agent for each login
* add list_logins endpoint
* serde(skip) for token
* fix api tests
* A few suggestions for login_token (#3991)
* A few suggestions.
* Fixing SQL format.
* review
* review
* rename cookie
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Remove explicit auth params (ref #3725)
Only take auth via header or cookie. This requires a new version
of lemmy-js-client for api tests to pass.
* rework api_crud
* remove remaining auth params, move logic to session middleware
* fmt, fix test
* update js client
* remove auth param from api tests
* Pass auth as header
* add !
* url vars, setHeader
* cleanup
* fmt
* update
* Updating for new lemmy-js-client.
---------
Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Rework the way 2FA is enabled/disabled (fixes#3309)
* postgres format
* change algo to sha1 for better compat
* review comments
* review
* clippy
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* User can block instances (fixes#2397)
* update comments
* review comments
* use route
* update
* add api test
* update tests
* fix
* fix test
* ci
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* persistent activity queue
* fixes
* fixes
* make federation workers function callable from outside
* log federation instances
* dead instance detection not needed here
* taplo fmt
* split federate bin/lib
* minor fix
* better logging
* log
* create struct to hold cancellable task for readability
* use boxfuture for readability
* reset submodule
* fix
* fix lint
* swap
* remove json column, use separate array columns instead
* some review comments
* make worker a struct for readability
* minor readability
* add local filter to community follower view
* remove separate lemmy_federate entry point
* fix remaining duration
* address review comments mostly
* fix lint
* upgrade actitypub-fed to simpler interface
* fix sql format
* increase delays a bit
* fixes after merge
* remove selectable
* fix instance selectable
* add comment
* start federation based on latest id at the time
* rename federate process args
* dead instances in one query
* filter follow+report activities by local
* remove synchronous federation
remove activity sender queue
* lint
* fix federation tests by waiting for results to change
* fix fed test
* fix comment report
* wait some more
* Apply suggestions from code review
Co-authored-by: SorteKanin <sortekanin@gmail.com>
* fix most remaining tests
* wait until private messages
* fix community tests
* fix community tests
* move arg parse
* use instance_id instead of domain in federation_queue_state table
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
* add logic to prevent downloading remote pictrs images
* apply formatting
* Do not attempt a pictrs fetch if the remote image is also on a pictrs instance
* Do not attempt a pictrs fetch if the remote image is also on a pictrs instance and cache_federated_images is false
* Generalising the no caching option to handle all remote images
* rustfmt
* Return None if the URL is not an image
* Updating defaults.hjson
* fixing typo
* Fixing typo
* Skip cloning the Url unless we need to
* using a HEAD request for checking the content type, saving bandwidth/improving perf
* Removing early returns
* Switching back to GET requests for Content-Type because pictrs does not handle HEAD requests
* Simplifying logic and using metadata_image instead of url if we do not get a pictrs thumbnail
* Removing unused import
* Return None as a thumbnail if caching is disabled
* formatting
---------
Co-authored-by: Djones4822 <david.jones4822@gmail.com>
* update api tests for new moderator view
* chage moderator view to be a listing type in get posts
Note: Internally, the listing type is called ListingType.ModeratorView,
but it's called "Moderator View" in the api endpoint
* fix formatting
* add support for moderator view to list comments
* add api test for moderator view when listing comments
* fix api test formatting
* retry tests
* don't filter out blocked users and communities when using moderator view
* fix cargo tests failing
* fix formatting
* fix previous merge
* Adding ModeratorView to listing_type_enums
* Fixing fmt.
* Adding a default to ListingType.
* Upgrading to use new lemmy-js-client.
---------
Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Dessalines <tyhou13@gmx.com>
* Move admin flag from person to local_user (fixes#3060)
The person table is for federated data, but admin flag can only
apply to local users. Thats why it really belongs in the local_user
table. This will also prevent the federation code from accidentally
overwriting the admin flag
* fmt
* try to fix api tests
* lint
* fix person view
* ci
* ci
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Replace Option<bool> with bool for PostQuery and CommentQuery (#3819)
* Replace Option<bool> from all list queries (#3819)
---------
Co-authored-by: Freek van Zee <freek.van.zee@mediamonks.com>
* Allow filtering posts and comments by whether they were liked/disliked
* Switch to 2 args - liked_only, disliked_only - taking bools
* Make liked_only and disliked_only Option<bool>
* Fix unrelated is_profile_view compilation error
* HTML sanitization in apub code
* Sanitize API inputs
* fmt
* Dont allow html a, img tags
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* add option to only show posts from moderated communities
* rename moderated_only to moderator_view and show blocked users in moderator view
* add test for moderator view
* bump lemmy-js-client version for moderation view tests
* fix yarn lockfile
* retry build
* Delete logfile
* retry checks
* remove unused select statement from list posts
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
* Resolves issue #3685
If user isn't authenticated with resolve_object, only allow a local search instead of possibly making an http request.
* Making sure to validate auth before doing a potential remote lookup.
* add new flag to api
* add new ui settings for local user
* remove extraneous def
* add props to application reg.
* fix clippy updated these
* re-order db schema entries
* remove dupe
* update lemmy sdk
* update lemmy js client
---------
Co-authored-by: Nutomic <me@nutomic.com>
I noticed that stopping the Lemmy process with ctrl+c wasnt working
because the activity channel isnt properly closed. This is now fixed.
Later we should also move the channel from static into LemmyContext,
Im not doing that now to avoid conflicts with #3670.
* Remove SendActivity and Perform traits, rely on channel
These traits arent necessary anymore now that websocket is removed.
Removing them allows us to use normal actix http handler methods
which are much more flexible, and allow using different middlewares
as well as setting response attributes.
* compiling and create post federating
* clippy
* rename methods, join outgoing activities task
* fix api tests
* no unwrap
* conditional compile
* add back getrandom
* make crates optional
* fmt
* add new function build_post_response_deleted_allowed
* PostDelete uses new function build_post_response_deleted_allowed
* RemovePost uses new build_post_response_deleted_allowed function
* code comments about mod or admin flag having other use
* reformat "cargo +nightly fmt --all"
* Try using drone cache plugin
* Try another path
* Include volume
* Fix formatting
* Include fmt
* Exclude cargo dir from prettier
* Don't override cargo
* Just do check
* Add cache key
* Use different cache plugin
* Add clippy
* Try minio
* Add quotes
* Try adding secrets
* Try again
* Again
* Use correct secret formation
* Add back clippy
* Use secret for the root bucket name
* Try drone cache instead
* Add region
* Add path-style option
* Include cargo clippy
* Include everything again
* Fix formatting
* Don't run clippy twice
* Add `allow` statements for tests to pass
* Adjust endpoint to be a secret
* Fix prettier
* Merge & fix tests
* Try to restart the woodpecker test
* Change the ENV var name
---------
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
