Add security checks for activitypub inbox #78

Manually merged
dessalines merged 6 commits from federation-authorisation into main 2020-08-07 15:34:36 +00:00
Owner

TODO:

  • consider whether we need to check the exact actor ID for create/update activities (or if comparing the domain is enough)
  • determine whether the checks for undo/remove are enough
  • maybe return a different error for slur checks (or just censor the slurs instead)
  • figure out why some tests are failing (either tests are wrong, or security checks are wrong)
TODO: - consider whether we need to check the exact actor ID for create/update activities (or if comparing the domain is enough) - determine whether the checks for undo/remove are enough - maybe return a different error for slur checks (or just censor the slurs instead) - figure out why some tests are failing (either tests are wrong, or security checks are wrong)
dessalines reviewed 2020-08-06 13:37:46 +00:00
@ -66,3 +66,4 @@
.await?;
if !is_mod_or_admin {
// TODO: more accurately, not_a_mod_or_admin?
return Err(APIError::err("not_an_admin").into());
Owner

Good idea, add a not_a_mod_or_admin string to en.json

Good idea, add a `not_a_mod_or_admin` string to `en.json`
nutomic marked this conversation as resolved
dessalines reviewed 2020-08-06 13:42:25 +00:00
@ -168,0 +174,4 @@
.as_single_xsd_string()
.unwrap()
.to_string();
check_slurs(&content)?;
Owner

Hrm, for comments, its best to do remove_slurs

let content_slurs_removed = remove_slurs(&data.content.to_owned());

Also just as a technical consideration / thought, if a comment isn't in our local DB because its been slur rejected (or rejected for any reason), any replies might also not come through because we don't have that comment parent_id.

Hrm, for comments, its best to do remove_slurs `let content_slurs_removed = remove_slurs(&data.content.to_owned());` Also just as a technical consideration / thought, if a comment isn't in our local DB because its been slur rejected (or rejected for any reason), any replies might also not come through because we don't have that comment parent_id.
Author
Owner

Thanks, didnt notice that it was handled differently for comments.

This means that federation for posts with slurs will just fail silently then, gonna need some thinking on how to handle it properly (mainly the case where two instances have different slur filters).

Thanks, didnt notice that it was handled differently for comments. This means that federation for posts with slurs will just fail silently then, gonna need some thinking on how to handle it properly (mainly the case where two instances have different slur filters).
dessalines reviewed 2020-08-06 13:43:44 +00:00
@ -182,3 +187,3 @@
updated: note.updated().map(|u| u.to_owned().naive_local()),
deleted: None,
ap_id: note.id_unchecked().unwrap().to_string(),
ap_id: check_actor_domain(note, expected_domain)?,
Owner

This is tricky, and maybe we should move the check out of the commentForm here. Because on comment creates / edits, we should check the specific actor_id and make sure they're the same.

But on comment removes, the actor id can be someone else.

This is tricky, and maybe we should move the check out of the commentForm here. Because on comment creates / edits, we should check the specific actor_id and make sure they're the same. But on comment removes, the actor id can be someone else.
Author
Owner

Look into the check_actor_domain() function, thats exactly what it does based on whether or not expected_domain is Some or None.

Look into the `check_actor_domain()` function, thats exactly what it does based on whether or not `expected_domain` is Some or None.
dessalines reviewed 2020-08-06 13:50:58 +00:00
@ -337,0 +351,4 @@
.unwrap()
.to_string();
let title = group.inner.preferred_username().unwrap().to_string();
// TODO: should be parsed as html and tags like <script> removed (or use markdown source)
Owner

The titles and preferred usernames I'm just treating as text, not html. But ya all the content type fields, like summary, do need this. It'd probably be easiest to do it this way, like the docs have, and not have to deal with html conversions at all.

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "summary": "A simple note",
  "type": "Note",
  "mediaType": "text/markdown",
  "content": "## A simple note\nA simple markdown `note`"
}
The titles and preferred usernames I'm just treating as text, not html. But ya all the content type fields, like summary, do need this. It'd probably be easiest to do it this way, like the docs have, and not have to deal with html conversions at all. ``` { "@context": "https://www.w3.org/ns/activitystreams", "summary": "A simple note", "type": "Note", "mediaType": "text/markdown", "content": "## A simple note\nA simple markdown `note`" } ```
Author
Owner

I think we are actually sending markdown now, but we should still sanitise that, could be that our markdown parser (or the parser in a Lemmy client) actually renders HTML that is embedded in markdown. Anyway its not related to this PR, I just moved that comment.

I think we are actually sending markdown now, but we should still sanitise that, could be that our markdown parser (or the parser in a Lemmy client) actually renders HTML that is embedded in markdown. Anyway its not related to this PR, I just moved that comment.
dessalines reviewed 2020-08-06 13:52:08 +00:00
@ -465,2 +477,4 @@
// because that would result in a database error (same data inserted twice)
// this seems to be the "easiest" stable alternative for remove_item()
to.retain(|x| *x != sender.get_shared_inbox_url());
to.retain(|x| *x != community.get_shared_inbox_url());
Owner

I still have to add columns for the inbox, so we aren't just generating them randomly.

I still have to add columns for the inbox, so we aren't just generating them randomly.
Author
Owner

What do you mean?

What do you mean?
dessalines reviewed 2020-08-06 13:56:19 +00:00
@ -57,2 +64,3 @@
let post = PostForm::from_apub(&page, client, pool).await?;
let post = PostForm::from_apub(&page, client, pool, Some(user.actor_id()?)).await?;
// TODO: not sure if it makes sense to check for the exact user, seeing as we already check the domain
Owner

Seems like bc we have checks where the actor and the post_creator / comment_creator are different (for removes), then the checks unfortunately will have to be all in these files, and not in the from_apub.

Seems like bc we have checks where the actor and the post_creator / comment_creator are different (for removes), then the checks unfortunately will have to be all in these files, and not in the `from_apub`.
Owner

But yes I also do think its a good idea to check not just the domain, but the exact actor_id.

But yes I also do think its a good idea to check not just the domain, but the exact actor_id.
dessalines reviewed 2020-08-06 13:59:30 +00:00
@ -42,1 +49,4 @@
let remove = Remove::from_any_base(activity)?.unwrap();
let actor = get_user_from_activity(&remove, client, pool).await?;
let community = get_community_from_activity(&remove, client, pool).await?;
// TODO: we dont federate remote admins at all, and remote mods arent federated properly
Owner

I don't see why we couldn't just start that now. A federated mod would pass this check.

I'll add in the chat about removes.

I don't see why we couldn't just start that now. A federated mod would pass this check. I'll add in the chat about removes.
dessalines reviewed 2020-08-06 14:00:18 +00:00
@ -55,0 +72,4 @@
} else {
Ok(())
}
}
Owner

Nice, I like this.

Nice, I like this.
dessalines reviewed 2020-08-06 14:03:52 +00:00
@ -110,1 +135,4 @@
let dislike = Dislike::from_any_base(undo.object().to_owned().one().unwrap())?.unwrap();
check_is_undo_valid(&undo, &dislike)?;
// TODO: need to implement Undo<Dislike>
Owner

Should be easy enough, just call the same function as the receive_undo_comment_like, as both just remove the comment_like row anyway:

blocking(pool, move |conn| CommentLike::remove(conn, &like_form)).await??;

Should be easy enough, just call the same function as the receive_undo_comment_like, as both just remove the comment_like row anyway: `blocking(pool, move |conn| CommentLike::remove(conn, &like_form)).await??; `
Author
Owner

Yep, but I will do it after this PR.

Yep, but I will do it after this PR.
dessalines reviewed 2020-08-06 14:04:36 +00:00
@ -43,0 +45,4 @@
// ensure that update and actor come from the same instance
let user = get_user_from_activity(&update, client, pool).await?;
update.id(user.actor_id()?.domain().unwrap())?;
Owner

This should check not just the domain, but the exact actor id.

...

Oh wait I think you're doing that below. Just the below check should be good enough then.

This should check not just the domain, but the exact actor id. ... Oh wait I think you're doing that below. Just the below check should be good enough then.
dessalines reviewed 2020-08-06 14:08:53 +00:00
@ -73,2 +73,3 @@
let community = get_community_id_from_activity(&activity).await;
// TODO: i dont think this works for Announce/Undo activities
//let community = get_community_id_from_activity(&activity).await;
Owner

Looks like the check makes sure its in the allowed_instances list. Not sure I understand removing this. If you've blocked an instance, you don't want to receive anything, including announcements from it.

Looks like the check makes sure its in the allowed_instances list. Not sure I understand removing this. If you've blocked an instance, you don't want to receive anything, including announcements from it.
Author
Owner

Problem is that Undo and/or Announce activities dont have the community set in cc, so this was throwing errors (dont remember exactly what it was about).

Problem is that Undo and/or Announce activities dont have the community set in cc, so this was throwing errors (dont remember exactly what it was about).
Author
Owner

Gonna have to test this again, we might have to move this check into the create/update etc files.

Gonna have to test this again, we might have to move this check into the create/update etc files.
dessalines reviewed 2020-08-06 14:10:38 +00:00
@ -77,3 +78,3 @@
check_is_apub_id_valid(&community)?;
verify(&request, actor.as_ref())?;
// TODO: probably better to do this after, so we dont store activities that fail a check somewhere
Owner

Good idea, you could move this below the match, and do

let res = match kind ...
insert_activity...
res
Good idea, you could move this below the match, and do ``` let res = match kind ... insert_activity... res ```
Author
Owner

Done, I thought maybe there was a reason to do it this way.

Done, I thought maybe there was a reason to do it this way.
dessalines reviewed 2020-08-06 14:11:40 +00:00
@ -161,3 +162,3 @@
let note = Note::from_any_base(update.object().as_one().unwrap().to_owned())?.unwrap();
let private_message_form = PrivateMessageForm::from_apub(&note, client, pool).await?;
let domain = Some(update.id_unchecked().unwrap().to_owned());
Owner

These domain checks aren't needed right? Just check the actor_id.

These domain checks aren't needed right? Just check the actor_id.
Author
Owner

Maybe not exactly needed, but cant hurt to have them. And like we discussed, checking the actor id doesnt really help with security, so this is the easier solution (I added some docs about this too).

Maybe not exactly needed, but cant hurt to have them. And like we discussed, checking the actor id doesnt really help with security, so this is the easier solution (I added some docs about this too).
dessalines reviewed 2020-08-06 14:13:43 +00:00
@ -133,3 +143,4 @@
apub: &Self::ApubType,
client: &Client,
pool: &DbPool,
expected_domain: Option<Url>,
Owner

Ya this can pry be removed. I think the http sig verify, and the checking that its the correct actor_id, should be enough. That's 2 checks, a domain would be a third.

Ya this can pry be removed. I think the http sig verify, and the checking that its the correct actor_id, should be enough. That's 2 checks, a domain would be a third.
Author
Owner

No, this is the main check to make that eg in case of a Create<Page>, both Create and Page are coming from the same instance.

No, this is the main check to make that eg in case of a `Create<Page>`, both Create and Page are coming from the same instance.
dessalines reviewed 2020-08-06 14:18:18 +00:00
@ -214,2 +225,4 @@
.as_ref()
.map(|c| c.as_single_xsd_string().unwrap().to_string());
check_slurs(&name)?;
check_slurs_opt(&body)?;
Owner

I must have overlooked where this got replaced, or I had in wrong in the first place.

Both here and in the api/post.rs, its better to do remove_slurs on the post body, not check_slurs.

Basically any long content field shouldn't reject, but just softly replace the slurs.

I must have overlooked where this got replaced, or I had in wrong in the first place. Both here and in the `api/post.rs`, its better to do `remove_slurs` on the post body, not `check_slurs`. Basically any long content field shouldn't reject, but just softly replace the slurs.
Author
Owner

Okay, done.

Okay, done.
Author
Owner

Updated. Turns out the check_is_apub_id_valid(&community)? in shared_inbox actually works fine, but there were some other problems that I fixed. Also the tests for remove post/comment are failing, but its working fine when I try it manually, so I could use your help there. Other than the failing tests, this should be good to merge.

Updated. Turns out the `check_is_apub_id_valid(&community)?` in shared_inbox actually works fine, but there were some other problems that I fixed. Also the tests for remove post/comment are failing, but its working fine when I try it manually, so I could use your help there. Other than the failing tests, this should be good to merge.
nutomic changed title from WIP: Add security checks for activitypub inbox to Add security checks for activitypub inbox 2020-08-07 13:19:37 +00:00
Owner

I am getting some uniqueness errors:

postgres_beta_1   | 2020-08-07 14:17:23.140 UTC [56] ERROR:  duplicate key value violates unique constraint "idx_activity_unique_apid
"
postgres_beta_1   | 2020-08-07 14:17:23.140 UTC [56] DETAIL:  Key ((data ->> 'id'::text))=(http://lemmy-beta:8550/activities/delete/e
cb32657-048c-4a08-97db-27bc0d9d2d75) already exists.
postgres_beta_1   | 2020-08-07 14:17:23.140 UTC [56] STATEMENT:  INSERT INTO "activity" ("user_id", "data", "local", "updated") VALUE
S ($1, $2, $3, DEFAULT) RETURNING "activity"."id", "activity"."user_id", "activity"."data", "activity"."local", "activity"."published
", "activity"."updated"
lemmy-beta_1      | [2020-08-07T14:17:23Z ERROR actix_http::response] Internal Server Error: LemmyError { inner: Failed to insert act
ivity into database: duplicate key value violates unique constraint "idx_activity_unique_apid"
...
...
postgres_beta_1   | 2020-08-07 14:17:23.602 UTC [56] ERROR:  duplicate key value violates unique constraint "idx_activity_unique_apid
"
postgres_beta_1   | 2020-08-07 14:17:23.602 UTC [56] DETAIL:  Key ((data ->> 'id'::text))=(http://lemmy-beta:8550/activities/undo/c26
c3dd4-9325-4125-b833-bd0c3e69bffa) already exists.
postgres_beta_1   | 2020-08-07 14:17:23.602 UTC [56] STATEMENT:  INSERT INTO "activity" ("user_id", "data", "local", "updated") VALUE
S ($1, $2, $3, DEFAULT) RETURNING "activity"."id", "activity"."user_id", "activity"."data", "activity"."local", "activity"."published
", "activity"."updated"
lemmy-beta_1      | [2020-08-07T14:17:23Z ERROR actix_http::response] Internal Server Error: LemmyError { inner: Failed to insert act
ivity into database: duplicate key value violates unique constraint "idx_activity_unique_apid"
I am getting some uniqueness errors: ``` postgres_beta_1 | 2020-08-07 14:17:23.140 UTC [56] ERROR: duplicate key value violates unique constraint "idx_activity_unique_apid " postgres_beta_1 | 2020-08-07 14:17:23.140 UTC [56] DETAIL: Key ((data ->> 'id'::text))=(http://lemmy-beta:8550/activities/delete/e cb32657-048c-4a08-97db-27bc0d9d2d75) already exists. postgres_beta_1 | 2020-08-07 14:17:23.140 UTC [56] STATEMENT: INSERT INTO "activity" ("user_id", "data", "local", "updated") VALUE S ($1, $2, $3, DEFAULT) RETURNING "activity"."id", "activity"."user_id", "activity"."data", "activity"."local", "activity"."published ", "activity"."updated" lemmy-beta_1 | [2020-08-07T14:17:23Z ERROR actix_http::response] Internal Server Error: LemmyError { inner: Failed to insert act ivity into database: duplicate key value violates unique constraint "idx_activity_unique_apid" ... ... postgres_beta_1 | 2020-08-07 14:17:23.602 UTC [56] ERROR: duplicate key value violates unique constraint "idx_activity_unique_apid " postgres_beta_1 | 2020-08-07 14:17:23.602 UTC [56] DETAIL: Key ((data ->> 'id'::text))=(http://lemmy-beta:8550/activities/undo/c26 c3dd4-9325-4125-b833-bd0c3e69bffa) already exists. postgres_beta_1 | 2020-08-07 14:17:23.602 UTC [56] STATEMENT: INSERT INTO "activity" ("user_id", "data", "local", "updated") VALUE S ($1, $2, $3, DEFAULT) RETURNING "activity"."id", "activity"."user_id", "activity"."data", "activity"."local", "activity"."published ", "activity"."updated" lemmy-beta_1 | [2020-08-07T14:17:23Z ERROR actix_http::response] Internal Server Error: LemmyError { inner: Failed to insert act ivity into database: duplicate key value violates unique constraint "idx_activity_unique_apid" ```
Author
Owner

What are you doing to get them?

What are you doing to get them?
dessalines manually merged commit cabbbf0afd into main 2020-08-07 15:34:36 +00:00
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: LemmyNet/lemmy#78
No description provided.