Proxy pictrs requests through Lemmy (fixes #371) #77

nutomic · 2020-08-04T14:57:17Z

nutomic commented

2020-08-04 14:57:17 +00:00

Not sure which thumbnail sizes or other features we are using, but everything seems to work fine in the frontend.

I might also add rate limit and auth check for uploads while I'm at it, but not sure how to make it send the jwt string.

Not sure which thumbnail sizes or other features we are using, but everything seems to work fine in the frontend. I might also add rate limit and auth check for uploads while I'm at it, but not sure how to make it send the jwt string.

dessalines commented

2020-08-04 15:09:44 +00:00

I'll try this out too. The main thing is that the urls don't change, and that its still accessible to the front end, and back end.

dessalines commented

2020-08-04 15:14:15 +00:00

Take a look at the lib.rs fetch_pictrs function. I think that might need to be checked too, cause its required for thumbnails.

edit: actually maybe its fine, the docker pictrs didn't change.

Take a look at the `lib.rs fetch_pictrs` function. I think that might need to be checked too, cause its required for thumbnails. edit: actually maybe its fine, the docker pictrs didn't change.

dessalines commented

2020-08-04 15:32:48 +00:00

The thumbnail fetching seems good.

One issue I did notice tho, is that I'm getting timeouts when I try to upload any file larger than ~1 MB.

lemmy_1     | [2020-08-04T15:31:31Z ERROR actix_http::response] Internal Server Error: Timeout
lemmy_1     | [2020-08-04T15:31:31Z INFO  actix_web::middleware::logger] 172.22.0.1:37376 "POST /pictrs/image HTTP/1.1" 500 34 "http://localhost:8536/create_post" "Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" 5.01821

The thumbnail fetching seems good. One issue I did notice tho, is that I'm getting timeouts when I try to upload any file larger than ~1 MB. ``` lemmy_1 | [2020-08-04T15:31:31Z ERROR actix_http::response] Internal Server Error: Timeout lemmy_1 | [2020-08-04T15:31:31Z INFO actix_web::middleware::logger] 172.22.0.1:37376 "POST /pictrs/image HTTP/1.1" 500 34 "http://localhost:8536/create_post" "Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" 5.01821 ```

nutomic commented

2020-08-04 17:14:15 +00:00

Okay I added a rate limit for image uploads, but it doesnt seem to work, I can still upload as many images in a row as I want...

I also tried to upload a 5.7 MB image (from this site) and it worked without problems in the docker/dev/ setup.

Okay I added a rate limit for image uploads, but it doesnt seem to work, I can still upload as many images in a row as I want... I also tried to upload a 5.7 MB image ([from this site](https://edmullen.net/test/imagefiletest.php)) and it worked without problems in the `docker/dev/` setup.

dessalines commented

2020-08-04 21:48:29 +00:00

I think I know why image rate limiting isn't working, check the mod.rs impl RateLimit.

And then the image route needs image, not message.

      web::resource("/pictrs/image")
        .wrap(rate_limit.image())
        .route(web::post().to(upload)),

I think I know why image rate limiting isn't working, check the mod.rs `impl RateLimit`. And then the image route needs image, not message. ``` web::resource("/pictrs/image") .wrap(rate_limit.image()) .route(web::post().to(upload)), ```

dessalines commented

2020-08-04 22:16:40 +00:00

Also I tried a few more pictures, and about 75% of them failed, with the same timeout error as above. It seems to stop working after the first few uploads go through.

nutomic commented

2020-08-05 11:44:05 +00:00

Fixed the rate limit, but strangely I could still upload more than 20 images in a row, without any problems. So neither is the rate limit working, nor am I seeing the upload errors that you get.

dessalines commented

2020-08-05 13:45:03 +00:00

Maybe its my local machine, I'll try deploying to test.lemmy.ml to see if it works.

dessalines commented

2020-08-05 14:51:36 +00:00

Figured out the rate limit issue:

rate_limit/mod.rs, line 114 should be false

        RateLimitType::Image => {
          limiter.check_rate_limit_full(
            self.type_,
            &ip_addr,
            rate_limit.image,
            rate_limit.image_per_second,
            false,
          )?;

Otherwise its a check only.

Figured out the rate limit issue: `rate_limit/mod.rs`, line 114 should be false ``` RateLimitType::Image => { limiter.check_rate_limit_full( self.type_, &ip_addr, rate_limit.image, rate_limit.image_per_second, false, )?; ``` Otherwise its a check only.

nutomic commented

2020-08-05 15:47:54 +00:00

Thanks, fixed. About your upload error, we could try how it goes running it on a server, or ask asonix if he has some idea.

nutomic referenced this issue from a commit

2020-08-05 16:00:03 +00:00

Proxy pictrs requests through Lemmy (fixes #371) (#77) fix check_only value for image rate limit Fix image rate limit Add rate limit for image uploads Proxy pictrs requests through Lemmy (fixes #371) Co-authored-by: Felix Ableitner <me@nutomic.com> Reviewed-on: https://yerbamate.dev/LemmyNet/lemmy/pulls/77

nutomic merged commit c34fffc2c4 into main

2020-08-05 16:00:03 +00:00

dessalines commented

2020-08-05 18:27:42 +00:00

I was wrong, I shouldn't have merged this. I'm getting that timeout error in production now.

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: LemmyNet/lemmy#77