LemmyNet/lemmy-stats-crawler
2
1
Fork 0
Code Issues 3 Pull requests Projects Releases Wiki Activity

Recursive, parallel crawl #11

Merged
nutomic merged 2 commits from recursive-crawl into main 2022-05-13 11:37:53 +00:00
Conversation 0 Commits 2 Files changed 5 +326 -172
nutomic commented 2022-05-10 00:55:00 +00:00
Owner
Copy link

This makes the crawl fully async (using recursion), so that instances can be crawled in parallel. In my test, it reduced crawl time from 80s to 11s.

This makes the crawl fully async (using recursion), so that instances can be crawled in parallel. In my test, it reduced crawl time from 80s to 11s.
nutomic added 2 commits 2022-05-10 00:55:01 +00:00
nutomic force-pushed recursive-crawl from cb5f20f397 to e6cd77cf87 2022-05-10 01:37:11 +00:00 Compare
nutomic force-pushed recursive-crawl from e6cd77cf87 to c254e50211 2022-05-10 09:49:52 +00:00 Compare
nutomic added 1 commit 2022-05-10 22:32:32 +00:00
nutomic force-pushed recursive-crawl from ce3d15ffe7 to 0079e72759 2022-05-10 22:56:54 +00:00 Compare
nutomic merged commit 575672cbe3 into main 2022-05-13 11:37:53 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LemmyNet/lemmy-stats-crawler#11
No description provided.
Delete branch "recursive-crawl"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?