Commit graph

38 commits

Author SHA1 Message Date
nutomic
575672cbe3 Merge pull request 'Recursive, parallel crawl' (#11) from recursive-crawl into main
Reviewed-on: https://yerbamate.ml/LemmyNet/lemmy-stats-crawler/pulls/11
2022-05-13 11:37:52 +00:00
nutomic
0c3ba08d6c Merge pull request 'Use lemmy structs' (#10) from lemmy-structs into main
Reviewed-on: https://yerbamate.ml/LemmyNet/lemmy-stats-crawler/pulls/10
2022-05-13 11:37:45 +00:00
0079e72759 Add stderrlog, structopt for better logging and params 2022-05-11 00:56:42 +02:00
c254e50211 Recursive, parallel crawl 2022-05-10 11:49:41 +02:00
2e2a4888d0 Use lemmy structs 2022-05-10 01:29:42 +02:00
nutomic
8509c19f50 Merge pull request 'Add min version check to avoid including old instances' (#9) from min-version into main
Reviewed-on: https://yerbamate.ml/LemmyNet/lemmy-stats-crawler/pulls/9
2022-05-03 12:19:39 +00:00
574536b151 Add min version check to avoid including old instances 2022-04-21 14:10:54 +02:00
9546936ddf Convert instance name to lowercase 2022-04-04 13:14:34 +02:00
4e95487842 Change default crawl depth to 20 2022-01-31 23:45:21 +01:00
a1115d474b Include require_application in crawl output 2022-01-29 17:18:29 +01:00
f110554993 Add command line param to exclude instances from crawl 2021-06-17 16:48:45 +02:00
nutomic
fefd708777 Merge pull request 'Sorting by active monthly desc.' (#6) from sort_active_monthly into main
Reviewed-on: https://yerbamate.ml/LemmyNet/lemmy-stats-crawler/pulls/6
2021-04-27 13:10:47 +00:00
60563fc7d9 Exclude test instances from crawl 2021-04-27 15:09:02 +02:00
d92de61d9c Include description field (if it is under 150 chars) 2021-04-27 15:06:28 +02:00
1751035334 Add support for Lemmy api v2 and v3 2021-04-12 17:08:14 +02:00
cd04d54b9a Update for API v3 2021-04-05 20:33:06 +02:00
f56a7dde4f Sorting by active monthly desc. 2021-03-24 19:43:59 -04:00
0fcf8fe523 Remove duplicate code, clippy fixes 2021-03-17 03:09:21 +01:00
95107b6a39 Include number of failed instances 2021-03-17 03:04:57 +01:00
5d5a9b863d Pretty print 2021-03-17 02:56:55 +01:00
9621b91f7c Add support for command line parameters 2021-03-15 22:39:09 +01:00
f01e077020 Add max depth parameter for crawl 2021-03-15 22:19:48 +01:00
5a09fa46c4 Dont redo crawls for unreachable instances 2021-03-11 15:47:55 +01:00
48a164de5a Move crawl code into separate file 2021-03-11 14:34:44 +01:00
ce909d5e72 Simplify crawl by using queue 2021-03-11 14:30:30 +01:00
0690824e9e Minor reduction in dependencies 2021-03-11 12:44:41 +01:00
f3876b4745 Remove instance filtering/sorting 2021-03-11 12:22:28 +01:00
71cdf40bae Simplified crawl loop, improved output 2021-03-11 03:25:51 +01:00
33580c52ea Add total_instances count, use consts 2021-03-11 03:01:15 +01:00
4426b0edf9 Improved crawling code, allow null icon 2021-03-11 02:41:24 +01:00
342fe6ded8 Make requests to same instance in parallel 2021-03-11 02:30:56 +01:00
27c23bd5d8 Improve output, refactor code, add timeout 2021-03-11 02:25:10 +01:00
c45771d9e7 Add instance name and icon 2021-03-10 18:17:17 +01:00
9cbed8f2a9 Output instance info as json 2021-03-10 18:06:21 +01:00
cb4e6e221a Add license 2021-03-10 17:39:43 +01:00
45ec994271 Crawl for number of online and total users 2021-03-09 19:51:35 +01:00
21be5c58ee Fetch basic data from lemmy.ml 2021-03-09 19:11:13 +01:00
5f871dcc96 Initial commit 2021-03-09 18:19:23 +01:00