66
Creepy crawlies
(people.kernel.org)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 30 Aug 2026
66 points (98.5% liked)
Opensource
6636 readers
246 users here now
A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!
⠀
founded 2 years ago
MODERATORS
They're not scraping to cache or store, they're operating as an agent - scraping or single user requests.
Which is obviously bad and damaging, especially on their scale and on repeatedly fetched websites that they could be caching.
Google indexed the entire web. It's baffling that such indexing is not the norm on these huge providers.
Just my interpretation anyway.
I considered this, but would agents really ask for single commits with such frequency? They tend to get individual files via HTML, or do a git clone if they needed commit history for some reason.