169
you are viewing a single comment's thread
view the rest of the comments
[-] Zak@lemmy.world 31 points 1 week ago

I host a few sites and I haven't had a problem yet. Sure, logs suggest a majority of traffic is bots, but the cheap VPS I'm hosting on hasn't melted so I haven't been motivated to do anything about it.

[-] Dave@piefed.nz 27 points 1 week ago* (last edited 1 week ago)

Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite "expensive".

Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.

[-] Zak@lemmy.world 9 points 1 week ago

I believe you, but the linked article is not about hosting Lemmy.

[-] Dave@piefed.nz 9 points 1 week ago* (last edited 1 week ago)

No it's not, it was more of a tangential comment.

I think this is the photography site he's talking about.

He mentioned "so many scrapers hit my site that it saturated my upload with generated content feeding the them." which seemed unusual to me. I'm not sure why such a site would have a public upload. It doesn't feel like a site that would have a technical challenge with the scrapers, but I also get the impression the main issue is the thought of hosting all his photos just for AI scrapers to train their image generation models.

[-] elephantium@lemmy.world 5 points 1 week ago

saturated my upload

I interpreted that as LP having the site running in their basement, and responding to scrapers saturates their outgoing bandwidth.

[-] Dave@piefed.nz 1 points 1 week ago

Ah that would make more sense!

[-] ms_lane@lemmy.world 9 points 1 week ago

Why not just use more static pages?

4chan has been 'static' forever, the backend builds each page every ~15s, pages served to users are always static. It's 'justwerked' for 20 years now on a shoestring.

[-] Dave@piefed.nz 12 points 1 week ago

Lemmy is built as a separate web app and back end that communicate via APIs. It makes it easy to support apps and alternate web interfaces, unlike Piefed which only started adding an API reasonably recently and doesn't have a fraction of the eco system that Lemmy has.

But I think an important part is that Lemmy is explicitly customised for users. While bots are getting the same logged out version, the intent of Lemmy is not to be scraped by bots but it's to be custom for each user. You're supposed to log in, subscribe to communities you like, block people you don't like, and have posts you've already seen hidden (if you set this in your settings).

Serving a custom page to each user based on their settings, blocks, and subscriptions isn't really suited to static pages.

[-] rollin@piefed.social 9 points 1 week ago* (last edited 1 week ago)

Sounds like 4chan rolled their own server-side page cache. It probably made more sense for 4chan to do that then, back when the internet was still pretty new.

Nowadays there are a few packages that perform a similar function, like Varnish. Varnish picks up requests from the internet and where possible, it returns the response as static HTML from its own cache.

There are other types of caching at different stages of the request lifecycle that can reduce the gap between dynamic and static website performance.

[-] The_Decryptor@aussie.zone 3 points 1 week ago

Back in the day people used Squid for that if they needed it.

[-] nykula@piefed.social 3 points 1 week ago

There were attempts to rewrite imageboard software with modern dynamic frameworks and caching infrastructure, but it scaled worse than oldschool index and thread html regeneration on post event, and was cancelled. The effort was called something like Infinity, by 8chan.

[-] frongt@lemmy.zip 3 points 1 week ago

It doesn't need to. Like any similar site, it can cache a lot of that in databases like redis, or cache the rendered page in part or in whole.

[-] Dave@piefed.nz 4 points 1 week ago

Parts will be cached but it's per instance. For a 100 user instance, what are the chances that a recent cache of All page 74 is available? Bots are crawling every page, and most pages are content from last month, last year, or 3 years ago.

[-] veniasilente@lemmy.dbzer0.com 2 points 1 week ago

I would expect Lemmy instances to have some limiters set for guest connections, especially for ones coming from datacenters. Like, it should be easy and senseful to only link and cache up to All page 10, or Hot page 5, and refuse guest loads of anything beyond that.

[-] Dave@piefed.nz 2 points 1 week ago

Ultimately this has been a recent-ish problem, and Lemmy simply doesn't have this sort of function yet. It's a goo thought, though.

As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don't limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.

[-] GameGod@lemmy.ca 2 points 1 week ago* (last edited 1 week ago)

All the logged-out views in Lemmy should be cached, otherwise you're going to have scalability problems. edit: you can probably do this without even touching the Lemmy code. There's already nginx serving lemmy.ca, which can be configured to cache it and respect the Vary header. (I suspect the admins are already way ahead of me here, but I just wanted to share the general approach, which is pretty universal.)

this post was submitted on 08 Nov 2026
169 points (98.3% liked)

Selfhosted

63989 readers
1280 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS