this post was submitted on 04 Aug 2026
111 points (100.0% liked)

Feddit UK

1714 readers
1 users here now

Community for the Feddit UK instance.
A place to log issues, and for the admins to communicate with everyone.

founded 3 years ago
MODERATORS
GreatAlbatross@feddit.uk
flamingos@feddit.uk
111
We're back! (feddit.uk)
submitted 1 week ago* (last edited 1 week ago) by flamingos@feddit.uk to c/feddituk@feddit.uk
29 comments fedilink hide all child comments
 

Futurama: We're back baby

Sorry about the long downtime everyone. What happened is sometime around 20:00 yesterday the instance went down. I wouldn't be able to ssh into the server until the morning, so I tried restarting the server through our hosting's web interface, but this didn't work.

When I got home I found the reason Lemmy wasn't working was because the database had suffered some corruption. Thankfully, the only lost data was 3 posts, but some data got duplicated in random tables that shouldn't be and this wasn't the easiest thing to track down.

Why did the server go down in the first place?

I don't actually know the real reason, what I suspect happened is that the frontend got taken down by the rampant scraping we've been subject to. To put it into prospective, just look at the jump in the size of our web logs:

Screenshot of compressed nginx logs from different days showing a jump from 60MB to 1.6G

These are compressed plain text, like it's unreal.

But back to the server, I think when I restarted the server to get everything back up, docker killed the database before it had shutdown properly hence the corruption.

Anyway, it's sorted now. It's going to be a bit quiet until we've caught up with the activities we've missed, so enjoy the quiet while you can.

top 29 comments
sorted by: hot top controversial new old
[–] poVoq@slrpnk.net 17 points 1 week ago (1 children)

Yeah, those gigabyte sized access logs is something we have seen for weeks.

Anubis has also gotten largely ineffective, and yesterday I had some time to sample some of the access log IPs and it is clear that this is a coordinated scraping attack from the Huawei ASN that additionally employs residential proxies mostly in Pakistan, Vietnam and Brazil.

I don't really have a good idea how to deal with it, but it is a complete PITA.

[–] filcuk@feddit.uk 2 points 1 week ago (1 children)

Would you know why Anubis isn't as effective?

[–] poVoq@slrpnk.net 3 points 1 week ago

Anubis doesn't really do anything if the bot knows about it and doesn't send the bits Anubis looks for. There are a few other heuristics that you can enable in the Anubis config, but it is a principal issue with how Anubis works. The general idea is that if a bot doesn't send the things Anubis looks for, then it should be possible to distinguish it from regular browsers by that, but then you are back at a manual whack a mole game and given the extensive use of residential proxies these days it is very much a losing battle.

Anubis recently added a honeypot with automatic IP logging though that might help in combination with fail2ban. But in the end if scraper bot authors are already circumventing Anubis intentionally, it is probably also only a temporary respite.

[–] matelt@feddit.uk 15 points 1 week ago (1 children)

All hail flamingo-cant!

[–] Zombie@feddit.uk 11 points 1 week ago

More like flamingo-can!

Hails!

[–] wewbull@feddit.uk 11 points 1 week ago* (last edited 1 week ago) (1 children)

Well done! We knew you would come through, honest!

On the scraping.... That's crazy. Is there anything that can be done to protect things?

[–] Snoopy@piefed.social 2 points 1 week ago

Using Anubis. But i'm not technical, anyzay that's a good new :)

[–] tetris11@feddit.uk 8 points 1 week ago* (last edited 1 week ago)

Phew! I was worried for a bit.

I notice that feddit.org use Anubis for bot mitigation. Is this something we do too?

[–] mannycalavera@feddit.uk 7 points 1 week ago (1 children)

Yikes sounds like a nightmare. I hope you managed to get some sleep!

[–] flamingos@feddit.uk 8 points 1 week ago

I did and it was surprisingly pretty good.

[–] TedZanzibar@feddit.uk 6 points 1 week ago

Those logs are insane! 1.6GB compressed?! Any idea what the load/throughput was like during that time?

[–] theOneTrueSpoon@feddit.uk 6 points 1 week ago (1 children)

Thank you for your efforts to fix it!

Just wondering, is there a backup place where things like this can be communicated? Blue sky or mastodon for example?

[–] echodot@feddit.uk 5 points 1 week ago

See this is why I hate working in the IT department, because it involves using computers, and they suck.

[–] Technoworcester@feddit.uk 4 points 1 week ago

Thanks for all your hard work!

[–] fakeman_pretendname@feddit.uk 4 points 1 week ago

Thank you for fixing it and keeping things going :)

[–] addie@feddit.uk 4 points 1 week ago

Awesome work, flamingos-cant. Appreciate all you do for us.

[–] gilesroberts@feddit.uk 4 points 1 week ago* (last edited 1 week ago)

What's the database underneath lemmy? Is there a transactional option so that it fails safe in the event of unexpected shut down?

Also thank you very much for fixing.

[–] BeardedGingerWonder@feddit.uk 3 points 1 week ago

Thanks Flamingos!

[–] alecbowles@feddit.uk 3 points 1 week ago (2 children)

Glad to see we are back. Those access logs are crazy.

I’m sure you will fine a lot of Chrome/12X, Chrome/13X and Chrome/14X user agents there. Seems to be the favourites of attackers/scrapers recently.

On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.

Thank you so much for the work you do!

[–] flamingos@feddit.uk 4 points 1 week ago (1 children)

On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.

Someone did actually reach out to my personal Mastodon account to ask what was happening.

Like florge said, there is a Matrix space that mostly serves this function, but that does require you to have a Matrix account so isn't the best for public notices. It'd be nice to have a page (eg status.feddit.uk) where I could put messages about what's happening. I'll need to look to see if something like that exists.

[–] wewbull@feddit.uk 2 points 1 week ago

The trouble with the matrix space is that when the site is down it's pretty much impossible to find the details for it.

[–] florge@feddit.uk 3 points 1 week ago

There's a matrix space here

[–] sh3llcmdr@feddit.uk 3 points 1 week ago

Thanks for fixing it

[–] Babalugats@feddit.uk 3 points 1 week ago

Great job. Although I think my eyes have been widened as to how big the fediverse actually is... I found all sorts of stuff that I still don't understand, communities I didn't know existed, tools that I don't know how to use and a lot of communities that have warnings i can't get past. Along with a whole heap of other stuff that I don't know how to get into.

[–] carlnewton@feddit.uk 2 points 1 week ago (1 children)

Do you have anything in place for detecting and mitigating DOS attacks? I wonder if you were to grep the logs by IP address you'd see the offending IP addresses. Though if you're using a load balancer or some other kind of proxy and don't have IP address forwarding configured, all requests might come through as the same IP address.

some data got duplicated in random tables that shouldn’t be

I'd also check that there's plenty of disk space spare, given that the log files are so large, as I've seen environments start to behave unusually once disk space has ran out, including SSH locking up.

[–] flamingos@feddit.uk 8 points 1 week ago

There's a setting in Lemmy that only let's logged in users make API requests. I'm going to activate that over night for now.

[–] jackal@feddit.uk 1 points 1 week ago (1 children)

Rotate your logs!!!

[–] buried_treasure@feddit.uk 3 points 1 week ago* (last edited 1 week ago)

Looks like they're already rotated daily.

Anyway by my understanding it wasn't the logs that caused the problem, they just demonstrate the extent of the sudden increase in requests by bots/scrapers.