Federation issues today! It turns out Lemmy handles each inbound federation request one at a time, finishing the first before accepting another, and at a certain point the inflow of items is too much and they start backing up.

Details in this new bug report: https://github.com/LemmyNet/lemmy/issues/4529

Our major issue is with Lemmy.world, and we aren't the only one having issues.

If Lemmy is feeling light on content today, this may be why.

Outgoing federation is fine, so if you post something it should go through fine. But if someone replies, currently there is about a 12 hour lag for content on Lemmy.world communities.

[–] flambonkscious@sh.itjust.works 2 points 8 months ago (1 children)

Wow, that's crazy

[–] Dave 2 points 8 months ago (1 children)

Unfortunately ActivityPub doesn't seem to support batching, so each post, reply, or vote has to be sent one at a time to every instance the community is federated with. Plus order is important for some items (e.g. if you edit a post multiple times, you need to make sure they are processed in the right order or you get out of sync).

Some smart people are looking at solutions, hopefully it's solved before everything collapses 😆.

It's unfortunately dependant on ping. When we receive an incoming federation activity, Lemmy gets info about the new thing but often then needs to send an outbound request to fetch the rest of the details. It's not until that request is complete that Lemmy can accept the next update. Until then, lemmy.world is just sitting there retrying to send the next one as Lemmy.nz won't accept it until it's done with the current one (no queue).

The further we are from Lemmy.world (distance between servers), the higher the ping and therefore the longer each post takes to process. Because we are hosted in NZ and they are hosted on the other side of the world (Europe I think?), we are disproportionately affected.

[–] flambonkscious@sh.itjust.works 3 points 8 months ago

Yeah it's quirky alright! Filed under growing pains - I'm glad my stuff isn't this complicated and believe in the brains trust (glad I didn't selfhost!)

[–] Dave 2 points 8 months ago

An update, we are now about 14 hours behind Lemmy.world. However, Lemmy.world identified "78k community likes + same amount of undos for one kbin.social user, as well as almost 32k from another kbin.social user."

They have banned the accounts, which should make Lemmy.world reject and not pass on that activity, which should mean a slow recovery back to not lagging anymore.

Currently it's still getting worse (now over 500k actions behind Lemmy.world) but I will monitor and hopefully see it improve.

[–] Dave 2 points 8 months ago* (last edited 8 months ago) (1 children)

I hope everyone has been enjoying a day of non-lemmy.world content 😆

We are very slowly catching up. At the current rate I'm honestly not sure we'll catch up before tomorrow, especially with Europe waking up soon.

Here's a nice graph of how far behind we are:

graph showing activities (actions) from lemmy.world to lemmy.nz being backed up, peaking at 550k behind and then starting to slowly drop back to 538k

We don't want that line to come above the bottom, being 550k actions behind is a pretty big issue. But Lemmy.world has had their federation backed up since kbin starting flinging tens of thousands of actions from three user accounts. It's unclear the cause, the user accounts seem pretty normal, but I've temporarily banned them for 24 hours in the hope that will have our server reject the actions and help us catch up. Lemmy.world have also done a lot on their end, trying to get it sorted!

[–] liv 2 points 8 months ago

Far out!