Federation issues today! It turns out Lemmy handles each inbound federation request one at a time, finishing the first before accepting another, and at a certain point the inflow of items is too much and they start backing up.

Details in this new bug report: https://github.com/LemmyNet/lemmy/issues/4529

Our major issue is with Lemmy.world, and we aren't the only one having issues.

If Lemmy is feeling light on content today, this may be why.

Outgoing federation is fine, so if you post something it should go through fine. But if someone replies, currently there is about a 12 hour lag for content on Lemmy.world communities.

[–] flambonkscious@sh.itjust.works 2 points 8 months ago (1 children)

Wow, that's crazy

[–] Dave 2 points 8 months ago (1 children)

Unfortunately ActivityPub doesn't seem to support batching, so each post, reply, or vote has to be sent one at a time to every instance the community is federated with. Plus order is important for some items (e.g. if you edit a post multiple times, you need to make sure they are processed in the right order or you get out of sync).

Some smart people are looking at solutions, hopefully it's solved before everything collapses 😆.

It's unfortunately dependant on ping. When we receive an incoming federation activity, Lemmy gets info about the new thing but often then needs to send an outbound request to fetch the rest of the details. It's not until that request is complete that Lemmy can accept the next update. Until then, lemmy.world is just sitting there retrying to send the next one as Lemmy.nz won't accept it until it's done with the current one (no queue).

The further we are from Lemmy.world (distance between servers), the higher the ping and therefore the longer each post takes to process. Because we are hosted in NZ and they are hosted on the other side of the world (Europe I think?), we are disproportionately affected.

[–] flambonkscious@sh.itjust.works 3 points 8 months ago

Yeah it's quirky alright! Filed under growing pains - I'm glad my stuff isn't this complicated and believe in the brains trust (glad I didn't selfhost!)

[–] Dave 2 points 8 months ago

An update, we are now about 14 hours behind Lemmy.world. However, Lemmy.world identified "78k community likes + same amount of undos for one kbin.social user, as well as almost 32k from another kbin.social user."

They have banned the accounts, which should make Lemmy.world reject and not pass on that activity, which should mean a slow recovery back to not lagging anymore.

Currently it's still getting worse (now over 500k actions behind Lemmy.world) but I will monitor and hopefully see it improve.