28
submitted 2 years ago* (last edited 2 years ago) by Baku@aussie.zone to c/meta@aussie.zone

I know there's been a few posts about this before, but it's been a month since the last one and it's still ongoing. It doesn't seem that any of the LW admins responded to Zag's post on their help community, and the last response from lodion/Nath I'm aware of was from 4 months ago when there were outright federation failures as opposed to just lengthy delays.

@Nothing4You@programming.dev posted a comment on the post from last month about the delays stating that it's an issue on our end as our server isn't keeping up. I'm not sure whether this is the case or not, and I'm not sure how to interpret the Grafana dashboard they linked to, but as it's a new reply on an old post, I wanted to note it.

Current federation delays seem to be around 7 days. It doesn't seem to be affecting posts themselves on Lemmy.world communities, but does affect all replies to them (even from users on other instances), and all upvotes on the posts. [Edit: on further investigation, this isn't the case. The current delays are at least 13 days, and this does actually affect posts too]

I don't want to sound too pushy, since the LW admins and Lodion/Nath are all volunteers, but I was hoping we might be able to get an update on what the cause is, and if it's an issue in Lemmy itself, if anybody's opened an issue on GitHub and the developers are aware.

(NB: I don't interact that much with LW, so all of my testing has been on the Boost for Lemmy community.)

all 26 comments
sorted by: hot top new old
[-] ada@lemmy.blahaj.zone 9 points 2 years ago

The next milestone release of lemmy is scheduled to include parallel processing per instance for incoming activities, which should resolve the issue.

I don't know what the timeframe on that is though

[-] Nothing4You@programming.dev 7 points 2 years ago

https://github.com/LemmyNet/lemmy/pull/4623 is on the 0.19.5 milestone, until parallel sending is implemented there won't be any benefit from parallel receiving.

0.19.4 will already have some improved logic for backgrounding some parts of the receiving logic to speed that up a little, but that won't be enough to deal with this.

[-] Nothing4You@programming.dev 5 points 2 years ago

stating that it's an issue on our end as our server isn't keeping up

this isn't exactly an issue in your end, unless you consider hosting the server in Australia as your issue. the problem is the latency across the world and lemmy not sending multiple activities simultaneously. there is nothing LW can do about this. as unfortunate as it is, the "best" solution at the time would be moving the server to Europe.

there are still some options besides moving the server entirely though. if you can get the activities to lemmy without as many delays am experience similar to being hosted in Europe can be achieved.

[-] Zagorath@aussie.zone 5 points 2 years ago

the “best” solution at the time would be moving the server to Europe

I believe the NZ instance basically set up a proxy right next to the LW server, which catches the ActivityPub messages and forwards them in batches to another proxy located in NZ, which translates back into a Lemmy-legal synchronous format. Rather a convoluted process, but it seems to work:

[-] Nothing4You@programming.dev 9 points 2 years ago

yes, that's about the second best option for the time being.

it's currently used by reddthat.com and lemmy.nz.

disclaimer: i wrote that software.

[-] Zagorath@aussie.zone 5 points 2 years ago

disclaimer: i wrote that software.

Oh, ha! Sorry for the lemmysplaining!

[-] lodion@aussie.zone 5 points 2 years ago

Is that software available for others to use? I've been busy lately, but Nath messaged me about this... I was traveling and forgot about it by the time I returned.

I'm open to the idea of setting up an EU VPS to proxy LW traffic temporarily. But the best fix is of course for Lemmy to operate in a way that doesn't cause this.

[-] Nothing4You@programming.dev 4 points 2 years ago

it's open source: https://github.com/Nothing4You/activitypub-federation-queue-batcher

I strongly recommend fully understanding how it works, which failure scenarios there are and how to recover from them before deploying it in production though. not all of this is currently documented, a lot of it has just been in matrix discussions.

I also have a script to prefetch posts and comments from remote communities before they'd get through via federation, which would make them appear without votes at least, and slightly improve processing speed while they're coming in through regular federation. this also doesn't require any additional privileges or being in a position to intercept traffic. it is however also not enough to catch up and stay caught up.
this script is not open source currently. while it's fairly simple and straightforward, i just didn't bother cleaning it up for publishing, as it's currently still partially integrated in an unrelated tool.
I previously tried offering to deploy this on matrix but one of my attempts to open a conversation was rejected and the other one never got accepted.

[-] No1@aussie.zone 2 points 2 years ago* (last edited 2 years ago)

A side question...

What's gonna happen when this is finally fixed?

Will there be a flood of thousands of posts all at the same time/short period as we 'catch up'?

[-] Blaze@sopuli.xyz 2 points 2 years ago
this post was submitted on 21 May 2024
28 points (100.0% liked)

Meta

745 readers
1 users here now

Discussion about the aussie.zone instance itself

founded 3 years ago
MODERATORS