Anyone got big plans for the long weekend? Apparently schools across NZ are closed on Tuesday, a bit of a last minute surprise for poor planners like us!

We have family visiting, and I also hope to do an upgrade of the Lemmy image caching system "pictrs". This will involve migrating the database into a postgres setup. It took 12 hours in my test run, and I expect it to perhaps take half that on the much beefier production server. In any case, if it's not available it should only affect images, but in my test run the site remained perfectly functional somehow.

We also have a bit of a pre-fetching setup running that is pulling posts and comments from lemmy.world before they are sent to us. We hope this will help us get back in sync with lemmy.world, but at the very least it should help us get comments and posts through a bit quicker. We can't pre-grab the votes so you may see a bunch of lemmy.world posts in New that have no votes.

We are currently tracking about 6 hours behind lemmy.world, but in theory people replying to you should now come through a lot quicker.

[–] NoRamyunForYou 3 points 8 months ago (1 children)

Got a whole heap of small projects that I have in my head that I want to get done, but nothing's being put into paper so Im beginning to have some doubts as to what I'll get done lol.

Ahh that Lemmy.world delay thing to explains why I haven't been seeing some alerts for replies etc. Still don't fully understand the whole "hours behind" thing and how it all works.

[–] Dave 5 points 8 months ago (2 children)

It can be a little tricky to understand exactly how the issue presents itself, because different parts move at different speeds. I'll try to explain a little more. I hope you were planning on reading a novel today because there's one coming 😆

Every time someone take an action (comment, create post, vote on post or comment) a little piece of information is created and added to a queue, this is called an "activity".

This activity is sent to the instance of the community it happened under. For example, if you vote on a post in a community memes@lemmy.world then lemmy.nz sends this information to lemmy.world, then the lemmy.world server sends that vote to every instance that has at least one person subscribed to that community.

Recently, Kbin.social has had issues where it has sent tens of thousands of activities to lemmy.world in a very short amount of time (specifically to the games@lemmy.world community). This is presumed to be a bug as it seems to be the same thing sent lots and lots. Now Lemmy.world tries to send these tens of thousands of activities to every instance that has at least one person subscribed to !games@lemmy.world.

As a result of this happening, a bug in lemmy was found. Lemmy.world sends the activity to lemmy.nz (or any other instance), then once that is complete it sends the next one. In theory it should just fire them all off in parallel, but it doesn't because if the order is wrong then it might end up in an inconsistent state (say, if someone edited a post twice, you don't want to get the second edit first then the first edit second).

Because they are sent one at a time, there is a limit to how many it can send. This limit is determined by the time it takes for lemmy.world to send something to lemmy.nz and then receive a response that it has been received. The round trip takes about 250ms, or 1/4 of a second. This limits us to receiving only about 4 activities per second. We can't improve on this speed, because the latency is largely because we are hosted in NZ and they are hosted in Finland on the other side of the world. Aussie.zone also has issues, but can handle slightly more than us as they are slightly closer.

Now lemmy.world is creating activities at around 4 per second (posts, comments, votes, and a few others put together), or just under, during week days. That means we are barely taking on the activities as fast as they are being created. Now suddenly Kbin sends a couple of hundred thousand activities within an hour and suddenly lemmy.world is sitting there trying to send them, but can only send 4 per second, so it takes a long time to get through them. Then add on that lemmy.world is generating new content at almost 4 per second and now there's really no chance to catch up on that backlog. Luckily the weekend comes, and that backlog comes down. Over three weekends we are down to only about 40,000 actions behind.

Then a couple of days ago lemmy.world got another spike of activities from Kbin and here we go again!

Now the "hours behind" thing. Someone posts to a lemmy.world community, and they get added to the back of the queue at say 4pm. It slowly makes it's way to the front of the queue as other activities are sent, and when it reaches the front it gets sent to Lemmy.nz and it's now 10pm. It took 4 hours for that post to be sent to lemmy.nz after it was posted to lemmy.world, so that's a 6 hour lag, 6 hours behind.

To make things more confusing, federation is not just one way, but when you post/comment/vote to a lemmy.world community the lemmy.nz server needs to send that activity to lemmy.world. Our queue going to lemmy.world is fine, sitting empty because kbin never sent us 100k activities, so it gets sent to lemmy.world straight away. So you have a one way lag - if someone posts to lemmy.world it takes 6 hours before it gets sent to us, but if we post to lemmy.world it is sent straight away. That makes conversations a bit one sided!

Now I'm going to add in a further point of confusion! Yesterday, another instance admin offered to turn on something they have been setting up to try to band-aid the problem. This is a server hosted in Europe (for quick federation) that gets the lemmy.world posts and comments, and sends a request to lemmy.nz for that content. When lemmy.world sends lemmy.nz details of a post, lemmy.nz has to do some things like sending a request back to lemmy.world to get some info about the post, generate a thumbnail, etc. Sometimes this can take seconds to do, so this slows down the process (is the theory). By asking lemmy.nz to grab posts before lemmy.world tries to send them, the theory is that any lag caused by lemmy.nz fetching this info will be mitigated. We aren't able to grab votes, but the hope is that it will be slightly faster getting through the backlog. Because lemmy.nz and aussie.zone are similar in their queue trends, were using them as a control to see if it helps (so far no obvious big change but we are slowly making ground on them so perhaps that's because of this).

At the very least, with this pre-fetching of posts and comments, if someone replies to you on a lemmy.world community it should now come through almost immediately, though it will take some time to see the votes which are still delayed.

And just to clarify, it's all based around communities. If a lemmy.world user votes on a post in a community on beehaw.org, it should come through straight away, as beehaw.org is responsible for sending the vote. If a beehaw.org user votes on something on a community on lemmy.world, it will be delayed, because lemmy.world is responsible for sending the activity.

I'm really hoping the easter weekend will lead to a drop in lemmy.world activity that will let us catch up, though we are still only another kbin bug from being behind again, at least until a lemmy software update is done to resolve the issue.

Anyway, hope that made things clear 😑 (though I suspect it made you more confused 😆)

[–] AWOL_muppet 5 points 8 months ago (1 children)

Solid description, you're the greatest, Dave!

[–] Dave 5 points 8 months ago

I know 😁

[–] Dave 4 points 8 months ago* (last edited 8 months ago) (1 children)

Here's a graph of the number of "activities" we are behind lemmy.world, and aussie.zone as well. Last time we had the issue (a couple of weeks back) aussie.zone recovered much faster than us. Now we seem to be gaining on them, 30k worse than them down to 10k worse than them over the last 24 hours. So perhaps the prefetching is working!

The up and down is just normal variance over the day, it improves when the other side of the world are asleep, and gets worse when they all wake up.

graph showing number of activities behind lemmy.world for both lemmy.nz and aussie.zone, with our higher backlog slowly narrowing in on aussie.zone's

[–] NoRamyunForYou 4 points 8 months ago (1 children)

WOW! Thank you for all that knowledge :) The relationships between the different instances, and how they work together suddenly makes sense - seems pretty simple when you break it down like that lol.

Do you think there could ever be a "perfect" solution to this problem? Or due to how this whole Federation thing works with all these different instances, is this just a limitation that can only get better to a point?

[–] Dave 3 points 8 months ago (1 children)

It's fixable. Just that it wasn't known to be a problem until recently.

Here's the issue raised for it, where there is a bunch of conversation about ways to fix it.

The main issue it keeping the order right, so you can probably fix that by simply sending the date and time along with it and appropriately handling it on the receiving side. This has it's own pros and cons that I think were discussed in that issue, but all in all, rest assured that top people are looking at it 🙂

Though it seems major enough that I think it will be a while before we get a fix released, and will have to limp along in the meantime.

[–] NoRamyunForYou 3 points 8 months ago (1 children)

Oh that's good to hear :)

Just skimmed through the issue, and there seems to be a lot of interesting discussion around it.

Pretty cool, really feels like we're at the infancy of something, hopefully growing into something great.

[–] Dave 3 points 8 months ago (1 children)

For sure! The great thing with federation is you can like the idea but not the implementation, and decide to do it better but get to interact with all the users still.

So we end up with a bunch of similar software that can interact: lemmy, kbin, mbin, sublinks, piefed. No single point of failure, so even if lemmy development was completely abandoned we could just switch software to something else and continue.

[–] NoRamyunForYou 2 points 8 months ago (1 children)

Oh okay - I didn't know about the interaction with different software. Had heard of Kbin before (haven't yet heard of any of the others), and just assumed it was a completely different type of thing like Mastodon.

That's pretty cool. How does the interaction with (for example Kbin) work? Can we see their communities, and interact with their posts and vice-versa?

[–] Dave 2 points 8 months ago* (last edited 8 months ago) (2 children)

Interesting you mention Mastodon, that uses the same protocol as Lemmy. So people on Mastodon can actually participate in Lemmy discussions (though things look and work a little different, you can often spot them because their replies to comments start with @user because that's how it works on Mastodon). Currently Lemmy users can't interact with Mastodon, but this isn't because it's not possible, just not implemented in Lemmy (yet).

Kbin has both micro-blog style (like Mastodon/Twitter) and thread style (like Lemmy/Reddit), and largely Kbin keeps them separate so they are more natural than a Mastodon user interacting with Lemmy.

But yes, we can see the Kbin communities (they call them Magazines), and they can interact with us pretty seamlessly (you know, when bugs in Kbin aren't causing lemmy to DDOS itself). Here's the Kbin.social meta community: https://lemmy.nz/c/kbinMeta@kbin.social

[–] NoRamyunForYou 2 points 8 months ago (1 children)

Oh haha! When I was writing down Mastodon, I had a feeling you were going to say they were compatible with eachother haha.

That's interesting how they can interact with eacother etc. Have you tried Kbin? if so, hows that experience?

[–] Dave 2 points 8 months ago (1 children)

I've only briefly tried Kbin. I'm not really into microblogging so it doesn't interest me too much. Many Kbin users are opposed to lemmy because the lemmy devs can be dicks and have some controversial political views, but largely I think the community is similar to Lemmy as it's also largely reddit refugees.

[–] NoRamyunForYou 3 points 8 months ago (2 children)

I understand that Lemmy is Open Source, but are those "dick" devs something to be concerned about?

[–] Dave 2 points 8 months ago (1 children)

Some people think yes, some think no. Personally I don't think so.

The devs have ultimate control, but there are many other people contributing. If people really don't like the direction, they can fork it and build the stuff we want.

Sublinks is an example of this, where a group of people are building a Lemmy a compatible platform that focuses on things the devs don't seem interested in such as moderation tools. Kbin has an equivalent Mbin, as the Kbin Dev seems unable to keep up and unwilling to delegate.

But ultimately, open source means we don't have to worry about this stuff until it actually becomes a problem. Also, sometimes you come across as a dick because running an open source is demanding, unforgiving, and you regularly face people who feel they are entitled to have you do they thing they want you to do. So it's hard to judge, really.

[–] NoRamyunForYou 2 points 8 months ago* (last edited 8 months ago) (1 children)

If a group were to go and fork it, and start developing independent from the original devs, they would still be able to interact with Lemmy right? (unless they can block eachtother?), but if any existing communities wanted to move over, they would basically have to start again from scratch because the fork isn't Lemmy anymore

edit: or is every server/instance a fork?

[–] Dave 2 points 8 months ago (1 children)

So when Facebook makes a site, they control all the code. If someone wants to make a messaging app that let's you talk with Facebook users, they have to get hacky because Facebook controls all the code.

Now think about email. Email is a protocol that is open, no one can stop you using it, and the code is not a secret. As a result, Microsoft may build mail servers (like Exchange, commonly used in businesses) and mail providers (like hotmail) and mail applications (like Outlook), but if someone wants to set up competing email (like Gmail), they can. What's more, it doesn't even matter that the code behind Hotmail and the code behind Gmail are secret, they can talk to each other using the common email language and what happens in their individual servers doesn't matter.

Lemmy is kind of like that, and in fact email is one of the better analogies to use when people don't understand federation because email is federated. You can email from whatever server or set up your own and no one worries they can't talk to each other.

You can have Lemmy and Kbin talk to each other even though they run completely different code, all because there's a language that they can speak to transfer data between them. Kbin or Lemmy could still do this even if not open source, for an example, Facebook's new Twitter rival Threads has recently started federating in a limited capacity with Mastodon, even though the Threads source code is proprietary.

I think I went on a bit of a tangent there, but let's get back to your question about forks.

When lemmy releases versions, they number them. But open source code gets tricky, because anyone can copy the code at any point and make their own changes. Some people will apply small changes to the code but keep the rest the same including updates, some will copy the code then develop independant from there, and many inbetween grey areas. Basically once you make the code open source you have no control over versions.

Most instances run the lemmy code directly from the lemmy devs, including lemmy.nz, and so this is not a fork. However, some instances will apply their changes, e.g. the lemmynsfw instance has small code changes they apply to support their aims (for example, age check prompt). Is this a fork? I don't know if there's an official answer to this but I would call this a "patch", a small bit of extra code they apply on top of the lemmy code, rather than copying and then developing separately.

I'm not aware of any lemmy alternatives that I would consider a fork of lemmy, but also understand that under "git", a source control system that gives its name to github, if you want to volunteer code to lemmy the traditional way is to "fork" the code, change the code to build your new feature, then do a "pull request" - basically asking the devs to pull your code into the lemmy code. While the term "fork" is used here, generally when people are talking about a project being forked they don't mean forking it to contribute, though the process starts much the same.

The closest lemmy relative is probably sublinks, which aims to be lemmy API comparable (the API is how the website you view, or the app you use, talks to the lemmy server - in lemmy the website part is completely separate from the server part and apps don't touch the website at all).

However, Sublinks is rewriting the lemmy server in a different programming language, so isn't really a fork as they didn't copy the code. Their aim though is that they keep the API the same as lemmy, so that you can use lemmy apps with it (and in fact currently use the lemmy website to access the Sublinks server).

As you can probably tell, with open source stuff it gets a bit blurry because anything is possible, it doesn't have to be contribute to the original or copy the code and do your own thing, all sorts of inbetweens are possible.

Hopefully I answered your question in this ramble because honestly I'm not sure that I did 😆

[–] NoRamyunForYou 2 points 8 months ago (1 children)

Thank you for all that detail!

That's really interesting around the blurriness with Open Source Projects. I feel a heck of a lot more educated around this federation stuff now 😂

Thank you for answering all my questions over the last couple of days - I'll let you enjoy your long weekend now lol.

[–] Dave 2 points 8 months ago (1 children)

Haha if it wasn't obvious by the verbal diarrhea, I'm always happy to talk about this stuff!

[–] NoRamyunForYou 2 points 7 months ago (1 children)

I'll keep that in mind it I have any questions in the future.

I really enjoy hearing about these types of things

[–] Dave 2 points 7 months ago (1 children)

If you have questions, feel free to ask 🙂

[–] NoRamyunForYou 2 points 7 months ago

Thanks for that :)

[–] thevoyagekayaking 1 points 8 months ago

They mostly fit into the "insufferable Internet communist" category, so nothing to be too worried about.

[–] CommunityLinkFixer@lemmings.world 0 points 8 months ago (1 children)

Hi there! Looks like you linked to a Lemmy community using a URL instead of its name, which doesn't work well for people on different instances. Try fixing it like this: !kbin@lemmy.nz

[–] Dave 1 points 8 months ago

Lol I'm not sure what this bot is referring to 😆

[–] Dave 1 points 8 months ago

Also if you haven't voted for our Lemmyvision canditate, please do! It's a three way tie currently and I don't want to have to do a tie breaker!

https://lemmy.nz/post/8465782