TL;DR: Lemmy upgrade some time early in the new year. I also recently enabled image proxying, let me know if you noticed.

I thought I'd make a quick post just to explain the plan for updating Lemmy.nz to the latest Lemmy version and another recent change to image proxying.

Lemmy upgrade

Some of you make have noticed that Lemmy 0.19.8 just came out, and yet we are still on Lemmy 0.19.5.

There have been various bugs in 0.19.6 and 0.19.7 that while not critical, there was also no reason to rush the upgrade and I decided to wait for 0.19.8 which had a fix for some thumbnails not working that was broken in I think 0.19.6.

My plan is to give a little time to see if others have issues, and then upgrade to 0.19.8. Most likely this will be early in the new year. I don't believe there should be any significant downtime so I will probably just do it when I am ready. Before I do, I'll do a test run and see how long it takes, then decide then.

As always, I'll post on the Lemmy.nz Matrix chat before I do it, and if I am having any issues.

Image proxying

You may have noticed I have enabled imaged proxying some 3 or 4 weeks back. This means all images are loaded from the Lemmy.nz server instead of whichever instance is hosting it. Personally I have found there to be significantly less instances of an image not loading. I have also noticed some images load a bit slower than before, which I think is if the image is not already cached then Lemmy.nz has to download it, then you have to download it from Lemmy.nz, so this tends to happen more on smaller communities.

However, overall I feel like the experience is improved with image proxying turned on (even if it has doubled CPU usage of the server).

As always, happy to answer any questions and hear your feedback 🙂

Xi wants to experiment with using xi/drag person-independent pronouns on this account. So far xi likes using the pronouns, but xi's only written 4 comments and already someone accused drag of talking about Chinese politics when xi mentioned dragself. Xi knows xi's pronouns are confusing, so xi would like to put xi's pronouns in xi's displayname to clear things up.

But the character limit for display names on lemmy.nz is only 22 characters, and xi's handle already takes up 16 characters. Xi would need 24 characters to put xi's pronouns in xi's display name. Xi knows some instances also have a separate field for pronouns, and the pronouns get displayed next to the name without taking up displayname characters. Xi would like a place, or at least enough room, for xi's pronouns, please.

I use bromine as my browser on Android and when I navigate to the lemmy.nz homepage in bromite the "All" button does not work. Works fine on lemmy.world, issue with latest update?

Welcome to the latest version of Lemmy!

Version 0.19.5 is a minor release with some bug fixes for 0.19.4. We were never running 0.19.4 so the bug fixes are less important than that new features in 0.19.4.

Release notes for 0.19.4 are here.

Release notes for 0.19.5 are here.

The upgrade took a bit longer than expected, perhaps 40 minutes of downtime. Partly this is because of an issue with postgres memory that happened before I started the actual upgrade.

Probably the two biggest new features for users are the ability to hide posts, and the ability to see any images you've uploaded (since lemmy started tracking).

There is also a new setting to change how upvotes are displayed, and we have a new option to have communities that don't federate. Maybe this community is a good target for that? Let me know your thoughts.

Let me know if you have any questions or see any strangeness!

Lemmy 0.19.5 has been released with some bug fixes, and we haven't upgraded to 0.19.4 yet, so I'm planning on doing the upgrade (to 0.19.5) this weekend.

Relase notes here: 0.19.4 / 0.19.5

No specific time, but in my test run it was less than 10 mins. This assumes that nothing goes wrong 🙂

I'll do it over the weekend when we normally have lower traffic.

As always, I'll post updates in the Matrix chat.

If anyone knows of any reason why we should hold off on the update, that would be good to know too!

Hi everyone,

Sorry, site seems to have been down from about 8pm last night through to about 7:30 this morning.

I spotted it last night and thought it was back up, but it seems some component wasn't so it was loading the pages but you couldn't do any actions.

It also means federation is behind because instances wouldn't have been able to contact us, and so that will be slowly catching back up. It may take a while because after a lot of downtime the instances will stop trying for a while.

Always happy to answer questions 🙂

A while back I made a post about Lemmy.world federation and a lot has changed since then so I thought I'd do an update post.

TL;DR Lemmy.world posts, comments, votes coming to lemmy.nz have been delayed by a gradually increasing amount over recent months, peaking at about 4 days behind, but we should be back on track soon.

Background

Check the above post for some background, but TL;DR our server is in Auckland, NZ and Lemmy.world's server is in Helsinki, Finland. We are about as far apart as we can get from each other. Because Lemmy can currently only send one action at a time (Post, comment, vote), we can only accept about 4 or 5 actions per second as it takes around 1/5 of a second to make the round trip. Lemmy.world is now creating more than this on average, which means we have been falling behind more and more.

Pre-fetcher

After that post, things continued to get worse. While hanging out on Lemmy Matrix rooms discussing the problem, someone (I'm not actually sure of their Lemmy account) offered to set up a pre-fetcher. Roughly how it works is it monitors lemmy.world for new posts and comments, then sends a request to lemmy.nz to see that content. Lemmy.nz then requests it from lemmy.world because it doesn't exist.

This helps because when lemmy.world sends lemmy.nz a post or comment, lemmy.nz then needs to make other requests. e.g. it might not know about the user, so it needs to request the user from their home instance, it may need to generate a thumbnail, etc. By pre-fetching the posts, this means when lemmy.world sends it's normal outbound federation lemmy.nz already has it so can move straight on to the next one. We can't prefetch everything, notably votes, so for a long time we have had lemmy.world posts show with zero votes until the federation activities start coming through. Unfortunately comments from lemmy.world users on lemmy.nz posts can't be pre-fetched so they were still taking a long time to come through.

Prior to this pre-fetcher being turned on, Lemmy.nz was doing a lot worse than aussie.zone (as seen in the above post). After turning it on, within not too long we were in better shape than aussie.zone, but unfortunately we were both still getting worse.

This is the state we have been in until yesterday. Gradually things were getting worse and worse until we were at about 4 days behind lemmy.world, so if a lemmy.world user posted on one of our posts then it took 4 days to show up (you might have noticed this if you got a notification of a reply to your post or comment that then said it was from days ago).

Batcher

The same user who created the pre-fetcher was also working on a batching process. The basic idea was that instead of lemmy.world sending each item halfway across the world, instead you add an extra server that is hosted close to lemmy.world. Lemmy.world sends their federation items to that server, then that server collects them up into one batch, which gets sent to some software running on the lemmy.nz server. That software then unbundles them into separate pieces again then feeds them into lemmy.

The idea here is that you greatly reduce the lag. Lemmy.world gets a very quick response from the extra server and so can send the next activity almost straight away. The software that passes it to lemmy.nz is on the same server as lemmy.nz so communication is very quick. And collecting up the items into a batch for the trip across the world saves a lot of time in back and forths, so we can keep our goal of receiving things in the correct order while also not having to send one at a time. Receiving in the correct order is important, for example, if you accedentally downvoted then quickly changed it to an upvote, you wouldn't want another instance to receive the upvote first and then the downvote as it would show you downvoted instead of upvoted.

This batcher I have set up (with a lot of help!) over the weekend, and turned on yesterday morning once testing had been completed and I could get lemmy.world to redirect their lemmy.nz traffic to this new server (this was done through a change in something called the hosts file, long story short it tells your server "ignore what anyone says, lemmy.nz is actually over here").

In the last 24 hours or so we have got from 1.5 million activities behind to about 970k activities behind. This puts us at about 2.3 days behind now, a huge improvement!

Pictures

Conclusion

So that's a breakdown of everything that has happened the last few months, hopefully this new batching process will bring us back in line with lemmy.world. If you see anything weird happening, please let me know!

Also this is a shout out to all the people who made this happen, and who are building all sorts of tools that we use. We have a selection of different front-end websites you can access lemmy through, an automod, prefetcher, batcher, and all sorts of help from others! I definitely couldn't do this stuff without help 😆

And as always, if you have any questions or want more detail on any of this, feel free to ask!

Edit:

We are now up to date! Yay!

TL;DR possible thumbnail and local image outage tonight

Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).

In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).

Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it's more than a month old.

Any questions, let me know!

Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won't work for votes which means you'll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.

Feel free to ask questions about that too!

Can we get some icons for c/support and c/wellington please?

This is triggering my OCD:

Thanks!

I though I'd better make a post about recent lemmy.world federation issues, since not everyone sees the daily posts.

TL;DR:

• Lemmy.world inbound federation is about 18 hours behind, and slowly catching up.
• Outbound is fine: If you post or comment on something in a lemmy.world community, it will be almost instantly federated to other users, but if they reply to you then it will take 18 hours before you see it.
• It was caused by an assumed Kbin bug in combination with a lemmy bug
• All other instances seem fine for inbound and outbound federation

On Thursday last week Lemmy.world began to receive hundreds of thousands of "activities" (actions - comment, post, upvote, etc) from Kbin to their communities. These actions then federate from Lemmy.world out to any instance with users subscribed to those communities.

Unfortunately, this also helped uncover a big issue with how Lemmy handles inbound federation. Actions are sent one at a time, and the next can't be sent until the last is done. This means the number of activities your server can receive depends on the latency between the sending and receiving server. In our case, the fact we are hosted in Auckland and Lemmy.world is hosted in Finland means that we can receive about 4 or 5 inbound activities from them per second.

When the Kbin issue happened, suddenly Lemmy.world had hundreds of thousands of activities to send to us in addition to the normal inflow of content. We got way behind in terms of Lemmy.world content coming to us. At the peak, we were around 28 hours behind, about 550k activities behind. We are now sitting at around 19 hours behind, and will likely remain that way until this afternoon, when it should improve some more (it's now peak time on the other side of the world).

Here is a graph of the progress. Aussie.zone has a similar problem to us, but is closer to Europe where lemmy.world is hosted so the latency issue is less pronounced. We are one of the worst affected instances due to how it's hard to get further away than where we are now.

As you can see, with Aussie.zone being a bit closer, they were hit hard but not as hard as us, and have now managed to recover. We are still probably days away from being back in near real time federation with lemmy.world, but each day does seem to improve quite a bit.

Other than picking up the server and moving it to Europe, there doesn't seem to be a lot we can do except wait and hope the lemmy bug is fixed before lemmy.world grows so large that users are creating content faster than we can receive it.

There have been some suggestions for short term band-aids. One is that we could set up a proxy in Europe, and filter out certain traffic. For example, we could prevent downvotes being sent to reduce the amount of traffic coming our way (since everything is sent one at a time, as per the ActivityPub protocol). I'm not super keen on doing anything that gets us out of sync with the others, though.

We could of course move hosting to Europe, but that gets a lot more expensive (since we'd have to pay for it instead of being free-loaders) and also means the site would feel more sluggish, since then that latency would be for users on every page load instead of the behind the scenes federation.

For now we are slowly getting back in sync with lemmy.world, and as long as there aren't more Kbin bugs then we should be ok for now. Unfortunately doing things like blocking Kbin.social doesn't seem to help, as lemmy.world still tries to send us the content from Kbin.social users. i.e. it's rejected by our server, not prevented from being sent, so it still takes a slot.

Any questions, suggestions, or concerns, reply to this post!

Some comments and votes from my posts on lemmy.world/c/news aren't showing up.

This what appears from my account

And this from lemmy.world

Here's the results of what I've called the Community Criteria Survey.

Note that you did not have to answer any questions, so there are different numbers of responses for different questions.

First up, the demographic questions. I'm keen to do a fuller census survey like Lemmy.ca does, and will probably do this annually around the anniversary of the site in June.

Which instance(s) do you have an account on that you use to interact with Lemmy.nz communities?

Note this was multiselect, so people could choose multiple.

Where do you live?

How old are you?

It seems the majority of our users are aged 30-50.

Next up are the questions about communities.

Do you think communities on Lemmy.nz should be focused on people who live in NZ, or should we add communities with a more general audience? For example, !politics@lemmy.nz discusses NZ politics, with a relatively narrow audience. We could add communities with a more general audience, such as !rocketlab@lemmy.nz, which could have interested users from all over the world, and discussion that is not NZ-centric. Which of the following best describes your feelings?

Mixed responses here, it seems we will be doing surveys before we add communities as the biggest share thought a case by case basis was appropriate, and the remainder were split on whether we should only have NZ targeted communities or are happy with more general communities.

Specifically looking at the case of a Rocket Lab community, and considering all factors such as: Rocket Lab was founded in NZ, and launches from Māhia Peninsula in NZ Rocket Lab is a corporate entity A Rocket Lab community may not have a lot of activity outside of launches A Rocket Lab community would have a general audience, being the first Lemmy.nz community not specifically focused on people living in NZ or with close ties to NZ Posts on communities on Lemmy.nz will show in the "Local" feed Would you support the creation of a Rocket Lab community !rocketlab@lemmy.nz?

I wasn't sure of the best community for this, but as this one sort of acts as our meta community I thought this was the best place for it.

I am trying to find this thread from kbin's mma community but it doesnt show up. https://kbin.social/m/mma/t/886629/Official-UFC-299-Sean-O-Malley-vs-Marlon-Vera-Live-Discussion

https://lemmy.nz/c/mma@kbin.social?dataType=Post&sort=New

Hi all, there has been previous discussion on adding a Rocket Lab community to Lemmy.nz. Some people have supported this and been excited about the possibility, some have been opposed and thought it would fit better on a more general instance.

This is a survey about this specific question, but it also includes some questions about yourself and your opinions on the kind of communities we want.

None of the questions are mandatory, so just answer what you feel comfortable answering.

It's my first try at making a survey, so give me all your feedback (either in the survey comments or as a reply here).

Anyone is welcome to answer the survey, but I will be taking the opinions of Lemmy.nz users into higher account since it's those users that will actually see a difference in their browsing experience if we add this community (since the posts will go in the Local feed).

You can find the survey here: https://survey.lemmy.nz/index.php/561246?lang=en

TL;DR: Downtime 2am tonight NZDT of around 2 hours.

Tonight we will have scheduled downtime while the host updates the hard drives on the server.

If you like technical details, the new hard drive array has an NVMe cache + full ZFS, compared to the current one which is an NFS share. This should cut out the networking layer and make hard drive access faster.

In terms of real world speed of the website itself, we may not notice much difference. Lemmy is heavy on the CPU particularly due to database access, but the database is largely held in RAM so we may not see a lot of difference in site speed.

It's likely to take around 2 hours, based on the host's migration of other services they host.

The host is https://fediservices.nz, which runs mastodon.nz and some other federated services. They don't run Lemmy.nz, but they donate hosting and so Lemmy runs on the same server as their services (so if you want to donate, that's who you donate to).

If you have any questions, ask away!

Web browser seems to work fine with the same login, and was previously working in sync.

Welcome to the latest version of Lemmy!

• If you're having trouble logging in, clear your cookies.
• If the bottom of the page tells you "BE: 0.18.5" then you need to shift-refresh your page (or clear browser cache).
• If your app doesn't work anymore, try to sign out and sign back in.
• If it really doesn't work, it's probably because Lemmy 0.19 introduces many big changes with how things work. Apps still being maintained by their developer should work. Apps that have not been updated in the last couple of months most likely will not work.
• If you're using an alternate web frontent (a.lemmy.nz, t.lemmy.nz, p.lemmy.nz, voyager.lemmy.nz) then you will most likely have to clear your cache, or at the very least log out and back in.

There's a post about this release by the developers here.

Also see the blog post about the original 0.19.0 release, which is where the major changes are.

Some key things:

• There's a new scaled sort that helps posts from small communities be seen in your feed
• Users can now block entire instances
• Two factor authentication should no longer lock you out of your account (no recovery codes, though?)
• You can now export your user data (community follows, blocklists, profile settings) and import into another server as a way of migrating accounts.
• "active user" stats shown on the home page will now count users that vote as active. Previously a user needed to post or comment.

Let me know if you see any issues!

TLDR:

• All going well, there will be an outage of approx 30 mins starting at approx 8:30pm tonight
• If things don’t go well, there may be a further outage
• Lemmy 0.19 has breaking changes, the key impact of this is that old lemmy apps that haven’t been updated in the last month or two are likely to break as they won’t have had the lemmy changes incorporated.
• Lemmy caches the website - after the update you will most likely have to clear your browser cache to get the new version, otherwise it won’t be able to communicate with the server. The easiest way is to click the browser refresh button while holding SHIFT. On mobile you may be able to do the same by long-pressing the refresh button in the menu.

Today's downtime

Just to start, I want to acknowledge we have had some downtime today. The guys that donate hosting to us (the fediservices.nz guys) had downtime today, so all their services have been intermittently down as well as us. It should now be solved, sorry about that. Hopefully no one urgently needed to browse lemmy.

Lemmy 0.19.2

After cancelling the planned update to lemmy 0.19.1 on learning of some serious federation issues other instances have had, a new version was released that has resolved these issues for those instances. Therefore, I am planning to upgrade this evening. See the previous post for more info.

Now is the time for objections, so speak up if you have any objections to us updating.

The starting time should be around 8:30pm tonight NZDT (when this post is 2.5 hours old), but I may start a bit earlier if I get kids to sleep earlier. Or later if kids are asleep later. I'll post progress in the lemmy matrix chatroom.

I'm hoping for less than 30 mins of downtime, but this of course depends on everything going smoothly.

Some bigger instances like lemmy.ml, lemm.ee, lemmy.ca, programming.dev, lemmy.dbzer0.com, aussie.zone, etc have now updated to 0.19, and after an issue was solved leading to 0.19.1, there don't seem to be further issues.

I have done a trial run at updating Lemmy.nz to 0.19.1, and it went well. Therefore I am planning on updating Lemmy.nz to 0.19.1 starting around 7pm NZDT tonight (when this post is 5 hours old).

The release notes can be read here. There are lots of mostly little changes. I'll do more of a summary after the update is completed.

Things to know:

• All going well, there will be an outage of approx 30 mins starting at 7pm tonight
• If things don't go well, there may be a further outage
• Lemmy 0.19 has breaking changes, the key impact of this is that old lemmy apps that haven't been updated in the last month or two are likely to break as they won't have had the lemmy changes incorporated.
• Lemmy caches the website - after the update you will most likely have to clear your browser cache to get the new version, otherwise it won't be able to communicate with the server. The easiest way is to click the browser refresh button while holding SHIFT. On mobile you may be able to do the same by long-pressing the refresh button in the menu.

Any objections, now is the time to speak up!

Any questions, feel free to ask!

Update: It seems there are still serious federation issues! The original github issue has been reopened and some sites are having many issues. So let's hang tight a bit longer.

I tried posting on /c/world, but the posts didn't show on /c/world. I checked modlog, I didn't find if I was banned. I also checked inbox and I couldn't find any messages if I was banned.

And I've tried posting on lemmy.world support but the post wasn't there

Any explanation for this?

I saw a (-2 new) in the comment count, someone may have deleted comments.

Is this a bug?

To find and subscribe to new communities is still quite annoying, or is it just me? Trying to find this community from Lemmy.nz but can't find it. https://lemmy.world/c/parenting

When I construct this URL myself I get an error https://lemmy.nz/c/parenting@lemmy.world

It doesn't appear in the search, only one from Lemmy.ml

I'm not sure if this is something going wrong, or just how Mastodon works.

Anywho, when I go to Communities - Subscribed and click through to this one: https://lemmy.nz/c/rugby@sh.itjust.works

There are posts that I think are there that don't show up in that view, eg: https://lemmy.nz/post/1077020

The only reason I know that post exists is because it does show up if I just go to https://lemmy.nz/ so i'm guessing it shows in the Subscribed Posts section, but if it isn't there then i'd be missing the post entirely I guess.