Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

you are viewing a single comment's thread
view the rest of the comments

[–] Tag365@lemmy.world 18 points 1 year ago (5 children)

I had a strange bug today where I wasn't able to upvote comments. So I cleared out my website data like the website suggested and I started having problems logging in. It would log in but then when I refreshed it wasn't logged in anymore. It stopped after a while but then when I clicked on an old tab when I refreshed I was logged out again. So, the log in issue must be something to do with how iOS Safari handles web cache on tabs.

[–] normalmighty@lemmy.world 8 points 1 year ago* (last edited 1 year ago)

Upvotes aren't responding for me, but the comment is marked as upvoted when I refresh the page. I think it just isn't rerendering on the frontend.

Edit: same thing is happening when posting comments. I refresh and the comment was posted, but without refreshing it's marked as loading forever.

[–] feitingen@lemmy.world 5 points 1 year ago

I had the same issue with Firefox on Linux. I had to clear website data again to fix it.

I had the F12-mode open to see what was going on with cache disabled, so that might have helped as well.

[–] Patrizsche@lemmy.world 5 points 1 year ago

Exactly same thing for me (but on Chrome on Windows), I emptied my cache and I just can't log in (says it worked, but it really doesn't)

[–] DarkWasp@lemmy.world 3 points 1 year ago* (last edited 1 year ago)

I’m having major issues with upvoting in safari on iPad. Seems like it will take either 10 seconds or so and work or not react at all.

Edit: Wiping all safari browsing history/clearing everything fixed it for 20 mins or so and it’s back to behaving the same way again.

[–] Tag365@lemmy.world 2 points 1 year ago (1 children)

I suddenly got "logged out" again, but when I opened a new tab and went to Lemmy.world I was still logged in.

[–] tpfm@moth.social 1 points 1 year ago (1 children)

@Tag365 @Tag365 You prolly ended up in another instance (lemmy.x)

[–] Tag365@lemmy.world 1 points 1 year ago

Wrong, it was the same instance, merely the tab was unloaded and then reloaded.