this post was submitted on 04 Jan 2024

427 points (99.1% liked)

Meta (lemm.ee)

3590 readers

14 users here now

lemm.ee Meta

This is a community for discussion about this particular Lemmy instance.

News and updates about lemm.ee will be posted here, so if that's something that interests you, make sure to subscribe!

Rules:

Support requests belong in !support
Only posts about topics directly related to lemm.ee are allowed
If you don't have anything constructive to add, then do not post/comment here. Low effort memes, trolling, etc is not allowed.
If you are from another instance, you may participate in discussions, but remain respectful. Realize that your comments will inevitably be associated with your instance by many lemm.ee users.

If you're a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K

Discord is only a back-up channel, !meta@lemm.ee will always be the main place for lemm.ee communications.

If you need help with anything, please post in !support instead.

founded 1 year ago

MODERATORS

sunaurus@lemm.ee

427

When this post is 48 hours old, lemm.ee will be going down for a database upgrade (Edit: upgrade complete!) (lemm.ee)

submitted 10 months ago* (last edited 10 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee

75 comments fedilink hide all child comments

Hey folks

This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.

We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.

I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.

As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!

Edit: upgrade complete!

I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.

As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner's Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I'm really happy with the solution now.

In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.

top 50 comments

sorted by: hot top controversial new old

[–] centof@lemm.ee 58 points 10 months ago* (last edited 10 months ago) (1 children)

In other words, lemm.ee will be down starting at UTC 2024-01-06T10:00 for 2+ hours. Convert to your local time via https://www.utctime.net/utc-time-zone-converter if you want.

Thank you sunaurus, for the heads up and for the ongoing maintenance you do!

[–] joyjoy@lemm.ee 15 points 10 months ago

I prefer using timeanddate.com since it has a fixed time URL.

[–] Nelots@lemm.ee 47 points 10 months ago (2 children)

So I wasn't going crazy, the long loads times were real. Glad to see upgrades coming soon!

[–] JimmyBigSausage@lemm.ee 20 points 10 months ago

Although the long page load times have been real, you still might be going crazy. Definitely a possibility on Lemmy (づ￣ ³￣)づ

[–] kratoz29@lemm.ee 6 points 10 months ago

My experience has been downgraded since the latest Lemmy update, all these caveats/workarounds are pretty much welcome in order to get this in a stable state again!

[–] negativenull@lemm.ee 24 points 10 months ago (1 children)

Hell yeah!!!

[–] shootwhatsmyname@lemm.ee 3 points 10 months ago (1 children)

That’s awesome! Are you monitoring personally or is this from somewhere public?

[–] negativenull@lemm.ee 6 points 10 months ago (1 children)

I run a bunch of small websites/tools. I setup UptimeKuma a while back to monitor my own apps. Since it's just a simple web monitor, I can monitor any http/s URLs. I setup a lemm.ee monitor a few days ago when things were starting to get unstable.

load more comments (1 replies)

[–] PanArab@lemm.ee 19 points 10 months ago

Thank you for all the work you do

[–] GrayBackgroundMusic@lemm.ee 17 points 10 months ago

No worries about the downtime. If that's what's needed to get back to 0.18 levels of performance, go for it! Thanks for all the work y'all do.

[–] Lemonparty@lemm.ee 14 points 10 months ago

Np, thanks so much for all you do!

[–] argo_yamato@lemm.ee 13 points 10 months ago

Thank you for your work on lemm.ee! I was actually looking to post somewhere to see if there was some slowness or if it was just me, looks like you answered my question.

[–] wintermute_oregon@lemm.ee 13 points 10 months ago (7 children)

What are the current specs and what are you going to?

[–] sunaurus@lemm.ee 10 points 10 months ago* (last edited 10 months ago) (1 children)

Currently the database is a managed DigitalOcean Postgres instance, but I am going to migrate it to a 32 thread 128 gb RAM dedicated server.

It's something that I've been hoping we won't need, as the managed database service has allowed me to not worry about patches, backups, etc (they took care of all of that automatically). Unfortunately it is clear now that further upgrades are just too costly on that service & actually the amount of configuration and tuning I would like to do is simply not possible there. So seems like right now, moving to the dedicated server for the database is the only option.

[–] wintermute_oregon@lemm.ee 2 points 10 months ago

Makes sense. I appreciate the reply back. I use to work on highly available fault tolerant systems. So I’m always interested how much hardware/etc to run something.

Sounds like some serious optimization needs to be done or there are a lot more transaction than I was expecting.

load more comments (6 replies)

[–] pfaca@lemm.ee 13 points 10 months ago

It's flying now! Thanks for the upgrade @sunaurus@lemm.ee.

[–] Xepher@lemm.ee 11 points 10 months ago

Really appreciate the advanced heads up. Thanks for being such a awesome instance admin!

[–] CluckN@lemmy.world 10 points 10 months ago (2 children)

When this comment is 2 hours old, I’ll have completed taking a colossal dump.

[–] dditty@lemm.ee 15 points 10 months ago (1 children)

1 hr old; hang in there champ! 🫡💩

[–] CTDummy@lemm.ee 15 points 10 months ago

Nothing after 5h. Died doing what he loved RIP Cluck

[–] 0ops@lemm.ee 4 points 10 months ago

Why the downvotes? I'm laughing like an idiot over here

[–] aniki@lemm.ee 8 points 10 months ago (1 children)

I just wanted to tell you both, "Good luck," We're all counting on you.

-Frank Drebbin, Police Squad

load more comments (1 replies)

[–] SurvivalMariner@lemm.ee 8 points 10 months ago

Thanks for the update and information. You really set an example for communication and level headedness in decision making. Thanks for providing this space for us.

[–] gregorum@lemm.ee 7 points 10 months ago

[–] Cavemanfreak@lemm.ee 7 points 10 months ago

It's become extremely responsive compared to before! Good job :)

[–] MoreCoffee@lemm.ee 7 points 10 months ago

Performance is night and day now, thanks for the hard work

[–] WeAreAllOne@lemm.ee 6 points 10 months ago

Yeap. Load times are waaaaay better. Many thanks for this!

[–] ominouslemon@lemm.ee 6 points 10 months ago

Thanks for the heads up and for the upgrade! Good luck

[–] Abnorc@lemm.ee 4 points 10 months ago

Things seem much improved! Good work and thank you for the work that you do.

[–] tsonfeir@lemm.ee 4 points 10 months ago

It’s sooo much faster now.

[–] SuperSpaceFan@lemm.ee 4 points 10 months ago (1 children)

The results are immediately visible in performance. Thank you @sunaurus!

[–] kratoz29@lemm.ee 3 points 10 months ago

Yeah, I noticed that too, I was afraid this upgrade would bring more issues, but so far everything is perfect!

[–] GrayBackgroundMusic@lemm.ee 3 points 10 months ago

Loving the new speed.

[–] EeeDawg101@lemm.ee 3 points 10 months ago

Whew thanks for your work sunaurus! Everything really does feel way more snappy.

[–] TWeaK@lemm.ee 3 points 10 months ago

Wahey, the best instance just keeps getting better!

[–] Nacktmull@lemm.ee 2 points 10 months ago

This is awesome! Thanks a bunch!

[–] Mixairian@lemm.ee 2 points 10 months ago (1 children)

I don't know if this is my imagination, but things are now loading way faster on the Lemmy Sync app since you completed the upgrade. Originally I had to really 20 seconds when opening the app for content to load and now it's instant. Thank you very much!

[–] narc0tic_bird@lemm.ee 3 points 10 months ago

Apps should be improved/just as affected, so it's likely not your imagination.

[–] 0ops@lemm.ee 2 points 10 months ago (2 children)

Is there a place to donate? I feel like you've earned a few bucks from me

load more comments (2 replies)

load more comments