Just hopping into the chain to say that I appreciate you and all of your hard work! This placeβLemmy in general, but specifically this instanceβhas been so welcoming and uplifting. Thank you!
Lemmy.World Announcements
This Community is intended for posts about the Lemmy.world server by the admins.
Follow us for server news π
Outages π₯
https://status.lemmy.world
For support with issues at Lemmy.world, go to the Lemmy.world Support community.
Support e-mail
Any support requests are best sent to info@lemmy.world e-mail.
Report contact
- DM https://lemmy.world/u/lwreport
- Email report@lemmy.world (PGP Supported)
Donations π
If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.
If you can, please use / switch to Ko-Fi, it has the lowest fees for us
Join the team
Keep in mind that the upcoming Lemmy update will probably fix this I think. (Replacing websockets)
Yes I really hope so!!
Maybe this is a dumb question, but why would replacing websockets speed things up? I read the Wikipedia page on it, but I guess I donβt understand it fully.
In general websockets scale badly because the server has to keep open a connection and a fair amount of state. You also can't really cache websocket messages like you can normal HTTP responses. Not sure which reasons apply to Lemmy though.
Thanks for the explanation!
The caching problem is definitely part of it from conversations on GitHub.
I really hope someone is doing some level of performance testing on those changes to make sure the changes fix the performance issues.
maybe related, but I've noticed that upvoting/downvoting has similar lag delays
Same. It would be good to fix this
I can't up/down vote at all.
At least the "reply" button goes away so I don't end up double- triple- or even duodecuple-posting! Thanks for all the hard work that must be going on behind the scenes right now!
I kept getting a timeout message from Jerboa which led me to think I hadn't been posted. So I ended up submitting the same joke to the Dad Jokes sub three times. Which actually is how dad might tell that joke.
Lemmy is now your digital dadlife assistant.
I think in that case it's a feature not a bug.
I get this occasionally with Jeroba too, I had assumed it was because I'm on Mint and the connection is shoddy but maybe it's an issue with the client.
Again, thank you for the outstanding work! You are awesome!
Also, the new icon for lemmy world is great!
Have you tried enabling the slow query logs @ruud@lemmy.world? I went through that exercise yesterday to try to find the root cause but my instance doesnβt have enough load to reproduce the conditions, and my day job prevents me from devoting much time to writing a load test to simulate the load.
I did see several queries taking longer than 500ms (up to 2000ms) but they did not appear related to saving posts or comments.
Been noticing this in the app Iβm working on. Pretty much all POST requests fail to return a response and just timeout after 60 seconds. A quick refresh shows that the new items do successfully get created though.
I assume that there is something that is O(N), which explains why wait time scales with community size (amount of posts, comments)
Oh, Big-O notation? I never thought Iβd see someone else mention big O notation out in the wild!
:high-five:
you are going to meet a lot of OG redditors in the next few weeks. Old reddit had Big O in every post, even posts with cute animals.
Thatβs pretty neat! Iβve honestly never seen it mentioned on Reddit before, so got a bit excited to see someone mention it here, admittedly maybe too excited.
there was a time, before the digg invasion, where someone would post a picture of a woman feeding 30 cats and there would be Big O jokes about how well this would work and crazy modifications to the situation to improve it. This would be almost every thread at one point on the site. I miss it.
That actually sounds like something I would have enjoyed. I joined Reddit around the time it started taking over, I think.
Those years before the Digg invasion are often the magic and mythic times OGs speak of, lots of amazing happened after too, but that early culture was almost entirely washed away by Digg and subsequent mainstreaming of some subs. the move from being a focus on quality link aggregation to a points game really helped push things down hill.
At this point I'm not sure meta-moderation is really a workable system and I kind of hate internet points. There are some new tools and techniques we can try in a system like this.
Thanks for your and the other Lemmy devs work on this. These growing pains are a good thing as frustrating as it can be for users and maintainers alike. Lemmy will get bigger and this optimization treadmill is really just starting.
Thank you for your hard work and keeping us up to date.
Does this behaviour appear on other big instances? E.g. lemmy.ml?
Yes it does, tried this workaround before.
Yes. Absolutely does happen on other instances that have thousands of users.
Great, so it's reproducible and Lemmy-the-app related, not instance-specific. Should be fixable across the board once it's identified and resolved.
In my case, the page keeps spinning but the post is not submitted, regardless of reloading the page or waiting for a long time. There was one case where I cut down significantly on the amount of characters in the post and then it posted, but I have been unable to replicate this.
I have the same issue with image posts. If I submit them through the app the posts counter on my profile goes up, but there's no post. I also can't retrieve any posts for my own account. It says I have 3 but it shows none.
Comments work OK so I'm not sure what the problem is. I was worried I got restricted or something.
Edit: Comments seem to work as the OP suggests though. I get the spinner but they've been posted already.
Edit2: It seems like I can't post link threads either.
Edit3: Even worse, it seems like I can't see my own posts but others can. So I might have spammed Warhammer40k and tabletopminis and I can't even clean up my own mess.
@ruud@lemmy.world Yo dude, first off huge props and a big thank you for what you have setup. Iβll be donating monthly while I am here. I appreciate that we have an alternative to Reddit at this critical moment in time.
I do have a question on your long term plans, do you want to continue to expand and upgrade the server, as funding allows, or is there a cap that you will close off the server to new members? Or perhaps make it more of a process to join?
Well if all the Reddit users would get over to Lemmy I guess all servers would need to scale up... but I think the server we have now is powerfull enough to grow quite a lot, as long as the software gets tuned ..
Thank you so much
I noticed that too, page keeps spinning but comments are posted immediately anyway.
I noticed this, thanks for the clarification
ok, so it's not just me. Hope it gets resolved soon!
its def more hung up today, oddly its only first level replies for some reason
Thanks for posting the workaround and for working to resolve the issue. Lemmy is a great place, and a real breath of fresh air after Reddit.
This is the biggest issue I have run into. Thanks for looking into it.
One of the large applications I was working on had the same issue, to solve it we ended up creating multiple smaller instances and started hosting a set of related API's in each server.
for example read operations like list posts, comments etc could be in one server. write operations can be clusered in one server.
Later, whichever server is getting overloaded can be split up again. In our case 20% of API's used around 3/4th of server resources, so we split those 20% API's in 4 large servers and kept the remaining 80% API's in 3 small servers.
This worked for us because the DB's were maintained in seperate servers.
I wonder if a quasi micro-services approach will solve the issue here.
Edit 1: If done properly this approach can be cost effective, in some cases it might cost 10 to 20 percentage more in server costs, however it will lead to a visible improvement in performance.
Iβve done this twice in the last 20 minutes and the content is not there. This workaround was working earlier today though.
This issues is affecting other instances from what I can tell
Good to know it isn't just me then.