kbin.earth meta

As I'm sure y'all have noticed, the server has been having stability issues the past week. Every once in a while, the server seems to "crash"; it will time out on all network requests for about 5 minutes, happening several times within an hour. Additionally, there have been several times over the past week when the server queues got backed up, resulting in incoming and outgoing federation traffic being delayed.

At first, I assumed the server hardware had simply become insufficient to handle all of the network and federation traffic we have been receiving, potentially due to a DDOS attack. I was honestly starting to think I would need to upgrade the server hardware, which is ridiculous, as the VPS is already quite powerful and is certainly expensive enough.

Luckily, @green_copper helped with debugging and noticed the server timeouts only occur when the microblog section is viewed (and sometimes when the combined section is viewed). So rather than a lack of hardware being the issue, I believe it actually has to do with the recent Mbin upgrade. My guess is that an SQL query was changed in one of the commits between Mbin v1.9.1 and v1.10.0, which resulted in a widely inefficient query, causing the server to freeze up and time out on requests.

I have and will be investigating the issue this week. I will start by analyzing which database queries are causing the timeouts, but if needed, roll back to Mbin v1.9.1 and try to figure out which specific Mbin commit introduced the bug. Additionally, I may have access restricted only to signed-in users while I'm working on this.

In the meantime, if you guys can refrain from viewing microblogs (or combined view), that would keep the server accessible for everyone and help me work on it faster, as I can't do anything while the server is frozen. If you want any realtime updates on the server issue (or anything else related to kbin.earth), feel free to join the Matrix room.

Thanks for sticking around and for your patience!

I have just completed the upgrade of kbin.earth to Mbin v1.10.0. Please let me know if you run into any issues.

Here are the changes listed for end users:

• Add support for lists of keyword filters of content; they can be scoped to content, comments and profile-overviews and can have an expiration-date
• Show the custom title of users and add the option to set one for yourself
• Add the option to also show content which was boosted by people you follow in the Combined view
• Pagination is now done by "anchors" (like that date of the last thread on the page) instead of page-numbers, to improve performance
• Add an indicator when a user has their "cake day"
• Fix search not returning most of Lemmy users and magazines
• Fix the search when searching content, users or magazines by their URL
• The date field of the search form now has a preset value
• Small UI improvements to make the elements of posts and threads more consistent (the layout for posts in Combined view can be toggled in the UI settings)
• Prevent opening a thread or post when touching interactive elements (buttons, spoilers, ...) on mobile
• Suppress notification for added content if it is more than two days old (for example after delayed federation)
• Fix infinite scrolling of comments
• Fix reply button for comments on locked threads and posts not being disabled

Full release post: https://fedia.io/m/mbinReleases@gehirneimer.de/t/4015612

I complained about it several times here and to various instances admins. Screenshot from same thread. I have literally been cursed out, called "stupid," "c***,” "b****,” and other unsavory slurs. I quit complaining because nothing was ever done, and I realize this is a free service offered and admins have lives, jobs, and plenty of work and expense by generously offering this service. I just thought it was worth another try.

Thank you

https://imgshare.cc/hw1e589o from the same thread.

I've had a couple of cloudflare errors logging on today, errors that were supposedly on the server end.

I also notice that my feed hasn't been updated for 8 hours (at least that's the time passed since the latest thread was published). Visiting other instances, it appears there have been several updates to my subscribed communities that aren't federated to KbinEarth. Threads, comments, votes, etc.

Is everything alright over there? Or have I just been shadowbanned 😓

Hey everyone, I just wanted to say thank you for supporting kbin.earth and Interstellar!

Thanks to all our generous sponsors, all expenses from the year 2025 (including server rental, object storage, domains, Apple Developer License, and Apple hardware) have been covered; almost no out-of-pocket needed from me. That's a huge deal!

To the Mbin team: none of this would be possible without you! You've forked /kbin and have kept supporting it ever since, adding new features, diagnosing and fixing bugs, recommending server setup improvements, and much more. You guys are quick to respond and are nothing but reliable! Special shout out to @BentiGorlich@gehirneimer.de and @melroy@kbin.melroy.org!

To Interstellar's co-developer: thank you @olorin99 for literally being around since the beginning. Interstellar's very first commit was on Dec 11, 2023, and yours was just a few weeks after, on Dec 28, 2023, adding initial support for microblogs. Thanks for contributing to and developing Interstellar all this time!

To kbin.earth's mods/admins: thanks for keeping this place clean and manageable! Especially checking for sign up requests every day :) @green_copper @kindenough @aasatru

Lastly, thanks to all the 1-off contributors, translators, anonymous sponsors, and anyone else I haven't mentioned. And of course, all you users who stick around here :)

Here's the list of user-facing changes from the release notes:

• We enhanced security so that after a user changes their password or 2FA, all the current sessions of that user will be invalid and the user will need to log in again.

• A new combined front page. You could make the combined front page the default in your profile settings. Which means you see both threads and microblogs combined on the homepage.

• We also introduced a new feature under general settings, where you can select "Who can send you a direct message" (defaults to everybody).

• We added support for magazine banners in Mbin (which is also compatible with Lemmy Communities).

• Mbin combines the thread form into just one form (instead of having article, link, and photo having a separate form). Mbin now also federates bans correctly (both incoming and outgoing bans).

• Global mods can now manage (view, approve and/or deny) account signups. Global mods can now also receive signup notifications, which will also come with a new menu item in the drop-down menu.

• Plus various other bug fixes and CSS layout improvements, ban notification fixes, and too many other fixes to mention here.

Hey all, Hoping to find someone who's been in a similar boat and was able to solve this. I am consistently getting a 500 Internal Server Error, but only when visiting /all on mobile (including the mobile "app"). I tried logging out and back in again. When I am logged out, /all works fine, but when I log back in I get the 500 error again.

Ideas?

Title. A story from maga.place showed up on my feed. Thank you.

Sorry that kbin.earth has been down today. Apparently, several people had issues yesterday, but I hadn't realized the full extent due to Matrix severely delaying my notifications.

The server unfortunately crashed while I was unavailable for the day, so I wasn't aware of the issue and couldn't fix it till I got back.

As an aside and general note for everyone, I don't intend on ever shutting down kbin.earth without notice. I also don't intend on completely disappearing either; I don't want people needing to wonder if something happened to me. If I plan to shut down kbin.earth (which I don't btw), I will give as much warning in advance as I can. (I'm not going to be a repeat of kbin.run)

Today only occurred due to an unfortunate combination of technology issues (my failure to receive Matrix notifications and the server crashing) and unavailability.

Again, sorry for the inconvenience. Best regards.

Over the last days and especially today I noticed quite a lag in federation from other instances. Comments and votes to my threads show up after half an hour or more.

I don't that much of an issue with this (as long as the stuff is federated at all), but maybe @jwr1@kbin.earth should check if the server is experiencing some underlying problems.

Sorry guys, but the spam users are getting out of hand. I've had multiple users (probably the same person) register for accounts and post digital art (in multiple communities) that sexualizes children, which I do not condone at all. Of course, they had to go and ruin it for any genuine person who wants to register.

Let me take a moment to restate the rules (which are always listed on the about page). They are simply: be respectful, no spam or advertisements, and no pornography or explicit content. Hopefully, it's implied that any content that could even be mistaken for CSAM is prohibited. If you have any questions or concerns about the rules, don't hesitate to reach out.

That being said, the only difference with registration is that you now have to fill out a text field explaining why you want to join the server, and then you have to wait for manual approval by an admin.

Since manual approval is now required, I am looking for a third admin who thinks they could help out, as I am not available all the time. Ideally someone who has already been using kbin.earth for a while and is decently active.

I see a new spam account among the new users section, but I don't see a way to report their profile? They don't have any comments, threads, or posts yet so I can't use the report functions there. Any ideas?

Hey all, I have been having problems getting offsite images to load via the image markup in comments. I see other Fediverse instances allow users to upload images directly, is that a feature that is active on KBin.Earth? If so how do I use it? I'm mainly using Interstellar as my client.

Thanks!

ClientException: Request failed with status 400: Bad Request: ("type":"https:VV tools.ietf.org/htmlV rfc2616#section-10","title":"An error occurred","status":400,"detail":"Bad Request"), uri=https://kbin.earth/api/ magazine/1/posts

Any hints what's up? Is my post too long?

Crossposted from https://gehirneimer.de/m/updates@kbin.melroy.org/t/766866/Mbin-security-disclosure

As most of the servers listed on the fediverse.observer and fedidb are not at v1.8.3 anymore, we need to talk about the security patch we released as part of v1.8.4. We have tried to get in touch with the remaining instance admins and gave them a week to update their instances.

In v1.8.3 a bug was introduced that caused a significant information leak on the user outbox endpoint, reachable through https://mbin.instance/u/username/outbox. This endpoint contains all public activities of a user. On servers running v1.8.3. this endpoint did not return JSON in an ActivityPub compatible format, but just serialized data. This serialized data contained nearly every bit of data Mbin has about a user: the IP, the email address, the private key to sign activities from this user, securely hashed passwords, 2FA secret and backup codes, etc. We think it is unlikely that someone made use of this, as this endpoint is not commonly used. Other ActivityPub software of course uses this endpoint to fetch data, but if that data is not in a compatible format it just ignores it.

We are very sorry about this and honestly very frustrated that it slipped by.

What can users do

The only thing you can really do is to change your password and two factor authentication (disable and re-enable it).

What can admins do

You could check your access logs for any requests on this endpoint not coming from a known fediverse software to research your instance specific instance.

What did we do / What are we going to do

To prevent this from happening again we introduced automated tests on that endpoint and will do so on similar new endpoints (like a magazine outbox) in the future.

We will also add a new command next release to generate new private keys for all users to prevent impersonation. However that might cause rejected activities for up to 24 hours. Every software we checked updates remote users at least every 24 hours including re-fetching the private key.

I am on a short break while driving, so I won't be able to reply. Somehow I am getting notifications like replies to comments, in threads and magazines where those notifications are off, I haven't participated. This started sometime in the wee hours Eastern USA time zone.

Thank you for your thread @green_copper, I have been looking into the issue and figured out one of the commits recently pushed to Mbin was borked (which I had updated to yesterday). I've reverted the commit and that seems to have fixed things.

The server is now working at max capacity to catch back up on federation, but it could take over a few hours, as the queue had over a million messages built up.

Thanks!

Since yesterday no external posts come in and also the votes are not synced.

Is it just me and if not is there already a suspicion what may cause this?

Exactly the title. I was considering setting up a PieFed server since that seems to be all the rage, and I thought I'd ask if y'all would have any interest in that.

It would be run with exactly the same polices as kbin.earth is run: lite defederation, be respectful, no spam, and no porn.

To add some info, PieFed doesn't have microblog capabilities yet, but it does have quite a lot of features that Mbin does not have, such as Feeds (combined community views), and the moderation/administration tools seem to be off the scale (in a good way).

I will admit, PieFed's API is nowhere near as mature as Mbin's, and definitely not Lemmy's, because that's just not what the PieFed developers are focusing on. Interstellar does provide PieFed support though.

Over the past week, I've seen the network traffic more than double from the usual amount, which has caused major noticeable slowdowns here. The total network bandwidth has gone from ~25GB on May 10th to ~58GB today.

I'm currently investigating the cause of the spike, but have not found anything yet. I can only assume it's from some form of DDOS attack.

In the meantime, I have temporarily doubled our server resources to account for the increased strain, which will hopefully reduce the number of slowdowns everybody's encountering. I'm hoping that sometime in the next few days, I will be able to figure out the root cause of the issue so we can get things back to normal.

Thank you for your patience.