this post was submitted on 27 Jun 2023
312 points (100.0% liked)

/kbin meta

6 readers
1 users here now

Magazine dedicated to discussions about the kbin itself. Provide feedback, ask questions, suggest improvements, and engage in conversations related to the platform organization, policies, features, and community dynamics. ---- * Roadmap 2023 * m/kbinDevlog * m/kbinDesign

founded 2 years ago
 

I discovered yesterday evening that Lemmy.ml is blocking all inbound ActivityPub requests from /kbin instances. Specifically, a 403 'access denied' is returned when the user agent contains "kbinBot" anywhere in the string. This has been causing a cascade of failures with federation for many server owners, flooding the message queue with transport errors.

This doesn't appear to be a mistake; it has been done very deliberately, only on Lemmy.ml. Lemmy.world and other large instances do not exhibit the same behavior. It also isn't a side effect of the bug introduced in Lemmy 0.18. You can observe by sending the following in a terminal

> curl -I --user-agent "kbinBot v0.1" https://lemmy.world/u/test
HTTP/2 200
[...]

> curl -I --user-agent "kbinBot v0.1" https://lemmy.ml/u/test                                
HTTP/2 403
[...]

> curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
[...]

> curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
[...]

Additional evidence of this not being a Lemmy 0.18 bug:

  • This occurs when making web requests to any location on the Lemmy.ml webserver, not just ActivityPub endpoints.

  • Go to https://fedidb.org/software/lemmy and pick an instance running 0.18.0. Perform the above commands, replacing the URL for Lemmy.ml with that particular instance's address.

If this continues, my instance may need to defederate from Lemmy.ml. This is especially problematic because Lemmy.ml continues to federate information outbound to other kbin instances while refusing to allow inbound communication from them.

Spoofing the user agent is less than ideal, and doesn't respect Lemmy.ml's potential wish to not be contacted by /kbin instances. I don't post this to create division between communities, but I do hope that I can draw awareness to what's going on here. Defederating /kbin instances entirely would even be better than arbitrarily denying access one-way. This said, we should all attempt to maintain a good-faith interpretation until otherwise indicated by the Lemmy developers. It's possibel that this is a firewall misconfiguration or some other webserver-related bug.

Relevant comment from me (#354 - [BUG] Critical errors/failed messages during messenger:consume)

Edits:

  • Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.

  • Someone has posed a question on Lemmy.ml about the block here: https://lemmy.ml/post/1563840

top 50 comments
sorted by: hot top controversial new old
[–] ernest@kbin.social 195 points 1 year ago (3 children)

It's possible that this is a consequence of the latest Lemmy update, in which a lot has changed. I have noted that kbin has some issues with request signature in communication with certain instances. I will try to check it tomorrow first thing in the morning.

[–] Upvotes_Kills_Birds@kbin.social 28 points 1 year ago (1 children)

Wow, thank you for the quick communication. Amazing aptitude. Almost nowhere else do you find a lead dev in the comment trenches letting us know what's happening, I'm kinda baffled.

[–] riktor@kbin.social 15 points 1 year ago

we love ernest!

[–] svovl@kbin.social 12 points 1 year ago (1 children)
[–] LollerCorleone@kbin.social 15 points 1 year ago (10 children)

Those links are working fine. Kbin is federating well with those instances. A bit of latency is normal.

load more comments (10 replies)
load more comments (1 replies)
[–] phi1997@kbin.social 48 points 1 year ago (1 children)

If they don't want us to communicate with the users of their server, they should be defederated. Sucks for the users, but they can hop to another server whose admins don't break communication inelegantly and ghost admins who want to know what's going on.

If it's just a bug, then obviously we should wait for it to be fixed, but if they're not even saying they'll look into it, I don't have much hope.

[–] ripcord@kbin.social 25 points 1 year ago (4 children)

Before assuming, has anyone...asked them?

[–] Deceptichum@kbin.social 27 points 1 year ago (1 children)

Before assuming no one has asked them, let’s read the post

“Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.”

[–] ripcord@kbin.social 20 points 1 year ago* (last edited 1 year ago) (2 children)

Ok, but, as the thread over there says, it's not like they're singling out kbin as far as the code shows. Just because no one there has answered yet doesn't mean they've refused to answer. Give it some time or some actual replies before assuming the worst.

Also that edit was added after my question.

It's been a few hours, this is way way too fast for the pitchforks to come out.

[–] Deceptichum@kbin.social 9 points 1 year ago

Code and config is different, it’s irrelevant if it’s not in the git.

[–] frex@kbin.social 8 points 1 year ago* (last edited 1 year ago) (1 children)

If I use a firewall to block kbin in user agent then it doesn't matter what code is running, it will throw 403.

Edit: Just checked out, the first request with kbin user agent, it doesn't have x-powered-by: Express, suggesting that it is blocked in nginx level, it doesn't even reach the backend Express. This will not be reflected in the code whatsoever.

 ~  curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
server: nginx
date: Wed, 28 Jun 2023 01:50:53 GMT
content-type: text/html
content-length: 146
vary: Accept-Encoding

 ~  curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
server: nginx
...
x-powered-by: Express
...

load more comments (1 replies)
load more comments (3 replies)
[–] svovl@kbin.social 40 points 1 year ago

In the spirit of a thriving fediverse, I suggest not to assume the worst of other parts of it before we know a bit more of what's going on.

[–] crowsby@kbin.social 34 points 1 year ago

They're moving to a paid API as a prank.

[–] camelbeard@kbin.social 34 points 1 year ago (2 children)

At first I thought it was maybe a config error to block bots. But after some testing I see that the 403 is only given when the word "kbinbot" is in the user agent. String that just return the normal response I tested are "testbot", "kbinbo", "binbot".

[–] Jajcus@kbin.social 18 points 1 year ago

Maybe it is some kind of automated rule 'block bots that cause lots of traffic'. kbin is becoming more and more significant, so it probably causes some non-insignificant load on Lemmy servers. Legitimate load, but the IDS rules would not know that.

load more comments (1 replies)
[–] Eisenhowever@kbin.social 28 points 1 year ago (7 children)

Doesnt surprise me, the developers of lemmy (which are owners of .ml) have an agenda. They are into censoring on ukraine news, and other stuff. I dont get why people are choosing lemmy over kbin when they are equally bad looking

[–] TheRoarer@kbin.social 28 points 1 year ago (2 children)

Wait why is kbin bad looking?

[–] TwilightVulpine@kbin.social 14 points 1 year ago (1 children)

It looks better than reddit.

[–] slicedcheesegremlin@kbin.social 16 points 1 year ago* (last edited 1 year ago) (3 children)

yeah, kbin is probably the nicest looking reddit alternative i've seen. Really sleek design without leaning into the overly overly mobile-focused watered down New Reddit bs, while also not nearly as hideous as old reddit.

load more comments (3 replies)
load more comments (1 replies)
load more comments (6 replies)
[–] Tyrannosauralisk@kbin.social 20 points 1 year ago (2 children)

There is a question about this up on lemmy.ml here:

https://lemmy.ml/post/1563840

[–] AnonymousLlama@kbin.social 13 points 1 year ago

At least they're equally as confused, hopefully this is just an issue in communication and the federation of content will be back in full swing soon

[–] btaf45@kbin.social 8 points 1 year ago

Hey Yote.zip is saying this. What's up with that?

" I’ll add that I have recently noticed that practically nothing that I post has been getting through to kbin."

[–] barista@kbin.social 16 points 1 year ago (1 children)

Relevant issue for gracefully handling 403 errors is here

load more comments (1 replies)
[–] luna@kbin.social 15 points 1 year ago (1 children)

Happy to see this. I've been avoiding .ml anyway, this makes my job easier. Fuck the Lemmy devs.

load more comments (1 replies)
[–] missingno@kbin.social 15 points 1 year ago (2 children)

If this is intentional, I have to wonder why they're doing it in such a troublesome way rather than defederating properly. If they want to defederate, so be it, but then just do that.

[–] phi1997@kbin.social 12 points 1 year ago (1 children)

This approach only makes sense as a blanket defederation of all kbin servers. Seems shady.

[–] jiml78@kbin.social 11 points 1 year ago* (last edited 1 year ago) (4 children)

Well, lemmy.ml is the lemmy devs. Kbin is a competitor to their software. They might hate that Kbin is growing really fast and I think is overall better than straight lemmy.

[–] luna@kbin.social 10 points 1 year ago (1 children)

I think it's more likely they just disliked the fact most Kbin users don't support genocide, while Lemmy has a history of that. The Stalinist side of Lemmy, moderated and endorsed by the developers, has made fun of Kbin.social before

load more comments (1 replies)
[–] harmonea@kbin.social 9 points 1 year ago* (last edited 1 year ago)

It would be pretty against the spirit of ActivityPub if this were the case. Holding out to actually hear from them on this, hopefully, rather than assume such bad things about them.

They just recently had a software update I believe, so hopefully it's just a side effect of that.

load more comments (2 replies)
load more comments (1 replies)
[–] Patrick-Haverkamp754@kbin.social 13 points 1 year ago (1 children)

has anyone message lemmy.ml admins to see what may be up? maybe this is a bug rather than outright blocking

[–] barista@kbin.social 15 points 1 year ago

Yes, folks have tried reaching out in their Matrix chat without luck so far.

[–] Hakaku@kbin.social 13 points 1 year ago

I suspect it's nothing more than an overeager web application firewall (WAF) blocking the requests automatically.

[–] r00ty@kbin.social 12 points 1 year ago

I was almost ready to say that maybe they're just looking for "bot" in the UA. But, no. It must be "kbinbot" anywhere in the UA, and case insensitive. So, pretty deliberate it seems.

[–] Zzinck@kbin.social 9 points 1 year ago (1 children)

Lemmy is no different than Reddit. They silence and ban anyone they don't agree with and do not care about the users.

[–] ripcord@kbin.social 23 points 1 year ago* (last edited 1 year ago) (6 children)

You're making one hell of an assumption here.

Most likely it is a misconfiguration or bug. Or misunderstanding.

load more comments (5 replies)
[–] Cloudless@kbin.social 8 points 1 year ago (4 children)

This will save me time. I am trying to block all communities hosted by the tankies instance.

load more comments (4 replies)
[–] ngmi@kbin.social 8 points 1 year ago (1 children)

I hope this issue is resolved quickly. It seems to be quite problematic.

[–] GunnarRunnar@kbin.social 15 points 1 year ago (1 children)

Hopefully it's a technical hiccup since every other option feels... shitty?

[–] sadreality@kbin.social 31 points 1 year ago (3 children)

well, i don't think the tankies liking their safe space being overload with "normies" decent takes.

just a speculation tho

[–] GunnarRunnar@kbin.social 18 points 1 year ago (1 children)

Oh it's a tankie instance. Not a big loss then.

[–] FaceDeer@kbin.social 8 points 1 year ago

Unfortunately Lemmy.ml was also the "flagship" Lemmy instance for a while, at least initially. I've got an account there myself because at the time I was signing up the only other large instances were offline for various reasons. I don't use it much any more and would be fine with never using it again, but I bet a lot of people wound up on it and didn't continue exploring further. So this is a bit of a mess and I would have hoped account migration tools would be better developed by now.

[–] gk99@kbin.social 16 points 1 year ago (2 children)

If that's the case then they can shut it down and move to lemmygrad where at least they're not pretending to be normal people. My biggest problem with lemmy.ml is that they will enforce their beliefs with "orientalism" bans without ever saying that this will happen if you post contrary to their stances.

Personally, I hope it's intentional and leads to other places defederating lemmy.ml as a result, finally cutting them out of the mainstream instances.

[–] sadreality@kbin.social 11 points 1 year ago* (last edited 1 year ago)

I mean... They try to maintain some veneer of propriety but than daddy Stalin and Mao worship comes out.

Ask them about tianamen square... 🤫

load more comments (1 replies)
load more comments (1 replies)
load more comments
view more: next ›