Selfhosted

Header is expired issue is big part of the current federation problem. And whether you know it like it or not, you’ve just made the matter worse. You’re not to blame though. I’ve done it too, along with many other people self hosting our own instance.

The way federation currently works is each write action must be federated outwards to each federated instance. A comment reply, such as this one, must be federated outwards by the hosting instance. An instance receiving a federation event must also discard messages that are older than 10 seconds.

Here lies the problem… popular instances like lemmy.world and lemmy.ml has thousands of users, and thousands of federated servers. Yesterday, when I checked, lemmy.world had 3600 users per day and 2200+ federated servers. If there’s a really popular post on a very popular community, and 10% of the users comments on it? Lemmy.world server must send 360x2200 = 700K+ outbound federation event messages. Each one of these are sent over HTTPS via TCP so they can’t send all of them at the same time, and the messages are put into a queue where the federation workers will send them out. Each worker will send the message and because HTTPS is over TCP, it is not fire and forget, the worker must wait for acknowledgement for the packets. If an instance owner gets bored because they’re not getting all the messages and shuts down? Now the worker needs to wait for that to error out and thereby delaying messages further down the queue. If it had to wait more than 10 seconds? Everyone down the queue will just get expired messages because the event is already outdated.

So now you’ve already created an instance and adding to the load of the network, just like me, what can you do? Keep your server online in a fast data center. Use Cloudflare to reduce latency. That way at least your server isn’t going to introduce too much latency to other servers down the queue. Hopefully the devs figure out something to make the process better. I’ve put in a more scalable notification fleet architecture change on GitHub already. Lets see if they can implement that or change other requirements on the system.

Can confirm that all of the above applies to my instance as well

I can only answer a couple.

3. This is by design, due to how federation works. Federation is literally just:

• Instance A requests information from Instance B
• Instance B responds to the request and sends it back
• Instance A follows Instance B
• Instance B now federates all future posts to Instance A

There's nothing more complicated to it, but it does mean that instances cannot know about other instances without being told, as there is no central location that instances connect to in order to find out about all other instances.

4. Only posts after subscribing are federated to an instance, it doesn't backfill. An option for admins of an instance to request a backfill would not be a bad option though, but as time goes on, backfilling an entire community could take too much data on instances.

5. Issue with Lemmy.ml, although when you see Subscribe Pending, you tend to still see things in your feed.

I'm only just now learning more about federation, so I'm no expert.

Is your instance setup to use NTP? federation requires fairly accurate time from my understanding, and shared hosting normally drifts quite a bit, which might be where your "Header is expired" issue comes from. It could also have something to do with your instance not processing incoming federation stuffs fast enough I think?

There is a chat space at https://matrix.to/#/#lemmy-space:matrix.org with people that run their own instances, which might help answer these questions

1. Yes, because posts/comments are created there and then federated to subscribed instances, which happens on a queue. Depending how popular that community is, there's probably a lot of other instances ahead of you in the queue, but lemmy.world is also a huge instance in general so they might be under al ot of load.
2. Same reason as above, federation is asynchronous and happens on a queue.
3. In theory you could probably write a bot that just auto-subscribes to communities in other instances, but you're going to have a LOT of traffic going through to your instance and you'll basically need it to be as powerful as those instances. Sending you only the data you've subscribed to receive is how all federated services work, an instance just can't reasonably get everything. However I've put some thought into this and I think a way to browse other communities without subscribing to them would be a game changer.
4. This could be a lot of things unfortunately. Have you tried making new posts?
5. Same on my end, I think lemmy.ml is waaay overloaded at this point and I think it might also be behind CloudFlare which complicates the federation process.
6. Seems like a problem on the kbin end. You might want to create an issue on the kbin's codeberg.
7. You'll probably want to post that in one of the lemmy communities or on its issue list (make sure to search first).

I can't even see this post on my selfhosted instance, so that's fun.

With regard to #6, you apparently have to search for the URL of the Kbin magazine (e.g. https://kbin.social/m/RedditMigration) and then you can subscribe as normal from that point on.

And yeah, the discovery and comments thing is super annoying. There will be completely different conversations based on where you're viewing the post, which kind of kills the idea of all these instances federating in the first place.

That being said, it appears to be feeds coming from lemmy.world and lemmy.ml that have the majority of the problems with comment/post sync. Unfortunately that's the selfhosted community.