this post was submitted on 18 Jan 2024
13 points (100.0% liked)

Meta

630 readers
1 users here now

Discussion about the aussie.zone instance itself

founded 1 year ago
MODERATORS
admin@aussie.zone
lodion@aussie.zone
13
Ongoing federation issues (aussie.zone)
submitted 10 months ago* (last edited 10 months ago) by Zagorath@aussie.zone to c/meta@aussie.zone
25 comments fedilink hide all child comments
 

I'm still seeing cases where comments don't seem to be federating correctly. For example:

Other instance aussie.zone
https://lemmy.world/comment/6786498 https://aussie.zone/comment/5886332
https://feddit.uk/post/6819925 https://aussie.zone/post/6020867
https://lemmy.ml/comment/7476880 https://aussie.zone/comment/5881947
https://lemmy.blahaj.zone/comment/5759107 https://aussie.zone/comment/5888459
https://midwest.social/comment/6313276 https://aussie.zone/comment/5889962

I'm not sure what the cause is, but the last few days it seems more of my comments are failing to federate than are succeeding.

all 27 comments
sorted by: hot top controversial new old
[–] No1@aussie.zone 6 points 10 months ago* (last edited 10 months ago) (1 children)

I'm claiming federation issues as the reason my posts aren't getting thousands of upvotes.

[–] maniacalmanicmania@aussie.zone 3 points 10 months ago (1 children)

I still can't believe you're not president of the world yet. Like wtf?

[–] No1@aussie.zone 3 points 10 months ago* (last edited 10 months ago)

Careful, or I'll turn to the dark side and federation will become a dictatorship so fast it'll make Palpatine spin in his grave. Darth No1orious 🤣

[–] lodion@aussie.zone 6 points 10 months ago (1 children)

☹️

I'll see what I can see...

[–] lodion@aussie.zone 7 points 10 months ago (3 children)

Ok, for some reason practically all instances were flagged as "dead" in the database. I've manually set them all to be requeued... server is now smashed as it attempts to updated the ~4000 instances I've told it are no longer dead. See how this goes...

[–] lodion@lemmy.world 6 points 10 months ago (1 children)

Replying from my lemmy.world account, seems to have caught up eventually.

[–] lodion@lemmy.world 1 points 10 months ago (1 children)

test

[–] lodion@aussie.zone 1 points 10 months ago

reply

[–] lodion@aussie.zone 6 points 10 months ago

Seems to have sorted it for the most part... not sure what caused it, will do some more digging.

[–] Zagorath@aussie.zone 4 points 10 months ago

Boss!

[–] lodion@aussie.zone 5 points 10 months ago (1 children)

Ok, something is busted with the lemmy API endpoint that shows current federation state. It is currently showing nearly all remote instances as dead:

But "dead" instances are still successfully receiving content from AZ, and sending back to us.

[–] lodion@aussie.zone 5 points 10 months ago (2 children)

Ok, there's more to this than I first thought. It seems there is a back end task set to run at a set time every day, if the instance is restarting at that time the task doesn't run... this task updates the instances table to show remote instances as "seen" by AZ. With the memory leaks in 0.19.1, the instance has been restarting when this task is running... leading to this situation.

I've updated the server restart cronjob to not run around the time this task runs... and I've again manually updated the DB to flag all known instances as alive rather than dead.

Will keep an eye on it some more...

For anyone curious, two of the bugs that are related to this:
https://github.com/LemmyNet/lemmy/issues/4288
https://github.com/LemmyNet/lemmy/issues/4039

[–] Zagorath@aussie.zone 2 points 10 months ago

Thanks for the update!

[–] Zagorath@aussie.zone 1 points 10 months ago (1 children)

Looks like what caused this might have been resolved? https://github.com/LemmyNet/lemmy/pull/4377

[–] lodion@aussie.zone 2 points 10 months ago (1 children)

Excellent, hopefully that is included in 0.19.3 that came out overnight... will upgrade shortly.

[–] Zagorath@aussie.zone 1 points 10 months ago (1 children)

It is. I found the page I linked from the release notes.

[–] lodion@aussie.zone 2 points 10 months ago (2 children)

Cool, we're on 0.19.3 now.

[–] Zagorath@aussie.zone 1 points 10 months ago

Sweet

[–] maniacalmanicmania@aussie.zone 1 points 9 months ago (1 children)

How do things look after the past week?

[–] lodion@aussie.zone 2 points 9 months ago

0.19.3 still has the memory leak, I've left Lemmy to restart once a day to mitigate this. Federation seems to be fine for the last week after some minor changes and the upgrade.

[–] maniacalmanicmania@aussie.zone 4 points 10 months ago* (last edited 10 months ago) (2 children)

I found that this post's body text is best viewed on desk/laptop for anyone trying to make heads or tails of all the links on mobile. Edit: Looks like my idea that this issue was due to some instances not being on 0.19.2 has been shot out of the water.

[–] Zagorath@aussie.zone 4 points 10 months ago (1 children)

Yeah fair call. I've edited it into a table which should hopefully be easier to read.

[–] maniacalmanicmania@aussie.zone 3 points 10 months ago

Ooo yeah scrollable table, nice work, much better at least on voyager 👍

[–] trk@aussie.zone 4 points 10 months ago

Cant argue with any of those examples.

In a delicious cherry on top, this thread is also not federating (ex: https://lemmy.world/c/meta@aussie.zone)