106
taught my machines to talk and this is how they repay me (lemmy.nz)
submitted 3 days ago by GrapheneOSRuinedMyPixel@sh.itjust.works to c/linuxmemes@lemmy.world
22 comments fedilink hide all child comments

have you ever successfully broken the zfs to the point where it's not recoverable?

top 22 comments
sorted by: hot top new old
[-] lemmyvore@feddit.nl 14 points 3 days ago* (last edited 3 days ago)

I'm honestly terrified to use ZFS. Roast me if you must. It has got to be the most arcane filesystem ever. I've tried learning it but it looks like a million obscure gotchas just waiting to fuck up your shit. Someone will post an error and people are like "ha ha you fucker yeah you didn't shprutzle your cache vdevs sideways. Noob mistake. Oh your data is ruined btw".

I mean look at that shit, "scrub repaired 0b with 5 errors". I feel like Richmond looking at the server lights in IT Crowd. "Is it good that it's doing that? Is it bad? I feel like I should be telling someone."

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 3 points 3 days ago

haha, no, depending on your use case zfs is mostly set and forget, you just have to get a set of sane defaults, which, at least for me, is way easier and saner than stuff like btrfs. Arch wiki is a good source for that.

scrub is a command that reads data from disks and verifies checksums, it can automatically heal the corrupt data if some redundancy is available. 0 bytes repaired means nothing was repaired, 5 errors mean that there are 5 instances where the recalculated checksums of data did not match the expected values that were calculated when initially written. Normally the command would output 'scrub repaired 0B with 0 errors', that's how you know that the pool is healthy.

shprutzling the cache vdevs sideways is generally required only for obscure optimizations, I've never seen a case where a wrong setting would lead to data loss.

The gotchas come from a single thing: the stupid CDDL licence, which is not compatible with GPL. That means that zfs is not included in linux kernel and most of the normal linux userspace filesystem stuff do not work with zfs.

Also, keep in mind that zfs is generally almost impossible to fuck up, what I've done here is a result of extreme dedication and like 8 years of experience.

[-] ggtdbz@lemmy.dbzer0.com 3 points 3 days ago

Just so I understand, since I was looking into using ZFS for my storage before hard drives went up by like 400%: you set up things in an obtuse way and you got a failure that can’t be recovered from?

I somewhat wrapped my head around how ZFS data storage itself works, and was feeling relatively confident that it’s what I’d use.

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 2 points 3 days ago

the problem here does not originate in zfs, this is a hardware failure due to a poorly supported SoC (vendor says PCIe works 'normally' since kernel 7.1, this system ran 6.19) and bad electricity supply

[-] sttal@lemmy.world 2 points 3 days ago

i've had my curse zfs moment in the past even though had it in a raid z2. Later i've found out that was an incompatibility from the controller firmware version and the backplane of disks.

What i took out of the situation is that if zfs broke in my case any other filesystem would have broken 10 times over.

Fixed the firmware redid the raid and it has been working for 4+ years without a flaw.

[-] ruuster13@lemmy.zip 15 points 3 days ago

Mirror vdevs are the only way to go. Accept the 50% and sleep at night.

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 8 points 3 days ago

that's no fun. also, it turned out that this is not a disk error, the zfs metadata got corrupted and now I get a panic when trying to mount this.

[-] possiblylinux127@lemmy.zip 8 points 3 days ago

ZFS is very thoughtful and paranoid so it shouldn't just corrupt. You probably have a silent hardware failure

[-] CompactFlax@discuss.tchncs.de 10 points 3 days ago* (last edited 3 days ago)

I’ve had this, with a single disk pool. An SSD.

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 19 points 3 days ago* (last edited 3 days ago)

my achievement here is that this is a proper raidz setup with multiple disks and ecc ram

[-] j3tt@lemmy.world 6 points 3 days ago

oof

[-] BakedCatboy@lemmy.ml 4 points 3 days ago* (last edited 3 days ago)

Raidz1 or z2? One of the reasons I went with z2 is because of the "raid5 is dead" thing, argument basically being, drives are so large that if 1 fails, you are statistically almost guaranteed to encounter a read error during the rebuild when you no longer have any redundancy, so double redundancy is a necessity nowadays.

[-] Damage@feddit.it 1 points 3 days ago

I have a raidz5 array of 8tb drives, one died, replaced it with no issues

[-] BakedCatboy@lemmy.ml 2 points 3 days ago* (last edited 3 days ago)

Raidz5 doesn't exist, I assume you mean raid5? And you wouldn't know, URE is sometimes silent, and because without parity during a rebuild you would be recalculating the parity bit and assumig every read on the remaining drives is successful. Except the URE rate for drives hasn't gone down to match the increasing size of drives - so for a full 1x read of a large drive, you're statistically at risk to get at least 1 URE.

In short, there is no integrity checking when rebuilding a single redundancy array after a drive failure, so just because a rebuild succeeds, doesn't mean nothing was lost or corrupted. With double redundancy, you will still have single redundancy during a rebuild, so you would be able to detect URE and retry reads or raise an error to the user.

Some articles may make it overblown - you won't lose your whole array, but any files stored in the block that had the URE during rebuild would be lost. And you might not know that it happened.

You can read about it here: https://www.servnetuk.com/insights/ure-is-raid-5-dead

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 2 points 3 days ago

I've seen a calculation somewhere for drive speed, size and scrub frequency where optimal number of extra drives was calculated. On fast storage and with weekly scrubs one extra drive should be mostly fine.

My philosophy is any raid is not really a data contingency solution - it just saves some time when the drive fails.

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 1 points 3 days ago* (last edited 3 days ago)

This is raidz1. Three 4TB NVME drives. I've built this machine as the primary server for services like immich as a fun project that enables me to say that my photo service runs faster than all the commercial stuff and is on ARM64. The downside is that the SoC is not supported by upstream very well so vendor patches had to be applied and there was some instability between that and my shitty electricity supply.

[-] ryannathans@aussie.zone 3 points 3 days ago

Copies = 2 might have saved you

[-] inari@piefed.zip 2 points 3 days ago

Unrelated, but how did GrapheneOS ruin your phone?

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 5 points 3 days ago

do you ever get a new shiny thing and it looks like it works very nice and coherently and then you dig a bit deeper and realize that it is just a bunch of semi-broken shit held together by glue and duct tape? that's the state of android, that is nicely obscured on pixels by google customizations

[-] inari@piefed.zip 4 points 3 days ago

Sounds like most software

[-] ellen@discuss.tchncs.de 2 points 3 days ago

Trying to import my pool causes a panic, so maybe? Though am still in the middle of trying to recover, just haven’t had time to work on it yet.

[-] GrapheneOSRuinedMyPixel@sh.itjust.works 1 points 3 days ago

I feel very sorry for you. I have the same problem (panic on mount) with this one, but at least it works when mounting read-only. I've been moving data with zfs send the entire day

this post was submitted on 01 Sep 2026
106 points (97.3% liked)

linuxmemes

32686 readers
41 users here now

Hint: :q!

Sister communities:

Community rules (click to expand)

1. Follow the site-wide rules

2. Be civil
  • Understand the difference between a joke and an insult.
  • Do not harrass or attack users for any reason. This includes using blanket terms, like "every user of thing".
  • Don't get baited into back-and-forth insults. We are not animals.
  • Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
  • Bigotry of any kind will not be tolerated. This is an LGBTQ+-friendly community -- if that is a problem for you, you should leave.
    • 3. Post Linux-related content
  • Including Unix and BSD.
  • Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of sudo in Windows.
  • No porn, no politics, no trolling or ragebaiting.
  • Don't come looking for advice, this is not the right community.
    • 4. No recent reposts
  • Everybody uses Arch btw, can't quit Vim, <loves / tolerates / hates> systemd, and wants to interject for a moment. You can stop now.
    • 5. 🇬🇧 Language/язык/Sprache
  • This is primarily an English-speaking community. 🇬🇧🇦🇺🇺🇸
  • Comments written in other languages are allowed.
  • The substance of a post should be comprehensible for people who only speak English.
  • Titles and post bodies written in other languages will be allowed, but only as long as the above rule is observed.
    • 6. (NEW!) Regarding public figuresWe all have our opinions, and certain public figures can be divisive. Keep in mind that this is a community for memes and light-hearted fun, not for airing grievances or leveling accusations.
  • Keep discussions polite and free of disparagement.
  • We are never in possession of all of the facts. Defamatory comments will not be tolerated.
  • Discussions that get too heated will be locked and offending comments removed.
    •  

    Please report posts and comments that break these rules!

    Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't remove France.

    founded 3 years ago
    MODERATORS
    poopsmith@lemmy.world
    zephyr@lemmy.world
    rtxn@lemmy.world