kuvwert

joined 1 year ago
[–] kuvwert@lemmy.dbzer0.com 7 points 2 weeks ago (1 children)

His fingers are interesting .... ....

[–] kuvwert@lemmy.dbzer0.com 6 points 5 months ago (1 children)

Holy Canoli You were right.

Checked the Synology's network interface page and it was negotiating at 100 Mbps to my Orbi router. Your comment about 11.4 MB/s lining up with a 100 Mbps ceiling is exactly what got me to actually check the link speed instead of just assuming gigabit.

Bought a cheap unmanaged switch, plugged the NAS and server into it directly, came up at 1000 Mbps, and now I'm getting 107-115 MB/s on the same tests. Updated the post and credited you. Appreciate you pushing back on it.

[–] kuvwert@lemmy.dbzer0.com 6 points 5 months ago (5 children)

You could 1000% percent be right here. I don't have an education on networking or system ops.

The NAS can sustain gigabit writes without complaint. My premis is the kernel isn't flushing at wire speed in a steady stream. It hoards 2+ GB in page cache then fires a burst of concurrent NFS RPCs all at once. The NAS can't ack them fast enough and the client declares it dead.

I ran three dd tests, same 2GB file, same NAS, only changed client config:

  • 128K buffers: 101 MB/s, dies at 2GB
  • 32K buffers: 2.1 GB/s (all page cache, nothing hitting the wire), dies even harder
  • 64MB dirty page cap: steady 11.4 MB/s, zero timeouts, clean finish

The NAS handled all 2GB in test 3 because it arrived as a drip instead of a wall.

At least thats the working theory :)

57
submitted 5 months ago* (last edited 5 months ago) by kuvwert@lemmy.dbzer0.com to c/selfhosted@lemmy.world
 

Double Check Your NFS timeouts to your NAS arent an NFS problem. They might be a dirty page writeback problem.

I'm really Sorry in advance for the wall of text here. I debated trimming this down but honestly the whole reason I spent months stuck on this is because nothing about it was obvious. The symptoms point you at NFS, your mount options, your network, everything except whats actually wrong. And because the defaults that cause it ship with basically every linux distro, Id bet money theres a ton of people out there with the same problem right now just blaming thier NAS or Jellyfin or whatever. For all I know this is common knowledge and I'm just the last person to figure it out, but on the off chance somebody else is out there googling the same NFS timeout errors I was, heres the full story. (TL;DR Below)

Ive been chasing NFS issues on my Proxmox cluster for months now and I finally found the actual cause, and it wasnt anything Id seen anyone talk about online. Figured Id write it up because I guarantee other people are hitting this exact same wall.

The setup: half a dozen VMs on Proxmox, all mounting a Synology NAS over NFS. Jellyfin, Audiobookshelf, Sonarr, Radarr, the usual self-hosted media stack. Things would work fine for a while and then randomly go sideways. Jellyfin stops mid-playback. Audiobookshelf loses track of where you were. Sonarr tries to import a downloaded episode and the entire container locks up. dmesg fills with nfs: server 192.168.1.50 not responding, timed out and youre rebooting things again.

The part that kept me going in circles for so long is that it was never consistent. An audiobook would stream for hours without a hiccup, but then Sonarr would try to move a 4GB episode file and the whole mount would go down. I could ls the mount and browse around just fine even while Sonarr was hung. Small file operations worked. Large writes didnt. But not always, Sometimes a big import would go through without a problem, and Id convince myself whatever Id just changed in my mount options had fixed it.

I went through all the usual advice. Switched from NFSv4 to NFSv3, which I was especially convinced was the fix because the timing lined up with when Id been experimenting with v4. It wasnt. I toggled nolock, tuned rsize and wsize down from 128K to 32K, tried soft vs hard mounts, checked the Synologys HDD hibernation settings, disabled TCP offloading on the virtio NIC. Nothing actually fixed it. Every time I thought I had it, the next import that was over the threshold would fail and i would scream.

Then at one point I gave a couple of the VMs more RAM, thinking the media workloads could use the headroom. Everything got worse after that. Like, measurably worse. I didnt connect the two at the time.

What finally cracked it was running a dd test to write a 2GB file to the NFS mount and actually watching the numbers. With the 32K buffer mount options, the write reported 2.1 GB/s. On a gigabit link. Obviously that data is not going to the NAS. The kernel was eating the entire write into the VMs page cache, saying "yep, done!" and then trying to flush 2+ GB of dirty pages to the Synology all at once. The NAS gets hit with a wall of data it cant process fast enough, NFS RPC calls start timing out, and everything goes to hell.

The default value for vm.dirty_ratio is 20, meaning the kernel will let 20% of your RAM fill up with dirty pages before it forces a writeback. On my 13GB VM thats 2.6GB of buffered writes. So the kernel would happily sit there absorbing data into RAM, and then try to shove 2.6 gigs down a gigabit pipe to the NAS all at once. And when I "upgraded" VMs with more RAM, I was literally raising the ceiling on how big that buffer could get. Thats why things got worse. The inconsistency made sense too. A 700MB file might stay under the background flush threshold and trickle out fine. A 4GB season pack would blow past it and trigger the whole mess.

The fix

Two sysctl values:

sysctl -w vm.dirty_bytes=67108864
sysctl -w vm.dirty_background_bytes=33554432

This caps the dirty page buffer at 64MB and starts background writeback at 32MB. Instead of hoarding gigabytes and flushing all at once, the kernel now pushes data out to the NAS continuously in small batches. Make it persistent:

# For distros using /etc/sysctl.d/ (Debian 12+, Ubuntu, etc.)
echo -e 'vm.dirty_bytes=67108864\nvm.dirty_background_bytes=33554432' > /etc/sysctl.d/99-nfs-dirty-pages.conf
sysctl -p /etc/sysctl.d/99-nfs-dirty-pages.conf

# For distros using /etc/sysctl.conf
echo 'vm.dirty_bytes=67108864' >> /etc/sysctl.conf
echo 'vm.dirty_background_bytes=33554432' >> /etc/sysctl.conf

Before: 2GB dd writes at 101 MB/s, dies at the 2GB mark with NFS timeouts and I/O errors. After: same test, steady 11.4 MB/s start to finish, zero NFS timeouts, completes cleanly. OK oK Yeah, the throughput number is lower, but Ill take a transfer that actually finishes over one that crashes every time.

I applied this across all six of my VMs that mount the NAS and the whole fleet has been stable since. Theyd all been independently building up multi-gigabyte write backlogs and dumping them onto the Synology simultanously. I was basically DDoSing my own nas from six directions every time anything tried to write a big file.

Then I checked the Proxmox host itself. 128GB of RAM. Four NFS mounts to the same Synology, including the one Proxmox writes VM backups to. All hard mounts with default dirty ratio. Thats a 25GB dirty page ceiling on the hypervisor. Every scheduled backup was potentially building up a 25 gigabyte write buffer and then hosing the NAS with it in one shot. And because the mounts were hard, if the Synology choked during the flush, the hypervisor itself would hang, not just a VM. I dont even want to think about how many weird backup failures and unexplained freezes this was behind.

Since applying the fix Ive also noticed that Jellyfin library scans are completing reliably now. They used to hang constantly and Id just accepted that as normal Jellyfin-over-NFS jank. The scans were generating thumbnails and writing metadata, building up dirty pages, and triggering the same flush that would take down the mount mid-scan. Audiobookshelf was doing the same thing. It would scan libraries and randomly lose connection to the mounted paths. That one was harder to pin down because audiobook files and cover art are small enough that the writes wouldnt always push past the threshold on their own. But if another VM had already half-filled the NASs tolerance with its own flush, Audiobookshelf tipping it over would be enough. Same underlying bug in every case, and I spent months blaming three different applications for it.

If youre running a media stack on VMs with NFS mounts to a NAS and youve been tearing your hair out over random timeouts, check your vm.dirty_ratio and do the math against your RAM. Bet you its higher than you think.

TLDR; If your NFS mounts to a NAS randomly time out during large writes, your VMs are probably buffering gigabytes of dirty pages in RAM and then flushing them all at once, overwhelming the nas. Symptoms in my case were Jellyfin stopping mid-playback and hanging during library scans, Audiobookshelf losing connection to mounted paths and forgetting playback position, and Sonarr/Radarr locking up completely when trying to import episodes. Set vm.dirty_bytes=67108864 and vm.dirty_background_bytes=33554432 on every VM (and the hypervisor) to cap the buffer at 64MB and force continuous small writebacks instead.


Edit 1: @deadcade pointed out that 11.4 MB/s is suspiciously close to a 100 Mbps link ceiling and they were right. Checked the NAS LAN1 network status and it was negotiating at 100 Mbps... The NAS was plugged into my router which has gigabit ports but was apparently negotiating down due to what i must assume is an issue with the router.

SO the real solution: I went to Bestbuy and grabbed a $20 gigabit switch, plugged the NAS and Proxmox host into it directly, and the Synology came up at 1000 Mbps immediately. Same 2GB dd test now completes at 107 MB/s from the host and 115 MB/s from the VM, no timeouts, totally clean.

So if i actually understand wtf is going on here... it was actually two problems stacked on top of each other this entire time.

The 100 Mbps link was the speed ceiling between the router and the NAS. The dirty page defaults were what turned that speed limitation into a catastrophic failure. The kernel would buffer gigabytes of writes and then try to flush them through a 100 Mbps pipe where the NFS RPCs would time out long before the data finished arriving. The sysctl fix worked because it accidentally rate-limited the client to roughly what the 100 Mbps link could handle. Fixing the link speed solved the actual bottleneck.

THANKS for the insight deadcade!

Both fixes stay though. 64MB dirty page cap on a gigabit link still saturates the connection at 115 MB/s and there's no reason to let a 128GB Proxmox host build up a 25GB write buffer aimed at a consumer NAS. Also check your link speeds.

Edit 2: Thanks again to everyone who chimed in with your fantastic insights and ideas.

[–] kuvwert@lemmy.dbzer0.com 1 points 6 months ago

I would love this. Have several devices sitting waiting for me to figure out how to hack them

[–] kuvwert@lemmy.dbzer0.com 13 points 7 months ago

Ah yes! I saw your mum in poundland last night

(I'm so sorry, it was right there)

[–] kuvwert@lemmy.dbzer0.com 1 points 7 months ago* (last edited 7 months ago)

I need to address the condescension in your response. I didn't approach this with an agenda to disprove you. I shared what I understood and acknowledged uncertainty where it existed. That's how honest discussions work. Your assumption that this is my "very first time looking into it" and that I need to "allow my position to evolve" is patronizing and unwarranted.

On the substance: Invasive classification: The fact that ecologists are divided on this is exactly my point about nuance. You cannot simply declare something settled because "a large number of sources" say so when the scientific community itself is debating it. The article I cited explicitly states experts disagree.

Buzz pollination: You moved the goalposts. Your original claim was that honey bees "cannot effectively pollinate native plants" full stop. Now you're talking about buzz pollination specifically. Yes, honey bees cannot buzz pollinate. But many native North American plants do not require buzz pollination and are effectively pollinated by honey bees. Tomatoes, blueberries, and cranberries need buzz pollination. Sunflowers, asters, goldenrod, and countless other natives do not. Your broad claim was incorrect.

Disease transmission: You completely missed my point. If the problem is industrial beekeeping practices creating disease reservoirs, then those practices are the problem. Colony collapse does not discriminate between well-managed hives and factory farm operations. It kills bees indiscriminately. Celebrating it as a solution is like celebrating a disease outbreak in factory farms instead of advocating for better practices.

Colony collapse as good: This is where your argument fully breaks down. Colony collapse disorder causes immense suffering to the bees themselves. If your concern is ecological harm, then advocate for reduced hive density, better management, or restrictions on commercial beekeeping in sensitive areas (which already exist in many places, as you noted with the USDA guidance). Celebrating the mass death of millions of bees as "a good thing" because it might inconvenience their owners is callous and doesn't actually address the ecological concerns you claim to care about. Beekeepers respond to colony collapse by importing more bees and intensifying their practices, not by scaling back operations. Your comparison to dairy farms going out of business is false equivalence. A business closing is a policy outcome. Colony collapse is an ecological disaster that happens TO the bees, causes them suffering, and does not reduce the overall population of managed hives because beekeepers simply replace losses.

I am genuinely interested in native pollinator conservation. But your position requires celebrating bee suffering as ecologically beneficial when the evidence does not support that conclusion, and better solutions exist.

[–] kuvwert@lemmy.dbzer0.com 14 points 7 months ago (5 children)

Its much more nuanced than that. Honey Bees are not native, but that does not technically make them invasive by most definitions. Oversaturation on a local ecosystem can push out native bees in some cases (maybe)...

https://www.technologynetworks.com/applied-sciences/news/native-north-american-bees-mostly-seem-untroubled-by-invasive-honey-bees-391892

As for the claim about them not being effective pollinators... Ive not come across anything that would make me believe that yet. In fact my understand was that its specifically because theyre good pollinators that they outcompete native species. Without additional information those two statements are incompatible.

They can spread diseases, but my understanding is that this is a result of the conditions the artificial colonies are kept in, not attributed to their inherent nature or biology, and might happen to any species that is subjected to those environments.

"Colony collapses are a good thing" does not pass the smell test in any capacity and I would disregard that opinion without some significant evidence to back it up.

[–] kuvwert@lemmy.dbzer0.com 4 points 9 months ago

Cool! I'm gonna try it out. Thanks!

[–] kuvwert@lemmy.dbzer0.com 4 points 11 months ago

Could you share your flow for that? Its sounds awesome

[–] kuvwert@lemmy.dbzer0.com 3 points 1 year ago

1/2 - where is 2?

[–] kuvwert@lemmy.dbzer0.com 3 points 1 year ago

Ive had some good experience with filestash but recently they swapped to collabora which brought me a host of headaches.

The Dev is awesome though.

[–] kuvwert@lemmy.dbzer0.com 2 points 1 year ago (1 children)

I doubt it, why dont you have sudo?

view more: next ›