Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil.
-
No spam.
-
Posts are to be related to self-hosting.
-
Don't duplicate the full text of your blog or readme if you're providing a link.
-
Submission headline should match the article title.
-
No trolling.
-
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
-
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
view the rest of the comments
[AIT] Well, let's see.....nothing really new on the server. Still waiting for something interesting besides *.arr. I did provision a new tower with the following specs:
This, obviously, will be an local AI machine
Nice. Did you have to sell a house to afford the parts?
LOL Not quite, however, I am choking on the 5090 but as I understand, that's the better option, and 32 GB seems to be the sweetspot for AI. I actually picked up the tower and original RAM for less than $500 USD.
First of all, I mean zero offense with any purchase decision. A 5090 is very good.
…But if I were paying that kind of money, I’d probably get a 4090 and a new motherboard/CPU instead. Maybe a used DRR4 threadripper system.
Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models, whereas on an 5090 you are stuck with Qwen 27B.
Having a fast CPU, with lots of RAM channels, with full PCIe bandwidth is much more important for that than having a 5090, where a 3090 or 4090 will get the job done.
Even if pure speed is your primary concern, you can tune a sparse 120B model (like Laguna) to run almost as fast as Qwen 27B on a 5090, and get at-least-good results.
It’s more finicky and involved, though. For sure.
Running an LLM on a 5090 is a task. Hybrid CPU + GPU inference is a hobby.
Why you do this to me? LOL Injecting your options. But seriously thank you for the advice. I'm really green in the AI arena. So I'm trying to feel my way around, trying not to spend money on equipment I'll regret later.
Be aware that you pay a significant performance penalty for going over occulink.
The hit modest for fully offloaded dense models (like Qwen 27B), but dramatic for hybrid inference of big MoEs.
Even my old 3090 got a noticeable performance gain going from a PCIe 3.0 x16 riser to a PCIe 4.0 one.
And that's why it's tentative. The reason I'm leaning towards the dock is that, while the 5090 will fit, physically in the case, I am worried about heat dissipation. Even now, when I fire up a moderate model and run it through some paces, I can see the temps rising fairly quickly. So, if I'm going to drop that kind of cabbage on a 5090, the one thing I don't want to do is watch it turn into a shrink-i-dink.
Duct it!
I have a 400W 3090 with zero case fans.
Its always cool, because I bought like $10 of weather stripping to "seal" its intakes against the edge of an SFF case. So it's always sucking in ambient air:
You can also severely undervolt a 5090 to like 250-300W, with very little performance impact. Honestly its stock speeds are kind of crazy.
...But again, you aren't gaining much over a 4090. A 4090 + DDR4 threadripper would be way faster than a 5090 + occulink, for many reasons. And TR CPUs are pretty reasonably priced compared to GPUs these days.
But even if you do go 5090, I'd highly recommend finding some way to shove it in the case and duct some air into its intakes. Its going to be way faster on a PCIe slot.
You could even get a riser and mount it somewhere else in the case, theoretically.
Shit I deleted my post. Yes I have been enjoying some greenery so you just have to hear me out. I have an old countertop ice maker that stripped it's gears. The refrigeration loop is fine. Pretty wild how small it is. I was wondering.....what if I provisioned some brackets on which to mount a muffin fan of unknown CFM, to blow across the coils thru several sheets of mesh of varying gauges to collect moisture drained off to an unknown location, and duct that indirectly into the computer, like a miniature Liebert server room cooling unit. I worry about the moisture tho. Constructing a desiccant wheel seems improbable. I could reheat the cool air upstream or mix warmer air in with the stream to keep the so the outlet air doesn’t stay at a very low dew point and “rewet” downstream components. Perhaps with a small filament heat coil. HMMmmmmm
Shit I deleted my post. Yes I have been enjoying some greenery so you just have to hear me out. I have an old countertop ice maker that stripped it's gears. The refrigeration loop is fine. Pretty wild how small it is. I was wondering.....what if I provisioned some brackets on which to mount a muffin fan of unknown CFM, to blow across the coils thru several sheets of mesh of varying gauges to collect moisture drained off to an unknown location, and duct that indirectly into the computer, like a miniature Liebert server room cooling unit. I worry about the moisture tho. Constructing a desiccant wheel seems improbable. I could reheat the cool air upstream or mix warmer air in with the stream to so the outlet air doesn’t stay at a very low dew point and “rewet” downstream components. Perhaps with a small filament heat coil. HMMmmmmm
Unfortunately condensation is, indeed, an intractable issue there.
Sometimes people put a heat pump in water cooling loops, where condensation along the pipe can be mitigated with insulation, but even that’s a little risky with a 5090 these days.
Enjoy your lovely greenery!