11
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 02 Nov 2026
11 points (82.4% liked)
LocalLLaMA
5579 readers
127 users here now
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
founded 3 years ago
MODERATORS
It's a two pronged attack for sure. Make local expensive, make cloud worse.
I think there's still some wiggle room to be had in the 8 to 16GB VRAM range...so I'm hoping Qwen 4 series brings its A game.
Qwen 125bb flash next is probably one of the best models I can run (strix halo), it’s fast and smart. But it burns too many tokens on reasoning even on low effort.
If they do a qwen4 125b a6b or whatever I will be happy. No idea if that’s what their qwen4 flash will be or not.
A Strix. In 2026. Must be nice :)
It is, however it’s still pretty slow and not many models find the sweet spot that Qwen did.
I feel most models would be better on a faster GPU as MoE sizes are too big right now. I can get comparable tok/s on large models compared to discrete GPUs but much slower on small sizes.
The curse and blessing of unified memory!
I was considering getting a BC-250 (a cut down PS5) with 16GB unified memory, but at 400GB/s bandwidth.
When they were cheap, it almost made sense. But realistically, I can get similar speeds with a MoE thanks to cpu and system ram, and 16GB is just not enough for anything top shelf (no, Prism MLs Bonsai models don't count. I don't care what the benchmarks say - they're lobotomised).
Still, given the chance to get a Strix on the cheap, I'd go for it. There's something very nice about a low power box that can run a 120B model at speed.
I am definitely happy with it.
One of the best things is I run some local services and I can have embedding models, decision models, and small LLMs always in memory for my hosted apps. Then it’s only a few seconds to load up a large LLM like qwen 125b or similar.
Kinda wish I’d gone in one those BC250s as well, there are good guides on them now. 400gb/s is pretty good.
Oh, your running some JEVs? How are they? I just can't figure out a use for em.
Yes, concurrency is amazing. I can pull some tricks with CPU (OCR, TinyBERT, Klein etc) while the GPU runs inference...but it's not a Strix.
Yeah, the BC-250 had / has real promise, esp with the core unlocks...but like P40s and P100s (server cards) people twigged on "this is cheap and fast". I can't get one locally for under $600 now. They use to be $150. By the time you fit it out properly, it's nearly $1K. Not worth it.
I’m running liquid d1 and clef.
Clef is good, but like 30gb. I mostly use d1 as a reranker type model for search and it’s good for that. I haven’t found many reliable use cases yet for them.
Meanwhile you made me look at prices. JFC.
Yeah, I could sell it for well over twice what I paid.
It’s insane.
I just don't get it. Like...who's buying these things in quantities vast enough to charge that much? Oh, I've heard Open AI snapped them and Mac Minis up in bulk for "testing" (whatever that means)... but given the cost now... why / who is buying them in quantities that warrant these sort of prices? It's not as if they're rare per se. Hell, there might even be a glut relative to actual demand. Yet, those prices...
Human psychology is weird.
I think it’s just the RAM prices? If you’re a reseller or training professionally renting or buying a b100 or something is way more powerful (but 100x the cost)