Local models are meaningfully useful* for agentic work now. This is a realization I came to over the last day or so.
I wrote a lot of custom mods for a game that recently switched engines. The migration is going rather smooth for most people because of wise architectural choices from the dev early on, and because they prioritized backwards compatibility in the update. But there are still some rough edges, especially if you had your hooks deep into the engine.
I recently vibeforked Unbiased Bob's fork of Strata to make it compatible, easy to use and optimized for my OS and hardware (NixOS 5800x3d 64gb, 7900xtx 24gb). Sol 6.1 got it working with Qwen 3.8 Flash Next IQ3 XSS (~75GB) at roughly 46tps 164k ctx with enough headroom to launch a heavy browser along with a triple-A game set to low in 720p for testing when I need to (plus space for zram swap)
I set it up in deepseek harness, gave it a 14MB error log in the game's working directory, then went to sleep. I woke up and it had run for 3 hours. The game launched with only two errors, which it quickly fixed from the new, vastly smaller logfile.
It hammered down every issue, making changes across 30 separate mod repos. Practically a one-shot, and all the features worked as originally intended.
I left a local coding agent running completely on its own for three hours across multiple repos across a dozen auto-compactions with a log file and no ability for it to verify its work besides a compiler. And it produced meaningful results. I can't speak to the code quality, but it worked. This was completely unthinkable even just two years ago. Even one year ago, the cloud frontier wouldn't be able to do this on its own, and now a local model can, running on my consumer gaming hardware from one or two gens back.
“Huh... So... We're here now..." I said to myself, with a bit of stunned disbelief, wondering what the future one or two years from now might hold.
Ornith is a nice fine-tune of Qwen that works reasonably well on reasonable hardware, Bonsai also seems to be often recommended but I haven't tried it personally. That said, from everything I've seen local agentic code is pretty computationally wasteful in many cases, it iterates heavily and confuses itself a lot and self-corrects often repeatedly. It hurts my soul to imagine the energy waste and environmentally damage if large numbers of people started using this routinely and carelessly.
It is meaningfully useful, I'll grant you that, and when paired with highly specific and well-bounded tasks it performs quite well. I hate the idea of relying on it too heavily though. If you just throw it at things blind and without you also providing assistance to the model, it will get the job done, eventually, just in a really painful and stupid way. It's not Claude, but for a locally-running substitute it's remarkably good, it handles coding assistance very effectively, but it's a very poor coder when operating completely independently. I find the best and most efficient use is to ask it for suggestions, keeping the pieces as small as reasonably possible, and then implement them myself based on what it suggests (either adjusting the model's edits or copy-pasting the model's suggestions). It's very good at locating the relevant code and it usually suggests a reasonable design, which I typically tweak based on my own experience. Maybe that makes me old-school, maybe I'm a luddite, but that's what my experience tells me to do. If you don't have any experience, I guess you're on your own, good luck with the models. They're decent, but they're not brilliant programmers (granted neither am I most days). That's my opinion and I'm sticking with it until I see evidence to suggest otherwise.
It doesn't run my GPU any harder than watching anime with my realtime upscalers and my CPU not much more playing lategame modded Minecraft. And I live in a region with very cheap energy with lots of renewables and nuclear. So if it can do work, I would otherwise have to do, for fractions of pennies while I sleep: it's totally worth it.
It takes electricity. I take moments of a finite life. This is the same reason I didn't review the code; for this case, I just wanted the result, I just wanted it to work, and I was curious how far a local AI would get. I didn't actually expect it to finish. I expected to come back to it looping or a complete mess and revert all my repos and then tell Sol to do it in 30 minutes.
They are far less token efficient, and dumber, and far slower than Frontier models running on dedicated hardware. It hallucinates and get things wrong all the time, but it seems to be able to correct itself and get back on track from new information much better than earlier generations of model. That really seems to be the key to its long-running success from what I've seen. When I tried this about half a year ago, once it started hallucinating or its context got over ~50k, it was basically game over.
My background is in low-level resource-constrained systems programming, mostly C and modern C++. Yeah, I can code laps around this thing even in languages I don't know well (not true for frontier) and I'd never trust it to write code that I would own and put my name to... But it can get results. It could do something low-stakes for me, while I slept, that was literally impossible a year ago. And it does it locally. That was mindblowing. I thought this approach (LLMs) would plateau a lot sooner than this. The fact that it can run outside of a datacenter is just a massive cherry on top.
As long as you don't publish it. Because then you (should) also have to take responsibility.
Well, I think a caveat to that is publishing source (no prebuilt release) as unvetted slop, so everybody knows. Though in this case, it was strictly for personal use.
But yes. Anything I release proper, for general consumption, is owned by me. And professionally, I never own even a line of code (or comment) that I don't understand or wouldn't have written myself. I may ask for a bunch of audits and slop tests for insights, but I'll keep those offline as "extra" unless it's something on par with I would have written. I'll never hallucinate important code, and anything i get paid for or contribute to OSS is important. Personal game mods and small utilities, not so much.
Good human.
Given that (personally affordable, ingnoring the bubble premium) hardware progress is saturating, how much headspace for improvement do we still have do you think? Would you currently put it at linear-ish, saturating, or too early to tell/not enough data?
You know, I can absolutely see a future where Nvidia and the other hardware makers turn their eyes back towards consumers as they see the build-out slowing and the upgrade budgets shrinking.
What if in one year, Nvidia puts out a GPU or all in one PC AI server thing that will let you run free models that are as good or better than current frontier ones, at a reasonable speed, and they charge like $3000 or $5000 for the thing?
Well, I guess if they want to be proper capitalists/enshittifiers, they would first sell them for 10x as much to corporate IT departments who can replace their ridiculous tech giant subscriptions with local compute augmentation. On-prem is back, baby!
Then they can rip off the gamers and the home lab enthusiasts for a while.
And finally, at long last, with the economy raked over the coals and a cool trillion in the bank, Jensen will briefly feel whole inside. Surely he will.
Jen-Hsun has certainly sold a lot of golden shovels, but he will find gamer gear market to be slim pickings in comparison to the fat enterprise checks. What I'm hoping is that some older enterprise gear will make it to ebay and such. Maybe some smaller AMD Instincts with HBM.
It's really hard to tell because I didn't even see it getting to this point. I'm running a 75 gigabyte model on a 24 gigabyte GPU and my friend with a 16 gigabyte GPU is actually getting better performance than me. Mostly just because it's a mixture of experts and Strata's storage tier caching works well.
I don't think it's saturated because we would have to, not quite saturate, but definitely wring out a lot of the gains from kernel and model architecture first before raw compute becomes a limitation.
I expected this approach to AI to plateau before now. There has also been evidence that scale is not the sole key, and that quality and approach and harness all play significant roles. I thought it would have died by now, but since it hasn't, I suspect it still has a long way to go. Given that I couldn't have predicted this a year ago, I don't think I can predict a year from now.
Thanks, food for thought.
The gap between big-boy models and local fare has been closing. That could mean scale is not necessary for further improvement, or it could mean we're asymptotically approaching some limit to this approach, but either way - existing consumer hardware is sufficient for some witchcraft.
Waiting for the bubble to pop to join the fray. Adding some surplus PV capacity meanwhile.