Technology

You can't right now because there is no dataset available for fine tuning. I'm just saying that it's possible to fine tune 8B-35B parameter model in int4 that will outperform those models ex. for single programming language and developer specific problems. You can read example of fine tuned 8B model here https://fermisense.com/when-machines-take-the-wheel/

I run Qwen 3.6 27B quantized down to Q4_K_M or even a variant Q5_K_S on my 8gb VRAM entry level AMD GPU, with support of my CPU and 32gb system RAM. Yes, I also limit the Context Length heavily to something like 18k. It's slow. But the point is, you don't need necessarily 16gb VRAM.

But it's better to use faster models for this type of hardware anyway. The MoE type of models (such as 26B A4B, or 35B A3B) are vastly, vastly faster for normal usage, but they are a bit worse in some cases. Also Googles QAT trained model versions also have less RAM requirements without losing much quality.

I'm just saying that, so others are not discouraged too much. It's not the same experience as the full version off course, but you can use them with some tricks on weak hardware too.

You don't want to run these models in RAM. I started using them on an RTX 3060-12Gb VRAM, and quickly added another, for 24Gb RAM. I also have 64 Gb RAM. Now it works well. Not lightning fast, but quite useable. VRAM is the key.

Sure they are. Especially with moores law dead. And especially since 2020; the more million dollars behind it, the more they run like one from a inexperienced single-dev on itch.

This is a pretty good idea I hadn't thought about. The whole wikipedia is about 37Gb.

I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.

This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.

I bought two RTX 3060 with 12 Gb each for a total of 24Gb VRAM. Sionce the cards are not top shelf anymore for gaming, they can be had used in the 200€ range, so my setup was 400€ over the cost of my existing PC. My PC, while not last gen was already pretty beefy, with a Ryzen 9 and 64 Gb RAM. I think 400€ to have a pretty good local AI setup is not bad at all.

I can run both the 27B and 35B version of Qwen3.6 on my 10GB VRAM 3080, and they run ok. You don't need to put the entire model in VRAM, even if it's probably beneficial.

I mean that's high but not unreasonable, especially with unified ram machines