I felt sorry for it and stopped it, 14min and the poor thing was just going around in circles.
I'm just looking at running some local llm on my m2 16GB mac and this was with the default settings for the qwen3.5 9b in LM Studio. Any pointers to get the best out of a M2?
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
Try Ternary Bonsai. It's a heavily quantized version of Qwen 3.8 27b. It won't do wonders, but it shoud be better: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev The page says you need a fork of llama.cpp, but the support was included in the latest release (0.7). You can read more about the mode on the main model page: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
Oh god no. Bonsai is such a waste of time! Totally a dull version of Qwen. Keeps getting basic things wrong and isn’t agentic at all.
But I’d be curious to hear if you’ve used it and had a different experience.
I personally run Qwen 3.8 27B at Q4_K_XL, but given OP's system requirements, I'd try Bonsai instead of a regular Q2 quant or the older Qwen 3.5 9B.
M2 16 GB… ah… let’s see how Bonsai works for OP…
Cries in old 1070 architecture
I had already downloaded a Bonsai 27B, just not really tested it yet
You can also try https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
It's more demanding, but also much capable.
I ran this for a good while, quite usable, like gipity 4o or 4.5