I felt sorry for it and stopped it, 14min and the poor thing was just going around in circles.
I'm just looking at running some local llm on my m2 16GB mac and this was with the default settings for the qwen3.5 9b in LM Studio. Any pointers to get the best out of a M2?
I like "I will assume 'Arcane' is a typo for 'docker'". I don't have any suggestions for you, but damn that poor LLM was so confused.
Honestly OP should habe retried the prompt with the correct spelling
Try either orinth 1.5 9b, or if it will fit orinth 1.5 35b a3b. I run 35 a3b on my 7gb rx7600. I use the iq4_xs quant. They are based on qwen3.6 but with some post training. Stuff to make the think and tool call better.
What is arcane? I also don't know
https://github.com/getarcaneapp/arcane
Just a Docker manager, I've moved from Portainer and slowly moving all my containers across.
Why? I don’t care for portainer’s commercial moves but it seems to work fine most of the time. How is arcane better?
Asking seriously. Not sarcastically.
Mostly because of their more and more restrictions on the free version.
Arcane seems to be the manager that is getting all the love at the moment, ie it was recommended a couple of times in the selfhosted community
Nice! Is it similar in features? A web app, an API, agents for managing multiple nodes, docker compose-in-the-browser support?
Yes to all of them, seems very feature rich
Take a look at dokploy if you want those features. https://dokploy.com/
It never gets mentioned but I've been running it for years (multinode cluster first, now single node) and am very happy.
You’re paying $4.5/mo for Dokploy for your self hosted setup??
No. It's been free for me ... Did they start charging for it?
EDIT: if I understand it correctly the pricing is if they host it on a server for you. Maybe I'm wrong, but its running for free on my server right now and I updated it last week and it doesn't ask for money.
Makes sense. SH through GitHub is free and self-managed. While anything else is paid based on their website’s pricing.
Try Ternary Bonsai. It's a heavily quantized version of Qwen 3.8 27b. It won't do wonders, but it shoud be better: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev The page says you need a fork of llama.cpp, but the support was included in the latest release (0.7). You can read more about the mode on the main model page: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
Oh god no. Bonsai is such a waste of time! Totally a dull version of Qwen. Keeps getting basic things wrong and isn’t agentic at all.
But I’d be curious to hear if you’ve used it and had a different experience.
I personally run Qwen 3.8 27B at Q4_K_XL, but given OP's system requirements, I'd try Bonsai instead of a regular Q2 quant or the older Qwen 3.5 9B.
M2 16 GB… ah… let’s see how Bonsai works for OP…
Cries in old 1070 architecture
I had already downloaded a Bonsai 27B, just not really tested it yet
You can also try https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
It's more demanding, but also much capable.
I ran this for a good while, quite usable, like gipity 4o or 4.5
If I recall properly, in system prompt you can set thinking level via keywords(quick google should do). And there are three levels "Xhigh, medium, low", Qwen defaults to Xhigh. So set it to something lower and maybe it will stop over thinking.
i guess my advice is misplaced, given the community name, but in case the goal is to get LLM inference for free, not necessarily a local one, you could try opencode (their free plan gives you access to some decent models), or nvidia's NIM if you don't want a vendor lock-in to any specific agent/chat app.
I assume NIM only works with nvidia cards not AMD?
it's a cloud service. It gives you access to an api. It couldn't care less about your GPU. Can be used on anything that is capable of internet connection. Specifically NIM has a max requests per minute restriction tho. Due to this, running ai agents on it would be a pain in the ass, but technically possible (still not as painful as local inference tho). And it still works fine for plain chats.
Large Language Model needs more largerer model what speks english gooder.
Quick question, running a 9070xt here, what's the quickest/easiest way to get started running this locally? What's the most privacy respecting/trusted/safe way to run this stuff? Not after a fully blown agent I guess, something to help with my coding mostly as a search engine. I'm pretty ignorant on how this all comes together sorry. Do I even bother?
That is my current journey - trying to figure it out.
I want this from a local LLM:
I started with Automatic1111 ~2 years ago for generative image AI but uninstalled it at some time. In the last weekend I've installed LM Studio and Bionic and have been recomended Hermes as an agent.
Models tend to go around in circles when you quantise the KV cache too much.
What quant are you running? You might get better mileage from a gemma4 model of similar size and quants.
I had been using a gemma4 model, E4B and E2B. The E4B seems to work well
Yeah… E4B is definitely better than A2B.
You can try bonsai 2 and oronith 1.5 9b.
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.