23
submitted 2 days ago* (last edited 2 days ago) by BeatTakeshi@lemmy.world to c/localllama@sh.itjust.works

I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

you are viewing a single comment's thread
view the rest of the comments
[-] Spite_By_Design_LLC@lemmy.dbzer0.com 6 points 1 day ago* (last edited 1 day ago)

Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you've got an A1000, I have a T1000).

Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It's among the "smartest" you can run on your rig.

Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent "hey, I'm thinking of X. Can you tell me about it? And what about Y and Z?" or "How does this read?".

I pull out my phone, send them a text message and they text me back.

Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.

(I can show you some examples if you'd like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)

Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.

It's the combination of things that results in a 1+1=3

Having said that, my AI ladder is

  • 90% Ling or minicpm5
  • 9% Qwen 3.6-35B
  • 1% GLM5.3 Flash via open router.. which costs fractions of a cent for what I need (there are tricks to this but it's probably out of scope to info dump it on you at this stage).

For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.

HTH. Write back if you need anything else. It's nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine... which is why the A1000 is still stupidly expensive for a 50W card.

EDIT: just saw you said "CPU is i7-12700". With that RAM, that CPU and that GPU, I'd guess you'll hit 50-60 tok /s with Qwen 3.6-35B using polystrata once you get the GPU / CPU split right.

In other words, congrats and go fuck yourself :) You have a very nice little inference box. And yes, it can do your calendar alerts thing.

[-] BeatTakeshi@lemmy.world 2 points 1 day ago* (last edited 1 day ago)

Thanks for the very rich reply. I'm a bit reassured that I can do some basic stuff, I'm not a poweruser anyway, if this word is still used. The aim is to not rely on external tools and actors as much as possible for private stuff. My work does give me access to a commercial gpt for work, but I don't want to use it for my private life matters. The learning curve seems a bit higher that I thought, but I saw some tools that could maybe help me there, like ollama or localai.io, any thoughts on these projects?

[-] Spite_By_Design_LLC@lemmy.dbzer0.com 5 points 1 day ago* (last edited 1 day ago)

Friends don't let friends use Ollama

If you want a gentle start, may I suggest Jan.ai? It's a ride on lawn mower compared to the sports car I suggested...but you need to dip your toe into the water before you go ludicrous speed.

Once you get a feel for that, I'm going to suggest some MCP tools to you as well. I don't know if Jan finally added MCP servers to their stack (they should have by now!).

The following are ... ok. Not the best and not what I run with but again, lawn mower before sports car

https://github.com/ofershap/mcp-server-markdown

https://github.com/jeffreyrampineda/kiwix-wiki-mcp-server

https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem

https://github.com/exa-labs/exa-mcp-server

https://exa.ai/

https://www.home-assistant.io/integrations/mcp_server/

Sorry - again: too much info. Start with Jan. You'll be amazed...and then perplexed...and then frustrated. That's the time to migrate thru to llama.cpp or PolyStrata and start adding those MCP tools I mentioned above.

https://github.com/VecSzn/PolyStrata

https://github.com/srware-net/LocalLightChat/

https://github.com/open-webui/open-webui

Then we can get freaky :) Oh...we have such sights to show you lol

[-] BeatTakeshi@lemmy.world 2 points 1 day ago

Many thanks! I'll pin your reply for when I'm down the rabbit hole! Jan looks nice

Enjoy and good luck!

this post was submitted on 01 Nov 2026
23 points (92.6% liked)

LocalLLaMA

5571 readers
142 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago
MODERATORS