Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I'm also set up with tailscale.

I wouldn't mind running a model at home, mostly to help document my setup. But I just don't trust it. So no.

A quick glance shows that I'm in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I'm running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I'm happy to have paid a "mere" $2400 compared to what similar systems cost now, and now I'm toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).

Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.

The server has tailscale installed, so yes I can access it anywhere.

I just bought a 32GB M6 Mac Mini that I'll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.

I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it's been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it's output into Claude Pro for vetting and it hasn't found any flaws in the simple scripts and things I've asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that's all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I'm just a little guy.

I self host both at home and at work.

At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I'm trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone's life a bit less admin-y without having to hire a full PA role.

Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC

Even ignoring all of the functional issues with LLMs/"AI" related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I'm tired of the wildfires and extreme storms from climate change).

So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.

I tried, but didn't want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.

I run Gemma4 12B on my RX9070 XT. it's not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing "agentic".

With my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8. I use it mostly for NLP processing of game server logs to create match summaries and I use whisper for audio-to-text transcription of videos.

While I do not have enough vram to put large logfiles into the LLM directly I was able to have it create a python script that generates a summary file at the end of each match. I feed the LLM the summary file and it generates color commentary that gets posted to Discord. Here is an example:

[🌟]Color Commentary: A Display of Skill on dod_napoli_b1! [📈]Multi-Player Action: On 08/05/2026 at 19:33:25, the action was intense on dod_napoli_b1, with key moments including the [Loft] capture by Inspector Lag (Allies) and [Axis Barn] capture by THE ROCK/BionicTwin (Allies). [🏆]Impact Players: Allies: Inspector Lag (Kills: 37), [415]Drago (Caps: 10). Axis: The Notorious C.U.M. (Kills: 39), [415]IronHeaD (Caps: 8).

$ollama list NAME ID SIZE MODIFIED
qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago

$ollama run llama3.1:8b

what is the best AI model to run on an Nvidia 3070 with 8gb of vram? The best AI model to run on an NVIDIA GeForce RTX 3070 with 8 GB of VRAM depends on several factors, such as:

1. Specific task or application: Are you interested in tasks like image classification, object detection, natural language processing (NLP), or generative models?
2. Desired level of complexity: Do you want to run small, simple models or larger, more complex ones that require more computational resources?
3. Computational requirements: What's your expected performance target? Do you need high-throughput, low-latency, or a balance between the two?

Considering these factors and the specifications of your GPU, here are some popular AI model examples suitable for an NVIDIA GeForce RTX 3070 with 8 GB VRAM:

Computer Vision:

1. YOLOv5: A lightweight yet powerful object detection model that's well-suited for real-time applications.
2. EfficientNet-B3: A compact and efficient image classification model, part of the EfficientNet family.

Natural Language Processing (NLP):

1. BERT-base-uncased: A popular language representation model for various NLP tasks like text classification, sentiment analysis, and question answering.
2. RoBERTa-base: A transformer-based language model that outperforms BERT on several tasks.

Generative Models:

1. StyleGAN2: A state-of-the-art generative adversarial network (GAN) for image synthesis and manipulation.
2. DDPM (Denoising Diffusion Probabilistic Model): A flexible and efficient generative model suitable for various image generation tasks.

Keep in mind that these models have varying computational requirements, memory footprints, and performance characteristics. You may need to experiment with different hyperparameters or architectures to find the best match for your specific needs.

To estimate which models will run smoothly on your hardware, consider the following:

• Memory usage: Make sure the model fits within your GPU's VRAM (8 GB). Larger models might require downsampling or other memory-saving techniques.
• FLOPS (Floating-Point Operations Per Second): Estimate the computational requirements of each model. The RTX 3070 has a peak FLOPS rating of around 12 TFLOPS, but actual performance may vary depending on the specific workload.

For more detailed estimates and comparisons, I recommend exploring resources like:

• TensorFlow Model Garden: A collection of pre-trained models for various tasks.
• PyTorch Hub: A library of pre-trained models and implementations.
• NVIDIA Deep Learning SDK: Offers a range of tools, including cuDNN-accelerated deep learning primitives.

Feel free to provide more specific requirements or task details if you'd like me to help with model selection!

total duration: 13.256527877s load duration: 71.80054ms prompt eval count: 466 token(s) prompt eval duration: 62.91049ms prompt eval rate: 7407.35 tokens/s eval count: 572 token(s) eval duration: 12.889617619s eval rate: 44.38 tokens/s

Last night I managed to run Qwen Flash Next iq2 XS using Strata in docker on a VM

I passed through my 3090, 64GB of DDR4, 12 vcores of my 5960x,

Managed 90t/s

I'll probably try a larger version soon but QFN already seems a lot more capable than 3.6 35b A3B, and runs better than Qwen 3.8 dense in llama.cpp

I don't want to create an extra post, so I ask here.

Where do you guys get your hardware from, for an affordable price?

I'd like to run at least a 27b model but my hardware is too weak. I'm on a 16GB unified mem, M4. Tried running Bonsai 27b (it's Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.

Ollama container, 2070 super, 3-7b, meta and qwen

I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.

I also use it for a self hosted mealie to figure out recipe->ingredient.

I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.

Tried running a decent Qwen on a 3060 TI but I only have 16g ram so it kept getting stuck and not completing tasks.

I'm building toward exactly this, so take it with that bias. The cheap tier works if you lower expectations. An N150 mini PC with 16GB runs small models through Ollama fine, but 8B models are slow on that class of chip, and you'll feel it on anything chat-shaped. Where it shines is async work: summarize this, tag that, answer when it's ready. For phone access I'd keep it LAN-only and use WireGuard or Tailscale to get home rather than opening a port.

Disclosure: I'm working on a small open-source home box that bundles local AI with offline Wikipedia and media. What would make you trust a box like that over a setup you built yourself?

I've thought about bodging something together with n8n and Tasker so I can just screenshot something or send it a voice note and have it add an event on my calendar.

That being said my server is an 8th gen i3 w/ 12GB of RAM and I'm not willing to leave my desktop running 24/7 so idk if I can run a model good enough to handle this over there. Even if it is a supposedly menial "OCR and call tool" workflow.

Qwen 3.8 27B, running on a RTX 5090. Works so well.