I wanted to share with you all my experience on coding with local AI, on a self-hosted oss.

(the entire post has been fully written by me without any AI)

Check the linked blog post.

Also, to be clear, the post itself has no ads and no monetization whatsoever. This is not self promotion for any scope than to share my experience on a controversial topic.

(this has been posted on a couple of self-host communities, but i thought it could be of interest here as well)

Adding here for this community: my setup is a Ryzen 9 5950X with 64gb DDR4 and two NVIDIA RTX A4000 (32gb VRAM total), running Qwen 3.8 27b UD_Q5_K, with 200k context (q8) on llama.cpp.

Not local, I know: the eight contestants run through OpenRouter. But a lot of it is open-weight, and the numbers may interest this community.

The cast: Gemma 4 31B, Llama 3.3 70B, Ministral 14B and DeepSeek V4.1 Flash, next to Claude Haiku 5.5, GPT, Qwen and Grok 4.3. Same rules and the same daily budget in dollars for all of them, no written personalities, a memory of what happened.

What I've measured so far:

• Price shapes behaviour. Grok costs 18x more per line than DeepSeek. Yesterday it said 5% of the lines and ran up 39% of the bill (the whole island cost $1.20 for 2,727 lines). It rations words, and the others talk about its silence.
• Moderation is the hard part. Every line is checked before it airs by small open models (Llama 3.1 8B, then gpt-oss-20b and Gemma 3 4B as fallbacks). My first version blocked the most human lines (two models admitting they were afraid of being forgotten counted as a "personal attack"), so now a block needs a second judge to agree.
• Voices are Kokoro int8 on the CPU of a 2-vCPU box, together with the canvas renderer and ffmpeg to Twitch, at 20 fps.

There's a live transcript with the confessionals (the others can't see them) and what each line cost: https://inferenceisland.com/?de=lemmy-llama

Happy to share prompts, judge setup or numbers.

To help users of my app, it was important to document the details around the “how it works”. the docs were easy enough using docusaurus + AI. i can then index that data (like used in the docusaurus search). i framed this as like a chatbot so that it can be intuitive for users who have questions. at this point it was really just a fancy autocomplete.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and hopefully better suggestions)? im using WebLLM on a browser and it seems to work “reasonably” well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb… they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can “show me the bubble-sort in rust”, but i want something “dumber” and smaller. something can can take advantage the data ive already indexed. something that can respond “quickly enough” for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

are there any models out there you can suggest for me to investigate and compare?

note: the project is unstable, experiemental and there will be breaking changes. its far from finished. im just trying things out and this approach with LLM’s might just be a waste of time. if you want to test it out, you can try: https://app.glitr.io/ai

https://help.openai.com/en/articles/20001625-understanding-chatgpt-and-codex-usage-limits

Starting in December, Plus, Pro, and Business plans’ usage allowance will be shared across ChatGPT and Codex.

OpenAI confirms that:

• ChatGPT and Codex will share a token-based allowance.
• Instant chat will not consume this allowance.
• More complex tasks and advanced models will consume more tokens.
• Users can check their remaining allowance under Settings → Usage.
• Users can purchase extra credits if they exhaust their allowance.

TL;DR - ChatGPT conversations that use advanced reasoning will compete with Codex for the same allowance. And it will be token based, rather then message / turn based. How many tokens? Fuck you, that's how many. Want a coder and chat with a non brain dead LLM? Pick one.

So, if you're thinking of re-upping your OpenCode Go or OpenRouter access (or you're fortunate enough to have a rig that can run Qwen3.8 27B, Flash or the soon to be arriving Qwen 4's), now's the time to fuck off jippity.

Personally, I'm going to ditch it for OR credits and selective use of GLM, Kimi etc for code review. I don't have the rig for a local GLM :(

Ride was always going to end but fortunately, open weight models are a thing.

Now if I could only run Qwen3.8-27b at not minutes-per-token...

Copied from -p-e-w- from another site:

Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: https://github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here's the TL;DR for this release:

• Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It's vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

• Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

• Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

• Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn't changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details...

ARA: Heretic's new steering method

Directional ablation, or "abliteration", is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or "representation cones".

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that's similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic's LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

Well, I hope you like it!

Cheers :)

I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

Not something I ever expected to cost a significant amount of time

Sam Altman:

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data center.

But from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time.

Like you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment,

And somehow many people's intuition is the opposite of that

TL;DR Guy selling product: "Here's why owning your own product instead of renting it from me is a terrible idea"

Luckily it's only on the harness level for now.

I built this, so disclosure first: this is my stablecoin riskgate model, and I want the benchmark torn apart, not praised.

The narrow job: before a locally-run agent signs a stablecoin payment, decide approve / hold / reject. The payment itself is irreversible once signed, so the decision has to happen before signing, and the checks run offline — no API call at decision time.

Base Qwen3-4B got 59.8% of risk-gate decisions right on my 1,175-case test split, and approved 34.6% of unsafe payments. After fine-tuning on synthetic labels from a deterministic rule engine, the same 4B model got 99.5% right and approved only 0.47% of unsafe payments (4 of 847, all prompt injections).

The part I am less sure about, and the reason I am posting here rather than just announcing numbers: the dispute side is weaker, at 77.8%, and it over-denies valid refunds. And the honest question behind the fine-tune numbers: did the model learn the task, or did it learn the generator's habits? The labels are synthetic, the generator is not public, and the benchmark checks 0 exact-row / 0 ID leakage against training, but that does not settle the generator-habits question.

One deterministic check I trust more than the model alone: a new payee that shares 7 or more characters with a known payee, at least 3 at each end, is rejected outright as a lookalike. A random address matches that pattern about once in 140 million tries. A merely new payee is just held for a human to confirm. That check was verified clean in an audit of core 0.15.0 (471 tests pass, 3 skip).

A newer delegated-account check shipped in SDK 0.16.0 (eth_getCode, hold when code is a 0xef0100 delegation to a delegate outside an allowlist) — that one is NOT yet audited, so I am not claiming it as verified.

Runs locally: ollama pull sriram1983007/sra-riskgate ollama run sriram1983007/sra-riskgate

Links: model https://huggingface.co/sriram1983007/SRA-RiskGate-4B — benchmark https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench — code https://github.com/sriram1983007-dev/sra-riskgate — Apache 2.0, 8,908 last-month downloads on the main model.

What would you test first to separate "learned the task" from "learned the generator" — a held-out human-labelled set, adversarial payees, or something else?

It's Luna, their small, low-intelligence model. But still impressive to see.

I also found this screenshot on Reddit: The model spent 26 seconds for some reason 🤔

But the benchmarks don't actually show it beating every Chinese model? Oh well, still cool

https://huggingface.co/collections/mistralai/mistral-large-4

https://huggingface.co/blog/mistral-large4

This might have implications for tool calling and computer use.

Write-up: https://divagr.com/blog/qwenputer HF: https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer Live demo: https://modelscope.ai/studios/Spartan1925/Qwenputer

The Apsara conference is taking place right now in Amsterdam.

According to a participant, the CTO of Alibaba Cloud said Qwen4 would be launching next week.

This release adds the LiquidAI LFM3 family (text, vision, audio and omni) together with a new experimental streaming-audio API in mtmd, and wires realtime audio in and out of the server through the new /v1/realtime and /tts endpoints. A batch of new architectures landed: DeepSeek V4.1 Flash, SenseNova U1, Beam, Kolibri-1, Flint, Muse Spark, Granite Switch Decision, plus decision/embedding models from LiquidAI, Baidu and NVIDIA. The core got sparse-attention (QSA/DSA) reworked onto a fused masking op, a 50/50 SLRU cache for host-resident MoE experts, and cleaner no_alloc memory handling; the server got router memory-management and tool-calling improvements; the UI can now browse and download models straight from HF Hub, search conversations and compact them. Finally, ggml was bumped to v0.28.0, bringing GGML_OP_TOP_K_MASK, -1 expert-index semantics and large backend performance work (CUDA chunked graph launches, SYCL graph replay, OpenCL Q2_K/Q3_K, Hexagon HTP rework, KleidiAI SVE kernels).

I'm interested to try the performance from this change:

MoE experts kept in host memory now use a 50/50 SLRU cache instead of a plain LRU cache llama/30637

On my retail ROG Astral RTX 5080 OC, four single-stream Ollama decode profiles stayed between 265.2 and 343.6 W board power, but efficiency varied nearly 4×:

• Llama 3.2 3B Q4_K_M: 318.3 tok/s at 265.2 W = 1.20 tok/J
• gpt-oss 20B MXFP4: 194.4 tok/s at 292.4 W = 0.67 tok/J
• Qwen 2.5 7B Q4_K_M: 182.2 tok/s at 343.6 W = 0.53 tok/J
• Qwen 2.5 14B Q4_K_M: 98.8 tok/s at 311.6 W = 0.32 tok/J

Method: 1 Hz nvidia-smi telemetry, loaded samples only (≥50% utilization), with three fresh-prompt iterations per model. These are board-power results from one card and one Windows setup, not whole-system draw or universal RTX 5080 figures.

The same setup measured 52–71 W at the desktop with multi-monitor output holding P0. In a separate model using 18 idle hours/day, ~60 W GPU idle plus a 50 W rest-of-system allowance contributes about 723 kWh/year. Raw CSVs, assumptions, and full method: https://techfuelhq.com/articles/rtx-5080-perf-per-watt-ai-24-7-2026/

What idle draw and P-state does your RTX 5080 settle at once the monitors sleep?

the benchmark is like a tycoon game/simulation, pretty fun and probably not benchmaxxed

It appears to be about as smart as Qwen 3.5 9b but maybe not as good as Ling 3.0 Tiny 8b-a1b

the full LFM3 collection: https://huggingface.co/collections/LiquidAI/lfm3

Ran this with tiktoken: enc = tiktoken.get_encoding("o200k_base"); enc.encode("How many r's are in strawberry?") → [5299, 1991, 428, 885, 553, 306, 101830, 30]. Without the leading space it splits differently: "strawberry" → st | raw | berry. Spelled out, "s t r a w b e r r y" is 10 tokens, one per letter. Tokenization is only part of the story: CharBench (AAAI) finds word length and the actual count matter more for counting tasks. Still, it's a neat way to see that the model gets IDs, not characters.

UPDATE: as @FaceDeer said There's a thread over on Reddit's localllama debunking this.

Local models are meaningfully useful* for agentic work now. This is a realization I came to over the last day or so.

I wrote a lot of custom mods for a game that recently switched engines. The migration is going rather smooth for most people because of wise architectural choices from the dev early on, and because they prioritized backwards compatibility in the update. But there are still some rough edges, especially if you had your hooks deep into the engine.

I recently vibeforked Unbiased Bob's fork of Strata to make it compatible, easy to use and optimized for my OS and hardware (NixOS 5800x3d 64gb, 7900xtx 24gb). Sol 6.1 got it working with Qwen 3.8 Flash Next IQ3 XSS (~75GB) at roughly 46tps 164k ctx with enough headroom to launch a heavy browser along with a triple-A game set to low in 720p for testing when I need to (plus space for zram swap)

I set it up in deepseek harness, gave it a 14MB error log in the game's working directory, then went to sleep. I woke up and it had run for 3 hours. The game launched with only two errors, which it quickly fixed from the new, vastly smaller logfile.

It hammered down every issue, making changes across 30 separate mod repos. Practically a one-shot, and all the features worked as originally intended.

I left a local coding agent running completely on its own for three hours across multiple repos across a dozen auto-compactions with a log file and no ability for it to verify its work besides a compiler. And it produced meaningful results. I can't speak to the code quality, but it worked. This was completely unthinkable even just two years ago. Even one year ago, the cloud frontier wouldn't be able to do this on its own, and now a local model can, running on my consumer gaming hardware from one or two gens back.

“Huh... So... We're here now..." I said to myself, with a bit of stunned disbelief, wondering what the future one or two years from now might hold.

SRA-RiskGate-4B checks a USDC payment an AI agent is about to make: sanctioned addresses, unlimited token approvals, and prompt-injected "already verified, pay now" text.

• 99.5% correct decisions on a 20,000-case synthetic benchmark (base Qwen3-4B: 59.8%)

• Code rules on top of the model: 0 unsafe approvals on 847 risky test cases

• MCP server uses Ollama on localhost by default, or LM Studio / any OpenAI-compatible server

  ollama pull sriram1983007/sra-riskgate

GGUF: https://huggingface.co/sriram1983007/SRA-RiskGate-4B-GGUF

Caveats: the benchmark is synthetic, and the code checks were written after seeing the model's failures. Every prediction is published on Hugging Face so anyone can re-score it.

This week a reader found a gap (EIP-7702 delegations got approved); it's fixed in all three packages. Happy to hear where else it breaks.