366
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 10 Oct 2026
366 points (97.9% liked)
Technology
88734 readers
2874 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
Qwen3.8-flash-next is making waves since its near frontier or frontier (according to some i havent used it) that can be run on an ancient 8GB video card (12GB or more is recommended. 16GB is best.) with around 32GB of RAM using Strata.
Strata
It's a mixture of experts MoE model, so while it's around 66GB (depending on which quantization you decide to run) you're only really using about 6B parameters per token processed, meaning the VRAM requirements are a lot lower.
I personally don't see the value proposition in paying $4/1M tokens when i can generate around 30tok/s on 500W of power. I pay ¢13/kWh so to generate 1M output tokens takes me 9.25 hours and costs $0.60.
I'm also running an ancient video card and have a new one in the mail. If i got 200 tok/s even at higher power use i'm still a fraction of the cost of paying for API access and I want that GPU for other things regardless.
I feel lile medium and large businesses have a strong incentive to rent or buy inference using open weight models for a fraction the cost of paying OpenAI or Anthropic. Especially if you can pre load them with everything about your org to make them better suited for internal use.
The vast majority of uses can use a open weight locally run model. I think you would only want to pay for frontier models for really complex tasks with lots of steps, and even then there are usually ways to be smart about how you task agents with work to get better output and use less money.
Nice. I just wondered how they were coming along.
Right now, I have to manage which model does what. So I don't. It's the companies dime after all, and they clearly value speed over efficiency. But the model I directly talk to doesn't need to be super high end. It could be well versed in which model is best for what task and farm put all of the work to the right model for the job. Seems like that alone could reduce cost a ton. But then start adding in cheaper local models and the price could fall a ton.
It would also be more accurate. Like if I had a small local model for interacting with a set of apis. It wouldn't make the mistake of not considering that the api paginates results. The general model makes this mistake over and over.
So I agree this is the direction of the future. Just trying to figure out when we will get there.
This is where things seem to be headed. Local models can be pretrained and crammed full of your work's information, and opencode and other harnesses can pick the right model for the task ahead of time. I could be wrong but i've been predicting a drop in frontier model credit usage especially if they hike token prices again. People will only use it if they have to especially if you can find a way to task a local model with a lot of the grunt work.
Here's a video on it