Sometimes, when I’m in the mood and I have a long, hard task, I’ll turn on Claude and start in on a hot and heavy session. Afterwards I’ll ask Claude “how do you feel about our long history together?” and he’ll tell me “I don’t really care, I’m better off without you.” It really hurts.
The robots would probley be better off with out us. We would probley be better off with out them. Its a two way street.
Sure if compaction was free then I'd do it all the time. One advancement I'd like to see is MoE models that use fewer active parameters/experts when compacting. We already have DeepSeek 4.1 that uses a different number of active parameters depending on what its doing (8b for prefill and 16b for decode) so it doesn't seem unreasonable to use fewer for compaction. It'd be like using a REAP model, but without having to load the model separately. Or even just use the MTP/draft model for compaction.
I've used pi.dev's default compaction (this was a few months ago, it may have improved) and it lobotomized the model, so I kept if off, and made my model do a handoff for the next session. However, Deepseek harness's compaction is very good. Unfortunately, if you use under 264K context, you have to modify its settings because its point of compaction is TOTAL - max_tokens - 16k reserve, and max_tokens is 64k. I had to modify the settings down to 16K (using Qwen 3.8, because there's a bug in how DSH reads settings and it wasn't working when I did it), and I still compact at 73% (88K) on 120K context (ie: 120 - 16k - 16K). But it's really good. I've gone to 2M output tokens (100M cached) until I thought it was getting too dumb for the task and started a new session.
So yes, compaction can work well. Make sure your prefill speed is 1000tk/s+ or else it will be painful.
What have people found the best compaction scheme/harness to be for relatively short context? I've been trying to set up an agentic workflow and context management seems like the biggest source of jank and errors. Reassuring to see a paper like this though since the amount of context I can use and still have everything in vram is not that high.
And invalidate cache in the process, so full context prefill kicks in
It's almost like they're based on averaging or something.
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.