LocalLLaMA

4948 readers

17 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

llama rpc exists and works really well (github.com)

submitted 4 days ago by troed@fedia.io to c/localllama@sh.itjust.works

5 comments fedilink hide all child comments

I've been back'n'forth for a while whether to invest in a second 5060Ti to be able to go dual GPU on my main workstation. The target has always been to be able to run Qwen 3.6 27B at a usable model quant, kv-cache quant and context size.

Today I tried out llama.cpp with the rpc option. I sort of knew at the back of my head that there were projects doing cross-computer inference but I didn't know it was built into llama.cpp.

Sure, this requires that you have another machine on your local network (latency matters) with a GPU - but in my case I actually have two. One server that has a 12GB A2000 and one Bazzite box with an AMD something also with 12GB VRAM.

I just compiled the latest llama.cpp HEAD with the rpc option enabled on my headless server and launched my main llama-server pointing to it. With the 12GB of the A2000 in the mix I'm now able to comfortably run the 27B model Q4_K_M, Q8_0 kv-cache and 140000 context size, and I've still not maxed out the available VRAM.

PP at ~400 (a bit low, will see if that can be tuned*) and TG at 25 (MTP and ngram enabled) makes this perfectly usable. That's the token/second during inference I would've expected from a dual GPU setup in the workstation anyway.

So, if you also happen to have another machine on your network with spare VRAM - try out llama rpc. I have no idea why it took this long for me to do.

*) This is literally my first run with random defaults - tuning starts now

you are viewing a single comment's thread
view the rest of the comments

[–] luthis 4 points 4 days ago (1 children)

Surely we should have a hardware solution soon, that's just a bunch of vram? I saw an article recently where a dude got a server GPU or something and jimmied it up to run in his PC. Worked great apparently

[–] mierdabird@lemmy.dbzer0.com 2 points 3 days ago

Yeah Nvidia V100's and AMD MI50's or V620's are the accessible ones