this post was submitted on 09 Apr 2025
38 points (79.7% liked)

Selfhosted

61475 readers
1007 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
ayyy@sh.itjust.works
curbstickle@anarchist.nexus
curbstickle_lw@lemmy.world
38
How to use GPUs over multiple computers for local AI? (lemmy.dbzer0.com)
submitted 1 year ago by marauding_gibberish142@lemmy.dbzer0.com to c/selfhosted@lemmy.world
64 comments fedilink hide all child comments
 

The problem is simple: consumer motherboards don't have that many PCIe slots, and consumer CPUs don't have enough lanes to run 3+ GPUs at full PCIe gen 3 or gen 4 speeds.

My idea was to buy 3-4 computers for cheap, slot a GPU into each of them and use 4 of them in tandem. I imagine this will require some sort of agent running on each node which will be connected through a 10Gbe network. I can get a 10Gbe network running for this project.

Does Ollama or any other local AI project support this? Getting a server motherboard with CPU is going to get expensive very quickly, but this would be a great alternative.

Thanks

you are viewing a single comment's thread
view the rest of the comments
[–] vane@lemmy.world 4 points 1 year ago* (last edited 1 year ago) (1 children)

If you want to use supercomputer software, setup SLURM scheduler on those machines. There are many tutorials how to do distributed gpu computing with slurm. I have it on my todo list.
https://github.com/SchedMD/slurm
https://slurm.schedmd.com/

[–] marauding_gibberish142@lemmy.dbzer0.com 1 points 1 year ago (1 children)

Thanks but I'm not going to run supercomputers. I just want to run 4 GPUs separately because of inadequate PCIe lanes in a single computer to run 24B-30B models

[–] vane@lemmy.world 1 points 1 year ago* (last edited 1 year ago) (1 children)

I believe you can run 30B models on single used rtx 3090 24GB at least I run 32B deepseek-r1 on it using ollama. Just make sure you have enought ram > 24GB.

[–] marauding_gibberish142@lemmy.dbzer0.com 1 points 1 year ago (1 children)

Heavily quantized?

[–] vane@lemmy.world 1 points 1 year ago* (last edited 1 year ago) (1 children)

I run this one. https://ollama.com/library/deepseek-r1:32b-qwen-distill-q4_K_M with this frontend https://github.com/open-webui/open-webui on single rtx 3090 hardware 64gb ram. It works quite well for what I wanted it to do. I wanted to connect 2x 3090 cards with slurm to run 70b models but haven't found time to do it.