Selfhosted

61624 readers

292 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

What budget friendly GPU for local AI workloads should I aim for? (lemmy.dbzer0.com)

submitted 9 months ago by pjusk@lemmy.dbzer0.com to c/selfhosted@lemmy.world

34 comments fedilink hide all child comments

Hiya,

Recently upgraded my server to an i5-12400 CPU, and have neen wanting to push my server a bit. Been looking to host my own LLM tasks and workloads, such as building pipelines to scan open-source projects for vulnerabilities and insecure code, to mention one of the things I want to start doing. Inspiration for this started after reading the recent scannings of the Curl project.

Sidenote: I have no intention of swamping devs with AI bugreports, i will simply want to scan projects that i personally use to be aware of its current state and future changes, before i blindly update apps i host.

What budget friendly GPU should i be looking for? Afaik VRAM is quite important, higher the better. What other features do i need to be on the look out for?

you are viewing a single comment's thread
view the rest of the comments

[–] drkt@scribe.disroot.org 10 points 9 months ago (2 children)

It's all VRAM, that's the bottleneck for even the best GPUs. AMD support is spotty so you should stay in Nvidia's claws unless you know what you're doing. Figure out what kind of money you're willing to part with, and then get whatever Nvidia GPU gets you the most VRAM.

[–] meldrik@lemmy.wtf 5 points 9 months ago (1 children)

The size of the LLM should be less than the amount of VRAM available.

[–] Jakeroxs@sh.itjust.works 4 points 9 months ago

Learning about quant level was helpful as well

[–] anamethatisnt@sopuli.xyz 3 points 9 months ago

Yeah, for a budget friendly AI GPU I would look for a 5060 Ti 16GB.