Selfhosted

61624 readers

345 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

What budget friendly GPU for local AI workloads should I aim for? (lemmy.dbzer0.com)

submitted 9 months ago by pjusk@lemmy.dbzer0.com to c/selfhosted@lemmy.world

34 comments fedilink hide all child comments

Hiya,

Recently upgraded my server to an i5-12400 CPU, and have neen wanting to push my server a bit. Been looking to host my own LLM tasks and workloads, such as building pipelines to scan open-source projects for vulnerabilities and insecure code, to mention one of the things I want to start doing. Inspiration for this started after reading the recent scannings of the Curl project.

Sidenote: I have no intention of swamping devs with AI bugreports, i will simply want to scan projects that i personally use to be aware of its current state and future changes, before i blindly update apps i host.

What budget friendly GPU should i be looking for? Afaik VRAM is quite important, higher the better. What other features do i need to be on the look out for?

you are viewing a single comment's thread
view the rest of the comments

[–] irmadlad@lemmy.world 4 points 9 months ago (1 children)

I've self hosted a few of the bite sized LLM. The thing that's keeping me from having a full blown, self hosted AI platform is my little GeForce 1650 just doesn't have the ass to really do it up right. If I'm going to consult with AI, I want the answers within at least 3 or 4 minutes, not hours. LOL

[–] Diplomjodler3@lemmy.world 9 points 9 months ago (1 children)

Quite so. The cheapest card that I'd put any kind of real AI workload on is the 16GB Radeon 9060XT. That's not what I would call budget friendly, which is why I consider a budget friendly AI GPU to be a mythical beast.

[–] irmadlad@lemmy.world 3 points 9 months ago

It would be super cool tho, to have a server dedicated to a totally on premise AI with no connectivity to external AI. I'm just not sure whether I can justify several thousand dollars of equipment. Because if I did, true to my nature, I'd want to go all in.