53
Uses for local AI? (lemmy.mtate.me.uk)
submitted 2 years ago by smeeps@lemmy.mtate.me.uk to c/selfhosted@lemmy.world
55 comments fedilink hide all child comments

Im using Ollama on my server with the WebUI. It has no GPU so its not quick to reply but not too slow either.

Im thinking about removing the VM as i just dont use it, are there any good uses or integrations into other apps that might convince me to keep it?

you are viewing a single comment's thread
view the rest of the comments
[-] thirdBreakfast@lemmy.world 5 points 2 years ago

I use the Continue VS Code plugin with Ollama to use a couple of different models (deepseek-coder-v2 & starcoder2) to recreate a local only Github Copilot type experience for coding. This is on an M1 Apple Silicon though. For autocomplete the generation needs to be pretty brisk - I'm not sure how that would go in a VM without a GPU.

[-] Amongussussyballs100@sh.itjust.works 2 points 2 years ago

How well does the M1 chip keep up? What size models are you running with it? Interested in getting an M1 laptop and I am curious.

[-] thirdBreakfast@lemmy.world 1 points 2 years ago 
starcoder2:latest       	f67ae0f64584	1.7 GB	3 days ago 	
phi3:latest             	d184c916657e	2.2 GB	3 weeks ago	
deepseek-coder-v2:latest	8577f96d693e	8.9 GB	3 weeks ago	
llama3:8b-instruct-q8_0 	1b8e49cece7f	8.5 GB	3 weeks ago	
dolphin-mistral:latest  	5dc8c5a2be65	4.1 GB	3 weeks ago	
codeqwen:latest         	df352abf55b1	4.2 GB	3 weeks ago	
llama3:latest           	365c0bd3c000	4.7 GB	4 weeks ago

I mostly use starcoder2 with Continue for code autocomplete, the big deepseek coder is a bit slow (I can feel it thinking), but it and the regular llama3 are good for chatbot type programming questions.

I don't really have anything to compare the M1 performance to. I guess the 8GB models output text a little slower than the web versions of the same models, and the 4GB ones about the same. Using ollama in the terminal, there's sometimes a 0.5-2 second pause before it starts outputting. Not with phi3 though - it's surprisingly snappy for the quality of answers.

this post was submitted on 24 Jul 2024
53 points (90.8% liked)

Selfhosted

61808 readers
476 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
ayyy@sh.itjust.works
curbstickle@anarchist.nexus
curbstickle_lw@lemmy.world