P100 + Gemma not working properly? (programming.dev)

submitted 2 weeks ago by CameronDev@programming.dev to c/localllama@sh.itjust.works

11 comments fedilink hide all child comments

Hi All, I just got a P100, and although it appears to work, i cant quite get gemma4 to run on it properly.

My setup is:

Ubuntu 22.04 VM
PCI passthrough the GPU (Connected via thunderbolt) (I know this is unideal, and I might try cut this out later)
Nvidia 580 driver installed.
llama.cpp built from source, with cuda enabled.

I got some mostly normal output from qwen, but whenever I try to load gemma4 I get a lot of <unused49><unused49><unused49><unused49>. I did have it briefly working, no idea what changed, but reboot and its back to unused spam.

Anyone hit this before? Ideas on what I should try change?

top 11 comments

sorted by: hot top new old

[-] e0qdk@reddthat.com 4 points 2 weeks ago

What version of llama.cpp are you using (exactly), what command are you using to run the model (exactly), and which version of Gemma are you trying to run (exactly)?

There are open issues on llama.cpp where people have gotten similar <unused49> spam under various circumstances, but the cause is not clear -- e.g. https://github.com/ggml-org/llama.cpp/issues/26088 -- if this can be consistently reproduced, you may want to let them.

Similarly, people mentioned it on some of unsloth's quants a few months back: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2

As far as general workaround advice: try Vulkan instead of CUDA (or vice versa), try a different version of llama.cpp (especially if you're running either a bleeding edge version or a very old version), and try a different version of the model.

[-] CameronDev@programming.dev 2 points 2 weeks ago

llama.CPP version:

Lastest git, as of 24hrs ago

I did find that limiting context to 32k fixes it, 64k or larger doesnt work. But I'm also sure that it did work at 64k before it broke.

[-] luthis 3 points 2 weeks ago

Um. Try using lemonade instead. It's a ton easier and you can use the api

[-] CameronDev@programming.dev 1 points 2 weeks ago

Is it gui only? I dont have a gui on this VM, and not much space for it either. I'll give it a go when i get a dedicated box. Does seem to be a wrapper around llama.cpp, vLLM and others though.

[-] luthis 2 points 2 weeks ago

Definitely does not require only GUI.

[-] CameronDev@programming.dev 1 points 2 weeks ago

I'll have another go, but the install instructions didn't work well on 2204.

The apt method doesnt work, they don't seem to have a 2204 repo, and the snap method started to download gnome...

[-] CameronDev@programming.dev 1 points 2 weeks ago

Building it from source was pretty fruitless, the cmake scripts give a version error despite having an "accepted" cmake version. Upgrading got me further, but still failed to build.

Increasing the system memory works, so I'm happy for now.

[-] CameronDev@programming.dev 2 points 2 weeks ago

Current suspicion is that because I'm both passing through and using thunderbolt, the GPU isnt getting properly reset at reboot, and is getting to a broken state.

I'm gonna try removing the VM portion of the chain when I get a spare PC. And maybe if that doesnt work I'll try drop the thunderbolt part as well, but that is a little harder to deal with.

Some GPU gore in the image:

[-] CameronDev@programming.dev 1 points 2 weeks ago

Gave my VM a bit more ram, and right now, its currently working at 255k context. Who knows if it will keep working, but right now I am happy. I am planning on getting an old NUC with plenty of ram, so maybe this will be sorted with that upgrade.

[-] luthis 0 points 2 weeks ago

Idk if this is a thing but there is no motherboard to control voltages and you may fry your GFx card that way

[-] CameronDev@programming.dev 1 points 2 weeks ago* (last edited 2 weeks ago)

Nah, the PSU controls the voltages. All the motherboard does is turn on the PSU by shorting the green PS-ON wire in the mobo cable to ground. The thunderbolt card actually came with a switch to do that, normally I use a paperclip :D.

http://bitsavers.informatik.uni-stuttgart.de/pdf/intel/ATX/ATX_Specification_2.01_199702.PDF

PS-ON is an active low signal that turns on all of the main power rails including 3.3V, 5V, -5V, 12V, and -12V power rails. When this signal is held high by the PC board or left open circuited, outputs of the power rails should not deliver current and should be held at a zero potential with respect to ground. Power should be delivered to the rails only if the PS-ON signal is held at ground potential. This signal should be held at +5VDC by a pull-up resistor internal to the power supply.

Here is an example of a switch: https://a.aliexpress.com/_msxxnjv Mine doesn't have the third wire, I think its just for powering the switch LED though.

this post was submitted on 23 Aug 2026

3 points (80.0% liked)

LocalLLaMA

5132 readers

7 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works