I had to get mine from ebay and wait a couple weeks as they came from China too. Also I recommend having a 3D-Printer and some Blower-Fans on hand as you will either have to buy or print your own fan shroud for these server cards.
userentity
joined 1 month ago
A P40 with 24GB is ~150€, a V100 (32GB) is ~600€. Both of these fit Qwen3.6 27B (The P40 is about 3x slower though). The V100 even fits 400k context with a Q4 KV-Cache , which means you can have two slots for parallel processing (llama-cpp). You don't even have to use system memory. One of my inference servers is running with 8gb of DDR3 and a 2nd Gen i7, so my old hardware has a good use again.
aw man blobs in oss? why is there no scanner for that?
Calling a "blacklists.rs" directly in the top level rust source tree hidden is an overstatement. The blacklist is not obfuscated or compressed. It is just a blacklist.
Qwen 3.6 27B MTP (Q4) is really great. ~24GB VRAM usage with one slot @ 262k . ~30GB with 3 slots at 128k. Also it does not struggle like Gemma if you use e.g. a Q4 KV-Cache. And it runs at 400-800 ppt/s and 20-60 itp/s on a V100.
But it has a competitor since a short while that is Laguna XS 2.1, which is really good for a A3B MoE. I'd never have thought a 30B MoE could be on par with a 27B dense, but it seemingly is.
That's why you couple it with your own, self-hosted yacy or searxng instance. Embedded world knowledge does not help a model if it becomes outdated. I just let my agent research, embed that knowledge to a little Qdrant server, so other servers are not bothered again and pull the information from there when needed again. With a little RAG you can have GPT at home.