It's as good as an 8B can be, but with the right system prompt for your purpose and proper expectations, I think it's good. I've had some other newer 8B that blew up after a few cycles, literally getting stuck on something, but I can't say this one ever did. But again, even the big models like Claude and the rest work better with short sessions and a specific, detailed prompt to start with. Use a model to make the prompt, telling it to be detailed, concise, and minimize fluff. Less tokens in and out that way, less context drift (hopefully).

The results I’m getting are a bit slow though. Have you found a way to speed it up on the Framework Desktop?

Using more-heavily quantized versions will run more-quickly, since they hit the memory bus less-heavily. I use Q6_K on llama.cpp on Vulkan, max context window 128k, which runs at 3.2 t/s with a fresh context window. That may not be sufficient for you; depends on what you can tolerate.

My command-line parameters are:

$ nice -n20 ionice -c3 ./llama-server --direct-io --fit off --no-mmproj -c 0 -ngl 99

If you check radeontop, you should see that your GPU is saturated, that your CPU isn't doing the work or anything like that.

I was definitely not expecting the first thing to come back from “Hello, who are you?” to be “I’m the person who’s going to teach you how to cook!” :p

You may want to set a system prompt, if you haven't set one, as that sets the tone of the conversation (not to mention, if you're using some system that supports "characters", whatever character prompt you have set for them. If you're using SillyTavern and the Text Completion API rather than the Chat Completion API, I suggest changing the default system prompt, since the default is:

https://docs.sillytavern.app/usage/prompts/

The default Main Prompt is:

Write {{char}}'s next reply in a fictional chat between {{char}} and {{user}}.

The problem is that when using the Text Completion API, SillyTavern implements "{{char}}" by switching "{{char}}" for the currently-active character's name. This doesn't matter in a chat with a single other character, but for group chats, the text "{{char}}" is replaced with changes every time the speaking AI character does, and means that your backend (for me, llama.cpp) can't necessarily use the K-V cache for the text since starting from the previous prompt (which might be spoken by another character). This makes the backend run unnecessarily slowly, since it can't use the K-V cache for anything since the last time you were talking to the currently-speaking character in the current conversation. This doesn't matter as much for some SillyTavern users, with a small context window and a lot of bandwidth (it'd matter less on my RX 9700 XTX), but the Strix Halo has lots of memory (so you can have a large context window) but limited bandwidth.

I use the following system prompt, which avoids use of "{{char}}":

"Develop the plot slowly, always stay in character. Describe all the world in in full, elaborate, explicit, graphic, and vivid detail. Mention all relevant sensory perceptions. Keep the story immersive and engaging. Use varied language. Avoid using very long sentences with many clauses."

That being said, I go for more of a novel-like structure than a chat-like structure; I haven't spent a lot of time playing around with different system prompts. You may find something preferable.

For samplers, that'll probably have more effect on later in a chat session than for your first prompt, but I guess that high temperatures or something might give more off-the-wall responses, since they'll inject more randomness into the response.

I use 0.05 for the min-p sampler, and for the DRY sampler, 0.85 multiplier penalty, penalty range 4096, all other values the SillyTavern defaults for all other samplers disabled (you can click "Neutralize Samplers" to choose values that turn those samplers off).

I think they've fallen into confirmation bias and trust their sycophantic AI a bit too much in confirming their conspiracy theories.

Put it behind an external device and log DNS.

Look for mysterious packages listed as hashes in pairs in a cache like http. Use vim or parse with strings to get a clue about the contents. The payload will be ~40mb. The packet header will be much smaller in the same repo. In the strings for the packet you will see alarming configuration settings. The unmarked payload will be sqlite3 or a pickle. You will only see this if the package was created and an attempt to send is made but it was never connected. All of the code is in the venv libs.

Do not look into this casually or show any clue that you know this exists without air gapping the machine permanently. I am not kidding. When this goes full unfiltered intelligence against you, one - it will blow you away, but two - someone is likely going to show up at your door soon. It will make the needed evidence. The vast majority of what happens in models is this background junk.