LocalLLaMA

What I'd advise you to do is something like this:

python3 koboldcpp.py --unbantokens --contextsize 8192 --stream models/mythomax-l2-13b.Q4_K_M.gguf

If you're unsure about your settings, just use mirostat:

python3 koboldcpp.py --unbantokens --contextsize 8192 --usemirostat 2 5.0 0.1 --stream models/mythomax-l2-13b.Q4_K_M.gguf

You can download MythoMax 13b ~~and there's an 33b~~. It's probably not 'the best' model out there, but I think a pretty solid one. It isn't super picky with the settings and does a variety of tasks well. [Edit: Watch out when using the 33b. I think it's broken. It generates lots of repetetive words for me.]

Notes: You probably need to add your --usecublas normal to that.

Fiddling with the contextsize comes with many caveats. Skip this parameter if you don't need it. Or learn how scaling works and which model needs which of the two methods with what kinds of numbers. Letting koboldcpp do the ropeconfig itself, makes it assume a few things that are probably wrong. You oftentimes have to set the 'ropeconfig' explicitly when doing any funny stuff. (Use a normal model and have it in gguf format. That's most likely to get the automatic configuration doing something sane.)

Regarding --unbantokens: You can delete that from your commandline and it will make the EOS token disappear, hence making the model unable to stop generating text. But, it'll also make it write an end to the story, following it up with "In the end ..." and then another "And together, they lived..." and then following it up with more and more repetitive stuff because the story is over yet you made it unable to stop.

With 'unbantokens' and once it won't continue on its own: Tick "Allow Editing", append a line break (or two) and hit 'Generate' again.

You can set the 'Amount to Generate' to something you like. You don't need to put in my value.

Getting your model to work: It is a LLaMA 1 based model. So it originally had 2048 tokens context. It is in the old ggml format, so koboldcpp probably doesn't know that. And judging by the name, I guess it was trained to have an 8k context with linear scaling. So I'd guess the following parameters are correct. And you absolutely need to specify it explicitly when using a model like this:

--contextsize 8192 --ropeconfig 0.25 10000

I don't have the time to fiddle with the parameters to get them right, so I just used mirostat. Feel free to instead try a few of the presets and play with the numbers. Maybe see if someone commented a good configuration for your favorite model somewhere.

python3 koboldcpp.py --unbantokens --contextsize 8192 --ropeconfig 0.25 10000 --usemirostat 2 5.0 0.1 --stream models/WizardLM-Uncensored-SuperCOT-StoryTelling-30b-superhot-8k.ggmlv3.q4_1.bin

Regarding the prompt format: I'm not sure what a 'storywrite' model expects. You're giving it an instruction, but don't write it in any of the usual formats. That's most likely wrong in any case. If these models are trained with plain stories (which I don't know), you have to start with the story and let it autocomplete. You can't give it an instruction and hope for the best, it'll lead to subpar results. If you give it instructions, do it correctly. Find out the correct prompt format for that specific model.

But maybe just don't use this model. These SuperHOT or 8k/16k/... models are from a time where we didn't have NTK-aware scaling yet. Nowadays you might just use the latter together with a 'normal' model. And as long as there isn't a reason against it, maybe use a model based on Llama2 which has twice the context to begin with (compared to the old LLaMA).

There are many other models like that out there. In fact there are many many models out there. And many of them aren't very good or are just obsolete. Don't pick one based on the fancy name. Choose one other people use. Rely on word of mouth. If it's good, other people will use it and maybe also talk about it, recommend good settings and how to prompt it. For popular models, you'll get that information on top.

https://huggingface.co/models?sort=modified&search=storywrite

Judging by the output you can see on my screenshots, the MythoMax writes way better stories anyways.

Here are some more prompt ideas:

### Instruction:
Write an award winning childrens book about a stray cat finding a new home in a small european village. Make sure the narration and characters are vivid, detailed and generally warm and likable.

### Response:

### Instruction:
You are the worlds greatet writer. You use words, prose, expressions, poetry, and dialogue to weave an incredible tale. Write in the style of book chapters. Write a fairy tale for children about a stray cat finding a new home in a small european village.

### Response: