LocalLLaMA

2267 readers

2 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago

MODERATORS

pax@sh.itjust.works

SkySyrup@sh.itjust.works

noneabove1182@sh.itjust.works

I want to train / fine-tune a model (sh.itjust.works)

submitted 1 year ago by Naked_Yoga@sh.itjust.works to c/localllama@sh.itjust.works

1 comments fedilink hide all child comments

I want to train, or more likely fine-tune, a model on about 20 years worth of email and text data that I've collected.

The goal would be to train it how to respond like me in simple cases.

It's there a particular base model I should start with?

I'm also interested in anyone's experience in doing this kind of thing themselves.

you are viewing a single comment's thread
view the rest of the comments

[–] ScrawnyStork@sh.itjust.works 4 points 1 year ago

I'm inclined to fine tune/make a Lora for Llama (2? I'm not sure how well it's supported yet), you definitely won't want to train from scratch. The big thing seems to be formating the dataset properly, although I haven't actually done any of this myself yet. this blog seems to have a few good articles about it that you might be interested in reading.