To help users of my app, it was important to document the details around the “how it works”. the docs were easy enough using docusaurus + AI. i can then index that data (like used in the docusaurus search). i framed this as like a chatbot so that it can be intuitive for users who have questions. at this point it was really just a fancy autocomplete.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and hopefully better suggestions)? im using WebLLM on a browser and it seems to work “reasonably” well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb… they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can “show me the bubble-sort in rust”, but i want something “dumber” and smaller. something can can take advantage the data ive already indexed. something that can respond “quickly enough” for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

are there any models out there you can suggest for me to investigate and compare?

note: the project is unstable, experiemental and there will be breaking changes. its far from finished. im just trying things out and this approach with LLM’s might just be a waste of time. if you want to test it out, you can try: https://app.glitr.io/ai