11
small LLMs are not totally worthless (lemmy.ml)
submitted 3 days ago by leanleft@lemmy.ml to c/localllama@sh.itjust.works
25 comments fedilink hide all child comments

alot of really small SLMs seem to be too small to even function properly.
however, there may be some unconventional uses for some of them.

one usecase keyword generation. heres what gemma-3-270m-it-qat-Q4_0 gave me

prompt: keywords similar to "construction"

Construction activities
*   Construction design
*   Construction equipment
*   Construction management
*   Construction materials
*   Construction methodology
*   Construction organization
*   Construction planning
*   Construction process
*   Construction project
*   Construction project details
*   Construction project management
*   Construction project management activities
*   Construction project management assessment
*   Construction project management checklist
*   Construction project management document
*   Construction project management evaluation
*   Construction project management framework
*   Construction project management methodology
*   Construction project management plan
*   Construction project management project
*   Construction project management report
*   Construction project management review
*   Construction project management schedule
*   Construction project management software
*   Construction project management strategy
*   Construction project management system
*   Construction project management tool
*   Construction project planning
*   Construction project scope
*   Construction schedule
*   Construction strategy
*   Construction work
```
                
90 TPS (but got stuck in a repeating loop)          
            
i think theres alot of other uses. but they arent really LLM answer that people usually reach for an LLM.
top 25 comments
sorted by: hot top new old
[-] hendrik@palaver.p3x.de 8 points 2 days ago* (last edited 2 days ago)

I don't think these are "keywords similar to construction"... That'd be something like "building", "contractor", "builder", "construction company", "remodeling", "roadworks", ...

And your model didn't really find any of them. I think that output is next to worthless. If you're a construction company and use this list for your SEO, nobody is ever going to find out you have an excavator and hire you.

[-] leanleft@lemmy.ml 1 points 2 days ago* (last edited 2 days ago)

well different prompts, at different temps, diff fine tunes , diff quants and quant quality, rag, and lora , are all going to give different results. this is just a single prompt using the model as it is pre-packaged.
you would have to test by running ~100 prompt variations and determine A)no useful value B) unidentifiably wrong answers (primarily if your requirements are unforgiving) C) no predictable/reliable performance whatsoever.
if thats turns out to be the case and its unacceptable..
A) scale up the model size slightly B) spend cash on hardware or service to run the appropriately sized model that accomodates the scale that you need(which could be prohibitively expensive).

[-] hendrik@palaver.p3x.de 2 points 2 days ago* (last edited 2 days ago)

Sure. I guess what I wanted to say is: Your example doesn't really support your claim. It kinda goes the opposite way. (But doesn't prove that, either.)

I think it's a bit of a double edged sword. Efficiency and small models are great. We also have some NLP tasks which don't really require big LLMs. But once you try to use AI, apply it to some real world problems, or embed it into some context... you'll notice performance issues really add up. I tried to use some 2B parameter models, spent some time tuning everything, built more logic and failsafes into the framework. But at the end of the day I scrapped that hobby project, because the model kept being very limited in its abilities. And issues kept piling up to the point it was utterly unusable.

A 270M model is barely able to output coherent sentences in one language. I don't think there's a lot of world knowledge stored inside, so you can ask it questions and expect it to do anything.

But keyword extraction is way easier. I don't think we need LLM inference for that. There are several NLP approaches to do it without a lot of compute. They're probably way better and more efficient than asking a chatbot to generate some.

[-] leanleft@lemmy.ml 2 points 2 days ago* (last edited 2 days ago)

im not aware of any nlp tools that generate new keywords(without attaching to an existing database).
i'm not advertising that all LLMs do what mainstream users expect it to do.
there are many applications, just for this single example. for other uses or the exact application.. you need to figure that out yourself.
it's true that larger modern LLMs are cool. but this post isnt about that.

[-] hendrik@palaver.p3x.de 1 points 2 days ago* (last edited 2 days ago)

I think we have several text extraction methods... You can look up the distance between embeddings... We have models which can assign keywords to images, topic modeling (and keyword extraction)... It's not entirely clear to me where we're going with this... But I'd say if you're not using a text corpus or "database" or working on text, you'd use word embeddings and just look up the distance between them. Very close ones will be synonyms. Next will be similar ones. That's way less effort than doing inference. And it'll generate the new keywords just from the embedding space itself. There's of course also traditional lexical databases. Something like a thesaurus or WordNet comes with least effort.

Coincidentally, I think the BERT model is still being used for tasks like this. It's an encoder model, but with a similar amount of parameters like what you mentioned in the post. But it's not being used for text generation in this way.

I mean I'm not really disagreeing here. We have a lot of small machine learning models and they're very useful. I just can't come up with generative-AI-like tasks for LLMs at that size category.

[-] DrBob@lemmy.ca 10 points 3 days ago

What is the point of that?

[-] abcdqfr@lemmy.world 10 points 3 days ago* (last edited 3 days ago)

Wrap it in a harness. Dump antipatterns. Keep the rest. Call it a ratchet. Loop that. Fail fast. Refine. See what sticks. Design a better harness. Augment that harness. If nothing good comes out, that's data and that is good.

[-] kingofras@lemmy.world 14 points 3 days ago

I aged a month just reading this

[-] venusaur@lemmy.world 7 points 2 days ago

Technologic. Technologic.

[-] ComradePenguin@lemmy.ml 3 points 2 days ago

I am trying to fully understand this. Can you please elaborate further? Are you thinking about a heuristic based harness?

[-] abcdqfr@lemmy.world 2 points 2 days ago

Not sure about keyword generation exactly as OP intends to apply this, but short simple code that needs to be validated on output seems like the better application. Models are tiny and fast, so mmap and run in parallel and race to what passes validation first. Each agent gets a git branch. Losers generate the anti patterns. Human or cheap tokens integrates the results from corpus back into harness. Next generation.

[-] ComradePenguin@lemmy.ml 2 points 2 days ago

Thanks! Will explore this further.

[-] abcdqfr@lemmy.world 2 points 2 days ago

There's some really close prior art here https://arxiv.org/abs/2404.00725

[-] ComradePenguin@lemmy.ml 2 points 2 days ago

Cool paper. This makes me want to try to experiment with this. I have a 8GB VRAM card and 32GB DDR4.
So not ideal, currently I just pay for inference. Trying to get a model to punch above its weight would be kinda interesting to experiment with.

[-] otter@lemmy.ca 1 points 3 days ago

Using it to tag content can make it easier to search later, bookmark management for example

[-] DrBob@lemmy.ca 2 points 2 days ago

So what's the advantage of a generated list over "construction"?

[-] otter@lemmy.ca 1 points 2 days ago

Mostly the ratio of effort to convenience

I wouldn't bother thinking up tags for every article I bookmark

However it is nice if a local model on the device can generate tags automatically

[-] DrBob@lemmy.ca 2 points 2 days ago

But they seem like iterative synonyms. Hardly unique enough to useful tags for either lumping or splitting.

[-] otter@lemmy.ca 1 points 2 days ago

Oh I misunderstood, I read construction as manually making tags

Yes this particular list wouldn't be that helpful. A different prompt or some post processing would be needed. I'm not familiar with this model either

[-] leanleft@lemmy.ml 0 points 3 days ago

anything you want it to be.
be creative !

[-] DrBob@lemmy.ca 2 points 2 days ago

I can't think of single useful function. You might as well have generated nonsense syllables and invited us to attach them to words to create meaning.

[-] corsicanguppy@lemmy.ca 1 points 2 days ago

one usecase keyword generation.

One use-case: keyword generation.

Glad to see this wasn't written by Ai, anyway.

[-] sleepmode@lemmy.world 3 points 3 days ago

For breaking out some simple but tedious processing tasks in scripts they can help save a lot of writing logic and filtering. But it can be a bit rough… I’m having to experiment more than I’d like.

[-] eleitl@lemmy.zip 1 points 2 days ago

I think the more interesting part that for still affordable hardware you could run something locally that is good enough and do not need to refer to commercial models.

[-] ThorrJo@lemmy.sdf.org 1 points 3 days ago

I will eventually experiment with small models for various text- and document-processing tasks.

this post was submitted on 18 Sep 2026
11 points (65.7% liked)

LocalLLaMA

5164 readers
38 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago
MODERATORS
pax@sh.itjust.works
noneabove1182@sh.itjust.works
Smokeydope@lemmy.world
MonsterBug@sh.itjust.works