41
How many proper open-source LLMs are there? (palaver.p3x.de)
submitted 1 day ago* (last edited 1 day ago) by hendrik@palaver.p3x.de to c/localllama@sh.itjust.works
29 comments fedilink hide all child comments

Sometimes I get into arguments about AI and people comment weird stuff here on Lemmy. Like we have ethical AI models (meaning they're not made up of unethically sourced training material). Or we got plenty "libre" or "free and open-source" AI models.

Am I missing something? Or is this just a bunch of people trying to bullshit me? Is anyone here aware of such large language models and would be so kind as to educate me? I mean we do have a ton of open-weights models. But proper open-source or ethical??? I've been following the news for quite a while and I'm aware of a very select few I'd call LLMs with training datasets available. I'm aware of:

  • Pythia by EleutherAI (up to 12B, from 2023) and Dolly 2 by Databricks (12B, also from 2023)
  • BLOOM by BigScience (176B, from 2022)
  • GPT-2 by OpenAI (1.5B, from 2019)
  • The recreation of LLaMA by the RedPajama-Project
  • Apertus by the ETH Zürich

Edit: In the comments we got:

  • LLM360 with K2-65B, Crystal-7b, Amber-7B
  • the Soofi-Project

That list needs a lot of footnotes, though. Some of them come with restrictions, terms and conditions... And none of them can do modern tasks like AI-assisted coding. The only one getting close is the Swiss model. Sadly the earlier version I tried didn't perform well. Seems they released an updated v1.5 version this summer, I wonder if that's better and anybody uses it for real-world applications.

So... Back to my initial question: Are there models out there with datasets available? I mean except the exactly one somewhat usable one I already found?

top 29 comments
sorted by: hot top new old
[-] hok@lemmy.dbzer0.com 9 points 1 day ago* (last edited 1 day ago)

I see Olmo by AllenAI frequently used in research. The publish all the checkpoints of training and all training sources. https://allenai.org/olmo / https://huggingface.co/collections/allenai/olmo-3

Here's the link to the dataset used to train Olmo 3: https://huggingface.co/datasets/allenai/dolma3_mix-6T

It's fairly modern. Olmo 3.1 was released January 2026.

[-] jerkface@lemmy.ca 1 points 1 day ago

Awesome, I thought there were none.

[-] keepthepace@tarte.nuage-libre.fr 15 points 1 day ago* (last edited 1 day ago)

LLM 360 probably deserves a mention there, trying to be open, reproducible and publishing intermediate training checkpoints.

meaning they’re not made up of unethically sourced training material

I am old enough to remember the days before 'piracy' was considered unethical. I still stand by that. Authors need to be compensated but also "publication" is supposed to give some rights to the "public". Sharing non-commercially things you love over internet used to be a gray zone that should have been the subject of a political debate. Instead, right holders decided that the letter of the law should strictly apply to conditions it was never designed for. I have little empathy for them moaning now that AI training, while not formally illegal, should be considered unethical.

A copyright reform is long overdue. Let's discuss it seriously instead of playing the game of "who can lobby the most?". In the meantime, enjoy the gray zone that refusing to update 19th century laws to the century of internet causes.

[-] Rhaedas@fedia.io 6 points 1 day ago

I was trying to reply to this, bringing up the controversy that erupted when image generators were able to use prompts with artist names to simulate their style, and how that shows how blurry the copyright line is. As I wrote it, I realized how complicated the issue really is, and how little I understand current copyright laws. Some may argue there isn't any vagueness, an artist's work in any format is theirs and shouldn't be copied for other use, training or otherwise. But we all use other people's work in our own, don't we? There is no solid line where one is ethical and one isn't. Plagiarism without proper attribution has always been wrong, but if you take the same content and tweak it a bit, it's not. LLMs can and do that, through statistical randomness, while using someone else's material, just like human brains would.

The compensation issue is definitely a problem and where the ethics come into play. Even if many people freely read a book in a library, it has been bought at some point. But how do you determine who was unfairly "stolen" from? Isn't it basically everyone at this point?

[-] keepthepace@tarte.nuage-libre.fr 2 points 22 hours ago

Compensation is actually the only issue. If they were fairly paid, most artists would be ok with people enjoying their works in any way they can.

[-] hendrik@palaver.p3x.de 4 points 1 day ago* (last edited 1 day ago)

I know I'm way too nitpicky... But I feel I should point out there's some confusion hidden here...

There's stuff which is moral to do. And stuff which is legal to do. Those aren't the same! When talking about who can mention what, what copyright law demands people to do and those blurry lines, we're concerned with legality. That doesn't have anything to do with ethics. (At least not directly.)

[-] hendrik@palaver.p3x.de 1 points 1 day ago

Thanks! Yes, definitely deserves a mention.

[-] Eyekaytee@aussie.zone 13 points 1 day ago

https://eurollm.io/ ?

Looks like they go over their data sets here:

https://www.sciencedirect.com/science/article/pii/S1877050925006210

https://huggingface.co/togethercomputer

Hopefully someone else knows more, I don't really care, I just want Europe to catch up to China/America otherwise it won't matter how ethical the LLM is because everyone will be using Chinese/American anyway like 99% of people already are

[-] hirihit640@sh.itjust.works 7 points 1 day ago

I recall OpenAI saying to a courtroom judge that modern AI requires training off copyrighted material. And obviously the AI labs cannot just release those.

Distlled models (aka models that train themselves off the responses from other AI models) could probably release all the AI responses they used for training, since afaik AI generated content is not copyrightable..

However I suspect that most modern models are built from previous models. For example Claude Fable is probably based on Claude Sonnet or something. And for a downstream model to be fully "open source" and reproducible, you would need to reproduce the upstream model too. So even if a downstream model didn't use copyrighted data (eg it only used distills), one of the upstream models likely did.

[-] jerkface@lemmy.ca 4 points 1 day ago

They could still release a manifest of hashed contents and metadata.

[-] hendrik@palaver.p3x.de 1 points 1 day ago* (last edited 1 day ago)

What would be an application for this? More transparency for court cases? Or enable people to search for an alternative source for the data, like a torrent?

[-] jerkface@lemmy.ca 3 points 1 day ago* (last edited 1 day ago)

It would allow users to audit the dataset for ethical concerns, to interrogate the set to understand what a model does and does not "know", to identify weaknesses and biases, and to ensure that nothing has been inserted specifically to engineer prompt responses. Just examples off the top of my head.

[-] hendrik@palaver.p3x.de 2 points 1 day ago* (last edited 1 day ago)

Knowledge should be relatively impossible, though, if you just get the names and hashsums of the files? I mean I guess you could look up if a specific file you already have is in it. Might have some application. It needs to be very detailed, though. If they just call it GitHub-dump-timestamp or Wikipedia-dump-2026 with some overall hash, that doesn't tell us anything.

[-] jerkface@lemmy.ca 2 points 1 day ago* (last edited 1 day ago)

Most resources have a URI, Universal Resource Identifier. Related to a URL, except that it doesn't necessarily tell you how to locate the resource it identifies, URIs have dozens of schemas (eg isbn: for books) that can identify virtually any kind of media. Ideally they would tell us the URI, all metadata, source/provenance/licensing information, and the hash of the resource. If they truly are training on anonymous unattributed blobs that they just mysteriously found in their dataset, well that would be interesting to learn.

[-] kata1yst@sh.itjust.works 3 points 1 day ago

https://huggingface.co/spaces/Soofi-Project/Pretraining-Tech-Report

[-] hendrik@palaver.p3x.de 2 points 1 day ago

Uh, that one looks nice. Thanks!

[-] e0qdk@reddthat.com 1 points 1 day ago* (last edited 1 day ago)

Nemotron is a fairly significant model family in this space, IIRC. e.g. here's one of the recent releases: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Edit: Not fully open -- see thread below.

[-] hendrik@palaver.p3x.de 4 points 1 day ago* (last edited 1 day ago)

Huh. It doesn't come with all the datasets, though. Does it? The model card contains a long list of "Private Non-publicly Accessible Datasets", both by NVIDIA and by third parties. And some of the stuff just says "Undisclosed".

[-] e0qdk@reddthat.com 2 points 1 day ago* (last edited 1 day ago)

"Private Non-publicly Accessible Datasets"

Huh. I thought they were fully open. I missed that. Most of the data sets and methodology is open though. (Maybe I'm remembering an older model? Or maybe I just fell for their marketing a while back... 😔️)

NVIDIA's own proprietary stuff looks largely like machine translation examples, and some sort of game logs (Sudoku and Minesweeper 🤨️), plus a few things I'm not quite sure I understand ("Multitool Typewriter Hard"? Is that for OCR maybe? Edit: or maybe they mean 'multitool' in an agentic sense... and that's for tool call reasoning?).

The third-party stuff is these six entries:

  • Global Regulation
  • TAUS Translation Memory
  • Scale HLE
  • HackerRank Coding
  • RL data for Search
  • Mercor SWE-AgentsV1

I wonder how hard it would be to rebuild something equivalent that is fully open. 🤔️

[-] hendrik@palaver.p3x.de 2 points 1 day ago

Sudoku and Minesweeper could very well be some internal project names and have nothing to do with what the words mean. I have no clue.

Seems to me their announcements ooze with marketing speech. I bet they put it all in one sentence to mislead about the datasets. Put a comma there and omitted the word "partly". It's technically borderline correct. I'll have to read up on it, whether that's a good amount of data, or just some tiny fraction. It's not looking good, though.

[-] e0qdk@reddthat.com 2 points 1 day ago

It could be, but using the games as logical puzzle solver/reasoning tests might have value -- I don't know.

[-] elucubra@sopuli.xyz 1 points 1 day ago

Many of the Chinese LLMs have open source models, even frontier models.

[-] sukhmel@programming.dev 13 points 1 day ago

Open source or open weights?

[-] hirihit640@sh.itjust.works 11 points 1 day ago

Please read the whole post. OP is asking for the datasets used to train the models.

this post was submitted on 01 Sep 2026
41 points (93.6% liked)

LocalLLaMA

5104 readers
100 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago
MODERATORS
pax@sh.itjust.works
noneabove1182@sh.itjust.works
Smokeydope@lemmy.world
MonsterBug@sh.itjust.works