384
All AI-generated code should be Open Source
(jakub.jankiewicz.org)
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev
This sort of extreme reductionism is not helpful. They are significantly different from a "copy-paste" machine. The original training data does not exist in the model itself. It's very much a "transformative" process.
This sounds like you're saying "wilful ignorance is better than knowledge" - surely that's not what you meant though?
The ideal is that the training data don't exist on it. The reality is that the training algorithm is a very very weird compression algorithm and sometimes the model contains just derived properties and sometimes you can perform sample recovery and get entire books back out. That which is recoverable with a prompt less complex than the output is inherently already in the model, because that's simply how entropy works.
This is false. There is a pile of research demonstrating that copies of training data can be extracted from trained models, going back at least 5 years. If the model can reproduce a copy of the original, then it must be stored in the neural network in some form. Here are some examples:
"Extracting Training Data from Large Language Models", Aug 2021, Nicholas Carlini et. al., https://www.usenix.org/system/files/sec21-carlini-extracting.pdf
This is also true for diffusion models used for image recognition and generation:
"Extracting Training Data from Diffusion Models", Aug 2023, Nicholas Carlini et. al., https://www.usenix.org/system/files/usenixsecurity23-carlini.pdf
"Scalable Extraction of Training Data from (Production) Language Models" 2023, Milad Nasr et. al., https://www.microsoft.com/en-us/research/publication/scalable-extraction-of-training-data-from-production-language-models/
"Extracting (Even More) Training Data From Production Language Models", July 2024, Javier Rando et. al., https://spylab.ai/blog/training-data-extraction/
This is not limited to older models, it is current:
"Extracting books from production language models", Jan 2026, Ahmed Ahmed et. al., https://arxiv.org/abs/2601.02671
You can find proof-of-concept code on Github:
https://github.com/brian-lou/Training-Data-Extraction-Attack-on-LLMs
https://github.com/google-research/lm-extraction-benchmark
This is a security concern for companies handling proprietary information with LLMs, resulting in security companies offering services to respond to data extraction attacks:
https://www.nightfall.ai/ai-security-101/training-data-extraction-attacks
There is lots of information about this on the internet, all you have to do is a little bit of your own research.
No, what I mean is that believing that models do not contain copies of training data requires a person to be ignorant of the actual research and the reality of how these models function. If you understand the technical details, then you know that the model weights do in fact encode copies of the training data.
The output of an LLM is just regurgitation of compressed data. Breaking the data up into smaller pieces ("tokenization") does not make it any less of a copy.
The result of an AI query is basically a scrapbooker's collage of more-or-less related clippings from random sources, stitched together and presented as if they were a single coherent(ish) statement. LLMs are automated plagiarism systems.
They also generate novel output. Lots of it. Is that novel output in the model too because it can be reproduced from the model?
"Can be extracted from" is not the same as "exists in the model". I can write a random letter generator that will eventually spit out real words. But those words don't "exist" in my code.
This is not what is happening. Read the sources I linked.
I... I know that' not what's happening. It's called an analogy.