We want to give developers agency over their source code by letting them decide whether or not it should be used to develop and evaluate machine learning models.
fuck them seriously; if you want to do that then don't steal the repositories in the first place.
Meanwhile at work we just had a training course that specifically said doing “opt out” instead of “opt in” violates the principle of informed consent.
Notice how a shit load of these people keep turning up to be rapists and pedophiles… They have no shits to give about informed consent. In their mind you don’t even have the right to the same agency they do.
not surprising since the venn diagram of ai bros and rapists is a circle.
Kinda hope it uses my code. It's so terrible there's no doubt it will make the resulting code from the model worse even if the impact is miniscule
I just opted out of all my repos except the God awful ones from middle and highschool. Those are basically prompt poison so fuckem
Modern tech companies love using the rapsist's model of consent.
I never really needed any justification, but ever since AI companies just take stuff illegally, and that openly, it has become my justification to just pirate the shit out of everything.
(Excluding indie games of developers I like).
The best way to opt out is by not using GitHub. Also opts you out of Copilot and a bunch of other stuff.
Already did that some time ago but I am still in the dataset.
Other git hosts are also getting scraped, and have had to implement counters because of it. For example, this is the kind of thing Codeberg shows crawlers. I've even seen people who self-host complaining about getting overloaded because of bots scraping their forge
note that former users would have needed to remove their GitHub data before August 2025 to not be in this dataset
lol my first ever JavaScript project is in the stack. No wonder their models suck so much ass
my commits are the reason we haven't achieved AGI
Thank you for your service 🫡 please keep committing trash code
A decade ago, if someone asked someone working on an Open Source project if they'd be ok with an AI reading their code and learning from it, they'd most likely say "yeah that sounds really cool!"
Somehow the tech-bros have fucked up AI so much that something that should be really cool seems creepy, lame, and nefarious all at once.
A little bit infuriating since huggingface itself requires login to access a large portion of the content on their site
When I hear this company's name I can't get the idea out of my head that it's related to the face huggers from Alien.
if I opt out, will the Roko's basilisk come after me?
Reported for cognitohazard. Delete this immediately.
(I kid, but someone really did report your comment.)
lmao
I think its hilarious that some people seem to actually take this science fiction variation on Pascal's wager seriously.
Much like... -reads notes- Microsoft is via Github... huh....
We want to give developers agency over their source code by letting them decide whether or not it should be used to develop and evaluate machine learning models.
crawled directly from GitHub and built to pre-train code LLMs with full-repository context
Repositories that opted out are removed from the dataset before each patch release.
"agency"
Which AI company will not use v1 which has all of the data but will use later patch releases instead which have less data?
Or just merge it
Wonder how many of those repos contain AI-generated code?
Since they stole my paper on ethics in computer science, maybe the model will learn to act better than its owners
Am I alone in not wanting to put my username into that field? If they don't have it will they then just decide that it's now a good time to scrape it? Or are they going to record that it was searched?
Pretty sure all my repos are MIT licensed for the betterment of everyone, but I'm not on the list! So I guess I'm not good enough, or they are failing to follow the attribution clause of it.
Strange because all of my MIT repos are listed - all the GPL ones are excluded.
Mine are all GPLv3 or forks of other repos and they are listed. I'm sanguine because the license allows for study and if it's good enough for humans I don't see why it's not for clankers.
For me, and a few other projects I checked, it only has non-GPL repos. But it also does not have everything that isn't GPL, despite the repos being much older than the cut-off date. But it does have repos without a license, which they are simply not allowed to copy.
I wonder if those repos have been deduplicated, and one of the forks (on some other person's account) is included instead. Unfortunately you can only search the first 5M records via the website, and I don't have time to play around with the API at the moment, so I could neither confirm nor deny that possibility
My dotfile repo is there and it doesn't have a licence. Meaning it's technically not open source. Didn't stop them
I just followed the opt out link and they're making you open a public issue with a Markdown list with all your repositories you want removed.
Surely there has to be a better way to do this (probably the point).
Also why are they storing 4.71 TB in a Git repo? How are they going to deal with removal requests?
EDIT: It seems like they're manually responding to the issues, wtf? There's a perfectly fine GitHub auth system that they could use to verify everyone's GitHub account / repository ownership
EDIT 2: This is apperently a collaboration between Hugging Face and ServiceNow, why is my companies IT ticketing system scraping all of GitHub?
tbh i'm thinking this alone isn't that bad from an archiver/datahoarder perspective
As long as the datasets are open, it is our best hope. I know it doesn't compensate the people whose work's copyright and licenses have been violated, but I think it's the only realistic hope we've got of getting out of this informational dystopia with a reasonably intact library of humanity's knowledge that hasn't been locked down and/or monetized. The AI scrapers and generators are in the process of burning down the great library of Alexandria that the Internet had become, and we are already starting to feel its loss. We cannot stop the wave of toxic pollution that is spreading through all our digital content now, but the archives from before this apocalypse started will become the most valuable thing humanity has ever produced. This is information war, and we are losing.
They’re not able to scrape private repos, surely?
Microsoft has access, and everything is for sale. So, maybe? Probably? 🤷
Noteworthy: They crawled only the default branch HEAD and inlined all source content.
- The file contents are included inline. The decoded UTF-8 source text is embedded directly in the dataset, so it is fully self-contained — you can start training the moment the download finishes.
- It reflects the state of GitHub in August 2025. The corpus is a direct crawl of GitHub repositories at their default-branch HEAD, capturing roughly two additional years of open-source code compared to The Stack v2.
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev