55
I built a programming language that LLMs can't use
(blog.picheta.me)
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev
This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.
Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn't been trained on. That's basically an ad for Claude Code..
Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.
And that's without any of this being in its training set. I wonder if they realized how big of a self own that is.
As well-intentioned as this might be, I doubt they understand how trained models work.
How the hell is this a "self own"?
The premise was that this language is significantly different from other languages in terms of being understood and replicated by an LLM.
The result showed that an LLM does exactly as well with it as you would expect.