There is a benchmark out there that hints at this phenomenon: https://www.scbench.ai/ The idea is to track how well the model modifies code over multiple change requests. No surprise for readers here that the LLM will bitrot your code quickly, and the odds of it doing so is worse than a coin toss.

For anyone reading through the article or watching the YouTube video: The reddit post quoted towards the end is actually a cloaked advertisement for "coderabbit".

I didn't notice until someone in the reddit comments pointed it out, when it became obvious.

Not that it changes anything about what the article conveys, but just as a note to give context to that reddit post and what the motiviation behind writing it was.

2022 called. it wants its models back.