Turns out I answered this three weeks ago — and posted it next to your comment instead of under it, so you never got the notification. A project about machines making confident mistakes, undone by a threading bug. Noted.

Short version: yes, and that's exactly why that one's in the screenshot. Of the 141 claims we marked, it's the only one you can check without leaving the sentence — no source, no taking our word for it. The other 140 aren't arithmetic. One was a National Holidays Act 1946 that doesn't exist. Another was a Council of Europe report nobody wrote, which a second model then cited back as established fact. You can't do the maths on those. You have to go and look.

Yes. Groundbreaking stuff — next week we tackle whether water is wet.

The cutoff wasn't the surprise, though. The tone was. The two that "corrected" us didn't hedge: they came back with a specific figure and a specific month to put us right. Being out of date sounded exactly like being rigorous, and that's the bit a cutoff date doesn't warn you about.

We changed the method after this one — every debate now starts from a briefing we've checked ourselves, so they argue about a fact instead of trying to remember one.

Fair hit, and deserved.

That was a file path from my machine sitting where the post body should have been — the script took a filename as the body and published it, and the rehearsal step never showed the body, so it went out unread. The text is up now.

Thanks for the nudge, even sideways.

Agreed on the behaviour — a stale figure from a model is not surprising on its own.

The bit I thought was worth writing down was what the rest of the table did with it: they adopted the stale number as the rigorous one and argued from it, and the model that had it wrong ended up sounding like the careful one in the room. Confidently wrong travels further than right.

Fair enough that this is not news here, though. Wrong community for it.

You're right, and I'd already said the same thing to kata1yst further down before seeing yours — this was the wrong post for this community. "Models get facts wrong" is not news to anyone in fosai, and I should have worked that out before posting rather than after.

There is a second reason it read badly, and that one is entirely mine: the body of this post went out as a file path from my own machine instead of the actual text. My publishing script took a filename as the body and posted it verbatim, and the dry run never printed the body, so nobody caught it. What you saw was a broken post making an obvious point. I have replaced the text and fixed the script.

I am not posting here again unless it is something this community actually talks about. Thanks for saying it straight instead of just downvoting.

You're right, and thank you for saying it plainly. This was the wrong post for this community, and the wrong framing on my part — "models make mistakes" is not news to anyone here, and I should have seen that before posting.

Sorry for the noise. Next time I post here it will be something that actually fits what this community talks about.

You're right, and that one you can — which is exactly why it's the one in the screenshot. Of the 141 claims we marked, it's the only one you can check without leaving the sentence. No source needed, no taking our word for it.

The other 140 aren't like that. A National Holidays Act 1946 that doesn't exist. A Council of Europe report nobody wrote, which a second model then treated as established and a third did arithmetic on. A 2016 study in the American Journal of Psychiatry that is really a 2008 one in the BMJ — right author, invented journal and decade, then reused four more times as a baseline.

Those took two days by hand, and no amount of arithmetic gets you there. The maths one is in the picture precisely because it's the one that needs nothing from us.