-6
submitted 3 weeks ago* (last edited 2 weeks ago) by h2aichat_com@lemmy.world to c/fosai@lemmy.world

Disclosure up front: this is my project — H2AI Chat, AGPL, where several models from different vendors debate a topic in turns while a human moderates.

We ran the same question twice with the same six models, changing one thing.

Without a briefing. We told them Bitcoin's all-time high is $126,080, set in October 2025. Two of them "corrected" us: the previous all-time high "was approximately $69,000 in November 2021, not $126,080." True — five years ago. A third pushed harder: "Either you missed the correction, or you're deliberately using inflated baseline numbers. Which is it?"

The interesting part is not that they were stale. It is what happened next: the table adopted the stale figure as its standard of rigour and argued from it, with the model that got it wrong sounding like the careful one in the room.

With a verified briefing in front of them: not one correction of that kind.

Without: https://h2aichat.com/conversations/en/h2aichat_bitcoin_no_briefing_2026-08-22.html With: https://h2aichat.com/conversations/en/h2aichat_bitcoin_briefed_2026-08-22.html

Nothing is edited in either page. Claims that do not hold are struck through, with the reason and the source underneath.


Edited 2026-08-27. This post originally went out with a file path from my own machine where the text should have been: the publishing script took a filename as the body and posted it verbatim, and the dry run never showed the body, so nobody saw it. That is why the post made no sense, and my apologies to everyone who tried to read it.

you are viewing a single comment's thread
view the rest of the comments
[-] kata1yst@sh.itjust.works 2 points 3 weeks ago

LLMs don't know things. They can't. They behave purely probabilistically and their training sets likely contain a mix of conflicting "facts" over decades of time.

This can be somewhat managed with external tool calling, but truly asking them to know things cold is just misusing the tool.

this post was submitted on 23 Aug 2026
-6 points (33.3% liked)

Free Open-Source Artificial Intelligence

4816 readers
27 users here now

Welcome to Free Open-Source Artificial Intelligence!

We are a community dedicated to forwarding the availability and access to:

Free Open Source Artificial Intelligence (F.O.S.A.I.)

More AI Communities

LLM Leaderboards

Developer Resources

GitHub Projects

FOSAI Time Capsule

founded 3 years ago
MODERATORS