this post was submitted on 19 Nov 2024
8 points (78.6% liked)

Futurology

1853 readers
90 users here now

founded 1 year ago
MODERATORS
voidx@futurology.today
Lugh@futurology.today
Espiritdescali@futurology.today
AwesomeLowlander@futurology.today
8
A small study found ChatGPT outdid human physicians when assessing medical case histories, even when those doctors were using a chatbot. (www.nytimes.com)
submitted 1 month ago by Lugh@futurology.today to c/futurology@futurology.today
7 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[โ€“] drspod@lemmy.ml 3 points 1 month ago (1 children)

They studied 52 doctors responses to standardized (read publicly available online) cases written in front of them.
...
Then they ran 3 trials solely with LLM and find that these were significantly better.

How do they know that the answers to the "standardized" publically available case studies were not in the training data of the LLM? Isn't it extremely likely that they were?

[โ€“] Darrell_Winfield@lemmy.world 3 points 1 month ago

It's very likely that they were in the training data. I forgot to include that as a point. Unfortunately, though, that's a very difficult variable to control in the LLM research.