by perching_aix 6 hours ago

The companies are 100% aware, yes, and so they did make quite a few changes over the years.

Current trend is that the models will try to explicitly steer you towards "asking better questions from your medical provider", rather than providing diagnoses. They do also evaluate whether something can actually be established rather than just listen and nod along. And so the "you must have very little faith in the intelligence of their members" goes right back against these failure mode ideas.

Now of course, given a sufficiently desperate person, they can probably torture anything they want to hear out of these models. But so can they out of actual people, so that's kind of a high bar. When you get to the point where people are willfully misreading a given piece of text, bets tend to be rather off.