You should not ask an AI for medical advice, and the reason why is simple and terrifying: the information may have been vetted by a person with zero medical expertise who was given 15 minutes to do it. The internal processes of AI training reveal a system that is fundamentally unsafe for handling sensitive and high-stakes topics like health.
One AI trainer, a writer by profession, recounted the “haunting” experience of being tasked with editing information on chemotherapy options for bladder cancer. She was acutely aware that a real person in a moment of crisis might rely on the information she was entering. Her anxiety was justified; a mistake could have devastating consequences.
This is not an isolated incident. In late 2024, a major tech company explicitly told its contractors they were no longer allowed to skip prompts on topics like healthcare due to a lack of expertise. This policy institutionalizes the practice of having non-experts curate medical information, a fireable offense in any reputable medical publisher but standard practice in AI.
The AI’s confident, authoritative tone can be dangerously misleading. It presents information as if it has been verified by experts, but the reality is that the “expert” may have been a frightened, unqualified contractor working against a ticking clock. When it comes to your health, the convenience of AI is not worth the risk of its unreliability.