It would be interesting to pass a law that if someone has a fatal medical condition science doesn't know how to cure, they are legally allowed to follow an AI advice. Maybe with some "best practice" recommendations, such as some expert doctors specifying the prompt. Maybe with the requirement that their progress will be monitored, and used as more data for the AI.
(Of course, mortality itself is a fatal medical condition that science doesn't know how to cure yet.)
With LLM-based AI passing benchmarks that would challenge people with a Ph.D in relevant fields, I'm left wondering what they can do for real-world problems for which nobody knows the correct answer, such as how to treat potentially fatal medical conditions with no known cure.
Are we at the point where AI can do better than curated medical references intended for doctors, such as UpToDate, at coming up with treatment plans for people with hard-to-treat and frequently fatal medical conditions, like Jake Seliger who died of squamous cell carcinoma, DF who delayed his death from fatal familial insomnia, or my late wife, who suffered from calciphylaxis before her death?