Most Health Questions Aren't Asked During Clinic Hours

OpenAI reported in January 2026 that more than 40 million people ask ChatGPT a healthcare question every day, that about one in four weekly users asks something health-related, and that most of those conversations happen outside the hours a clinic is open. Among people who use AI for health information, 84% use ChatGPT and 49% use Gemini.

So picture the moment. A woman, eleven at night, with a symptom she's been ignoring or a test result she doesn't understand, asking something that will answer straight away and sound reasonable.

The obvious question is whether the answer is any good.

What does the research say?

It depends on the question, and the pattern is consistent: these systems are better at explaining than at judging.

On menopause, a Mayo Clinic team put 35 questions to four AI systems and had four blinded experts score them. Published in the journal Menopause in 2026, it found ChatGPT 3.5 answered 70% of patient-style questions correctly, ChatGPT 4.0 managed 60%, and Gemini 30%. On harder clinician-style questions the best score was 67%. Every system's answers were rated difficult to read.

Across women's health more broadly, a December 2025 benchmark built by seventeen clinicians and pharmacists tested thirteen models on questions chosen because a model had already got them wrong. Failure rates were around 60%, with the best model passing just over half. The weakness shared by every model was missing when a situation was urgent. It's a deliberately hard test, so treat the headline figure as a stress test rather than everyday performance. It's also a preprint, not yet peer reviewed.

And sometimes they do well. A study published in July 2026 found three AI systems accurately identified contraindications for hormonal contraception against WHO criteria. In a blinded comparison, 43 clinicians found ChatGPT's answers on common PCOS questions clearer than a standard patient resource.

Why can't she tell the difference?

Because the wrong answers sound exactly like the right ones.

A study across twelve models and five medical specialties asked each model to answer clinical questions and rate its own confidence. The gap between how confident they sounded when correct and when wrong was between 0.6 and 5.4 percentage points. There's no tell. Same tone, same length, same steadiness.

What does that mean for brands?

It shifts the question. If a model can't reliably tell when it's wrong, what decides the answer is what it read.

That makes AI visibility a supply problem, not a placement problem. Nobody can buy a spot in an AI answer, and citation patterns swing wildly between assistants and months. What you can affect is whether accurate, specific, well-sourced answers about your category exist in the places these systems look.

Three things worth doing:

• Write down 15 to 20 questions a customer would really ask, run them through the assistants your customers use, and note what comes back and what gets cited.
• Publish the specific answers, not the category ones. The narrow questions are where these systems struggle.
• Be present where the source material is made, which usually means the communities where those questions get asked first.

That last one is what we built the Ayana Agent for, with every reply approved by a person on the brand's team. What we won't do is promise placement in AI answers, because nobody controls it.

Sources: OpenAI health usage data (January 2026); EMARKETER US Digital Health survey (January 2026); Karam and colleagues, Menopause 2026;33(5):515-520; "A Women's Health Benchmark for Large Language Models", arXiv 2512.17028 (preprint, December 2025); Psilopatis and colleagues, Journal of Clinical Medicine (July 2026); blinded PCOS comparison (April 2026); cross-sectional confidence study, JMIR.

This isn't medical advice. No AI answer, even a correct one, replaces a clinician.