Introducing the Global Health AI Benchmarks

Introduction

When the Indian non-profit Myna Mahila tested its sexual and reproductive health chatbot with an LLM judge, the judge failed one of the chatbot's answers for calling the user didi, a word common in colloquial Hindi that means elder sister. Yet when it scored the same answer for empathy, the same judge praised the use of didi as culturally resonant.

The contradiction is small, but it points to a bigger problem. Health organisations around the world now use AI to talk to users, support health workers and help doctors reply to patients. But most checks of these tools rely on general AI benchmarks, which test models on questions written by others, often in a different language and setting. A high score may say little about how a tool serves a Kenyan mother texting in Swahili, or a nurse midwife in India asking for guidance in Telugu.

Earlier this year, The Agency Fund and Endless Health launched the Global Health AI benchmarking initiative with 2 goals. The first was to help health organisations build their own benchmarks: real cases from their services, scored against standards set by their own experts, so they could see how well different AI models do for their use case and where today's leading models fall short. The second was to bring together what the 9 teams learned and share the patterns they had in common with others in the field.

Altogether, the work spans 25 languages and 8 countries. We gave grants and held regular working sessions. The benchmarks, and the choices behind them, were owned by the organisations.

The benchmarks

The 9 organisations' AI tools cover 4 areas of health care. Each name links to a full summary of that team's benchmark.

Health helplines that patients chat with

• Myna Mahila (India): Women in low-income neighbourhoods in Mumbai ask a chatbot about sexual and reproductive health. Benchmark: 1,200 questions in 4 languages, scored by doctors, social workers and a woman from the community.
• Jacaranda Health (Kenya): Mothers text a maternal health helpline through pregnancy and after birth, and danger signs go to human agents. Benchmark: 528 real exchanges in English, Swahili and a mix of both.
• SameSame (South Africa and Zimbabwe): Queer young people use a mental health service on WhatsApp, where AI grades self-harm risk so staff can route each case. Benchmark: Messages from real conversations, each given a risk level by 2 labellers.

Tools for frontline health workers

• ARMMAN (India): Nurse midwives ask for guidance on high-risk pregnancies as they work. Benchmark: 1,010 real questions in Hindi and Telugu mixed with English. 100 answers were labelled by 2 reviewers each, from a group of 4 clinicians.
• Intron Health (6 African countries): Health workers ask questions by voice, and AI transcribes, translates and answers spoken clinical questions. Benchmark: 5,200 audio clips in 19 languages and more than 40 accents.
• eHealth Africa (Nigeria): AI turns community health workers' speech into text and works out what they are asking for, such as reporting a symptom or asking for a referral. Benchmark: 30,000 recordings across 4 dialects, tested before the tool goes live.

Tools for doctor consultations

• Pinky Promise (India): At an online women's health clinic, AI drafts responses to patients' follow-up questions, and a gynaecologist edits them before sending. Benchmark: 996 drafts. 2 doctors reviewed 582 of them between them, and both reviewed 49.
• Intelehealth (India): For telemedicine doctors serving rural patients, AI reads the case history and gives a ranked list of likely diagnoses and a treatment plan. Benchmark: 1,000 case histories.

Tools for after a clinic visit

• Penda Health (Kenya): Patients leaving outpatient clinics get AI-written instructions on their medicines, test results and vital signs. Benchmark: 1,000 visits, reviewed by 21 clinical and pharmacy staff in English and Swahili.

Broader lessons

The teams worked on different problems in different countries, but the same lessons kept coming up. Many had less to do with the AI itself than with how to measure it: who judges an answer, what standards they use, and which data they test it on. The Findings on the home page give the full evidence for each one.

1. LLM judges should be checked against human experts

An LLM judge is only useful if it agrees with the human experts it stands in for, and getting there often takes more than 1 round. Jacaranda Health tested 9 judges. All of them agreed poorly with its experts, and the most lenient judge passed 125 answers that clinicians said were unsafe. Myna Mahila tested several judges on 40 answers and chose the one whose scores were closest to its experts. But on the full set of 1,200 answers, that judge scored far more harshly than the clinicians on the 2 criteria that carry the most risk: whether an answer sends a woman with an emergency to a doctor, and whether it makes unfair assumptions about her.

A judge that disagrees with experts usually leans the same way on every answer, too lenient or too strict, rather than making mistakes at random. ARMMAN's judge was so strict that the team had to loosen it.

2. Human experts often disagree

2 qualified clinicians often score the same answers differently. They tend to agree on objective calls, such as whether a dose is right, but differ on judgement calls, such as whether an answer is complete. Pinky Promise's 2 doctors agreed on safety 98% of the time, but on completeness only 65% of the time. Disagreement also runs across groups, not only within them: at Myna Mahila, doctors, social workers and a woman from the community often judged the same answer differently. This sets a limit on how consistent we can expect LLM judges to be.

What helped was getting experts to agree before scaling up. A few of them label a small set of answers, compare where they differ, make the labelling guidelines clearer, and repeat until they agree 70 to 80% of the time. Only then is the full dataset labelled. Asking experts to write down the reason behind each score speeds this up. Pinky Promise's doctors were asked to write a reason for each low score, unless it was obvious. This showed where the guidelines were confusing them, and later helped align the LLM judge.

3. Multiple specific dimensions work better than a single overall rating

A single overall score, such as answer quality, often hides what is wrong. 2 reviewers can both give an answer a low score and disagree about why. The score alone cannot tell you whether the problem is accuracy, safety or tone. So most teams scored a few specific, separate dimensions instead: is it accurate, is it complete, is it clear, is it safe. These dimensions should not overlap. ARMMAN cut 7 overlapping criteria down to 3.

Splitting the score this way also makes it easier to build an LLM judge you can trust. Each dimension can have its own judge, so when the judge for one dimension disagrees with experts, it can be fixed on its own without changing the others. Agreement is worth checking dimension by dimension too. Pinky Promise's judge agreed with its doctors 97% of the time on whether a response acknowledged the patient's worry, but only 52% of the time on whether it used everyday words. A single overall agreement number would have hidden that gap.

4. Start with yes or no questions for each dimension instead of 1 to 5 scales

Many teams began by rating each dimension from 1 to 5, and found that reviewers could not agree on what separated a 3 from a 4. LLM judges struggled with this too. We recommended that most teams move to yes or no questions. A yes or no forces a decision, while a finer scale lets reviewers avoid choosing: compare "is this answer complete?" with "how complete is this answer, from 1 to 5?". After some testing, Myna Mahila found yes or no too blunt, because most answers sat somewhere in between, and settled on 3 points for completeness: complete, partly complete and incomplete.

5. Bigger, pricier LLM judges are not always better

On the 40 answers Myna Mahila used to pick its judge, Gemma 4, an open model, agreed with its doctors and social workers 63.7% of the time. 2 closed models, Claude Haiku 4.5 and Gemini 3.1 Flash Lite, agreed with them only 44.4% and 47.6% of the time. SameSame found GPT-4.1 Nano, at $0.14 per 1,000 messages, slightly more accurate on risk levels than Gemini 3.1 Flash Lite, at $0.26 per 1,000 messages, though the sample was too small for firm conclusions. Jacaranda Health's most expensive judge did only a little better than a model nearly 30 times cheaper.

6. Performance and expert agreement for less-resourced languages can be lower

Teams consistently found AI weaker in low-resource languages, which have little text online for models to learn from. The best speech models in Intron Health's benchmark got 5.5% of words wrong in French, but 47.7% wrong in Akan. Myna Mahila's chatbot averaged 94% on accuracy, but that average hid 98% in Hindi and only 85% in Minglish, which is Marathi typed in English letters.

What gets worse differs by team: accuracy for Myna Mahila, clinical safety for Jacaranda Health, transcription for Intron Health. For Penda Health, Swahili instructions were as accurate as English ones, but only 78% of the Swahili medication instructions read naturally. So it is worth checking every dimension in every language. Experts agree less in the weaker language too. ARMMAN's labellers agreed on completeness 96% of the time in Hindi and 76% of the time in Telugu. Myna Mahila's social workers agreed on completeness 65% of the time in Hindi and 52% of the time in Minglish.

7. LLM judges bring their own assumptions about good answers

The didi example was not a one-off. 8 of the 9 judges that Jacaranda Health tested rated cultural fit lower than its Kenyan clinicians did. Judges from OpenAI's public HealthBench benchmark gave low scores to medically correct, locally appropriate answers from Myna Mahila's chatbot for mentioning breastfeeding in public or suggesting fish during pregnancy. Jacaranda Health wrote local phrasing, such as the Swahili greeting habari mama, into its checklist, so its judge would score against local practice and not its own idea of politeness.

8. Decide what you want to learn before choosing the data

How a team chooses its data decides what it can learn from the benchmark. Myna Mahila wanted to test emergencies, so a third of its questions were high risk, far more than in real use. Jacaranda Health wanted to capture a normal day, so it sampled real messages at random. These are opposite choices, each suited to its purpose, but the scores cannot be read the same way: Myna Mahila's describe a deliberately hard test, not a typical day. Penda Health sampled at least 30 visits per clinic so it could report results for each clinic. What is left out matters too: Intelehealth leaves out cases that need imaging, which makes its data less like what its clinics see.

Looking ahead

Each team's methods, scoring rubrics and results are on this site, with a worked example on each team's page. The complete datasets stay with each organisation.

These benchmarks have limits. They measure how well an AI tool performs, not whether health workers use it or patients end up healthier. In our AI evaluation framework, how well the tool performs is only the first of 4 levels. Several benchmarks rely on small samples, and LLM judges were least reliable at deciding whether an answer fits local norms, which may be the hardest thing to measure.

The programme also suggests that the costly part of evaluation is rarely the LLM judge. The real cost is the time of human experts: labelling answers, finding where they disagree, and settling those disagreements over several rounds.

If you are building or evaluating health AI services in a similar setting, we would like to hear what you think. We hope these benchmarks will be reused, adapted and challenged.

Team and collaborators

Each benchmark was built by the team running the service, at Myna Mahila, Jacaranda Health, SameSame, ARMMAN, Pinky Promise, Intelehealth, Penda Health, Intron Health and eHealth Africa.

Team

• Aman Dalmia, The Agency Fund
• Shubham Sharma, The Agency Fund
• Ananya Iyer, The Agency Fund
• Linus Wong, The Agency Fund
• Temina Madon, The Agency Fund

Supported by

Endless Health and The Agency Fund

Acknowledgements

Our thanks to the doctors, nurses, social workers, community health workers and annotators who labelled the data.