Family planning AI chatbot benchmarking

The START team supported the Gates Foundation's strategic planning for improving Family Planning outcomes among young adults aged 18 – 24 by proposing a feasible evaluation approach, beyond traditional RCTs for large language models, AI-powered family planning chatbots. The project involved deriving an evaluation design strategy of the family planning chatbot in Kenya and Senegal, key informant interviews with subject matter experts, and a benchmarking process of the chatbot’s performance across key metrics, including safety, accuracy, data privacy, and security. The team evaluated feasibility, safety, accuracy, and user engagement considerations, ultimately recommending a pragmatic, implementation science–informed evaluation strategy appropriate for adaptive digital tools.