Converting implementation issues into quantitative findings to challenge complex interventions evaluation: insights from a systematic review

International audience Study objectives: Combination HIV prevention interventions (CHPI) aim to provide the right mix of biomedical, behavioral and structural interventions, and are considered the best approach to curb the HIV pandemic. CHPIs are complex in nature since they add up many components that are expected to interact with one other and to be effective, according to a predefined theory of change. Besides, the implementation of these interventions shall deal with more pragmatic issues such as the stakeholders involvement. Thus, the set of activities undertaken in the course of these interventions raises methodological challenges, and so does the impact evaluation. Currently, the use of diverse quantitative methods and approaches is recommended to assess the impact of CHPIs, allowing to deal with the intervention context. However, it is unclear to what extent these recommendations are sufficient and translated into practice.Method: Hence, we decided to map and critically review the quantitative methods used to assess theimpact of CHPIs on HIV transmission.Results: Our literature search identified a total of 58 original articles concerning 46 CHPI indexed in Web of Science, Scopus and PubMed and published from inception to August 2022. Most of the included studies targeted female sex workers (39.7%), were published after 2000 (94.8%), and were conducted in Asia (36.2%) or in sub-Saharan Africa (44.8%). Less than half of the included studies reported a significant reduction of HIV transmission, regardless of whether it was a primary outcome (49.1%) or not (40.0%). In real-life settings studies reported a significant reduction of HIV transmission more frequently (58.3%) compared to those conducted in controlled settings (31.8%). Most of the included studies used quasi-experimental designs (62.1%). Among the included studies, 87.9% used statistical methods to assess to what extent the interventions reduced HIV transmission as intended. Overall, statistical methods were more accurately reported in experimental studies. Quasi-experimental studies had to account more frequently for confounders and measurement biases either in the design or in the statistical methods. Only nine quasi-experimental studies (15.5%) used mathematical models to assess to what extent the interventions reduced HIV transmission as intended.Discussion: Diverse quantitative designs and methods are currently in use, depicting the intrinsic complexity and the contexts of these interventions. Yet within this diversity, two studies used causal pathway analysis to derive estimates of CHPI impact while three others adjusted their estimates for intervention implementation outcomes, demonstrating the possibility of more comprehensive quantitative approaches. This diversity also confirms that the recommendations on the use of quantitative methods are translated into practice. Nevertheless, our findings highlight a lack of reporting which may be related to major methodological issues and point to the need for updating some key principles of impact evaluation studies. In order to move towards a more comprehensive manner to consider the intervention impact, these principles should address the question of the primary recipients of the evaluation findings, account for the data generating process, and assess some implementation outcomes, especially fidelity, alongside impact estimates.