<b>Head-to-head comparative evaluation of four commercially available Artificial Intelligence systems for detecting referable diabetic retinopathy in a Tanzanian population</b>

Objective

Comparative evaluations of commercially available artificial intelligence (AI) systems for use in diabetic retinopathy (DR) screening – particularly studies that identify systems by name – are limited, constraining procurement and implementation. The study aimed to identify commercially available AI systems potentially suitable for DR screening in a low-resource Tanzanian setting and to compare their accuracy for detecting referable DR.

Research Design and Methods

Through a scoping review and expert consultation, we identified AI systems potentially suitable for implementation. Systems confirmed as suitable, and whose developers agreed to participate, were evaluated. Performance was assessed on a dataset of retinal images collected from a Tanzanian DR screening programme. The primary outcome was sensitivity and specificity for detecting referable DR. Additional implementation data, including regulatory approvals, referral thresholds and additional product features, were also collected.

Results

Four commercially available AI systems (Medios AI / Remidio, MONA, Ophtai and SELENA+) were evaluated. Among 689 people included in the test dataset, 379 (55·0%) had referable DR and 93 (13·5%) had proliferative DR. Sensitivity for detecting referable DR ranged from 83·9%-93·7%, with lower specificities ranging from 70·3%-79·0%. Sensitivity for proliferative DR exceeded 98% for all four AI systems. All the evaluated AI systems were CE-marked medical devices; one system (Medios AI / Remidio) functions offline as standard.

Conclusions

Several commercially available AI systems demonstrated high sensitivity for detecting referable and proliferative DR, supporting their potential implementation. Consensus of minimum performance thresholds and consideration of implementation factors such as regulatory approval and offline functionality are needed.