Still Malignant if You Were a Woman? Auditing Group and Counterfactual Fairness in Dermatological AI

International audience Deep learning models achieve remarkable performance in skin lesion classification, but concerns remain regarding whether such systems behave consistently across demographic groups or under counterfactual changes to sensitive attributes. In this paper, we present a fairness audit of a multimodal skin lesion classifier that integrates dermoscopic images with patient metadata. Our contributions are threefold:(1) We design a multimodal CNN that fuses image features with structured patient metadata, enabling controlled testing of demographic sensitivity. (2) We assess group fairness by comparing performance metrics across sex groups.(3) We assess counterfactual fairness by evaluating whether model predictions remain consistent when sensitive attributes are altered while keeping the image input fixed. Experiments on the HAM10000 dataset show that the multimodal EfficientNet achieves 85.78\% accuracy, an 86\% F1-score, and an MCC of 0.78. Although slight performance differences are observed between sex groups, statistical tests indicate that these disparities are insignificant, suggesting no systematic group-level unfairness under our metrics. However, counterfactual analyses reveal isolated prediction changes when the sex attribute is perturbed, indicating that fairness vulnerabilities may still arise despite non-significant group differences. These results highlight the importance of integrating both group and counterfactual fairness evaluations into the development of medical AI systems.