A Vision-Transformer-based Ensemble Model for Multi-label Disease Classification on CT Scans and Chest Radiographs

The increase in the volume of X-rays and computed tomography (CT) images has drastically increased the workload on radiologists. As a result, a computer-aided solution with the capability to classify CT scans is needed to reduce this workload. In this paper, a vision-transformer (ViT) based model for multi-disease, ensemble-based classification is proposed. ViT, Data2Vec, and SegFormer models were fine-tuned to carry out the classification of selected diseases, namely, effusion, pneumonia, and pneumothorax. Normal cases of the selected diseases were included in the dataset. The datasets were obtained from two sources: the chest X-ray dataset from the Nigerian Institute of Health Chest Clinic and the optical coherence tomography (OCT) images dataset containing 6,621 images from University of California, San Diego. The images were preprocessed using random cropping, horizontal flipping, and normalization. The dataset was partitioned into training, validation, and testing sets. Model training was done in 10 epochs. The evaluation metrics showed a better performance from ensemble learning compared to other individual transformer models. The weighted average performance for all metrics was 90.56% precision, 90.58% recall, and 90.48% F1 score. The model is useful for classification of multiple diseases and can be used by radiologists.