Common bean production in Tanzania is threatened by diseases such as bean rust and bean anthracnose, with early detection critical for effective management. This study presents a Vision Transformer (ViT)-based deep learning model enhanced with adversarial training to improve disease detection robustness under real-world farm conditions. A dataset of 100,000 annotated images augmented with geometric, color, and FGSM-based perturbations, simulating field variability. FGSM was selected for its computational efficiency in low-resource settings. The model, fine-tuned using transfer learning and validated through cross-validation, achieved an accuracy of 99.4%. Results highlight the effectiveness of integrating adversarial robustness to enhance model reliability for mobile-based plant disease detection in resource-constrained environments.