Smallholder farmers often struggle with crop disease management due to limited access to diagnostic resources. Despite advancements in AI-based plant disease recognition, most existing tools remain costly, complex, and challenging to deploy in low-resource agricultural environments, revealing both a theoretical and practical gap. This study developed a user-centered hybrid system for identifying tomato diseases. The framework integrates deep learning-based image classification with an ontology-supported symptom search. Designed using Shneiderman’s principles and evaluated by domain experts through heuristic and think-aloud protocols, the system achieved 98.7% validation accuracy in image classification while facilitating flexible natural language queries. Theoretically, the study presents a hybrid framework that combines deep learning and semantic reasoning to enhance explainability. Practically, the system improves accessibility for smallholder farmers, offering a scalable solution for early disease detection. The proposed model can be adapted to various crops and regions, addressing gaps in the usability and deployment of agricultural AI.