AutoCleanDeepFood: Auto-cleaning and Data Balancing Transfer Learning for Regional Gastronomy Food Computing

Abstract Food computing has emerged as a promising research field, employing artificial intelligence, deep learning, and data science methodologies to enhance various stages of food production pipelines. To this end, the food computing community has compiled a variety of data sets and developed various deep learning architectures to perform automatic classification. However, automated food classification presents a significant challenge, particularly when it comes to local and regional cuisines, which are often underrepresented in available public domain data sets. Nevertheless, obtaining high-quality, well-labeled, and well-balanced real-world labeled images is a challenging task since the manual curation of data requires significant human effort and is time-consuming. In contrast, the web has a potentially unlimited source of food data but tapping into this resource has a good chance of corrupted and wrongly labeled images. In addition, the uneven distribution among food categories may lead to data imbalance problems. All these issues make it challenging to create clean data sets for food from web data. To address this issue, we present AutoCleanDeepFood, a novel end-to-end food computing framework for regional gastronomy that contains the following components: (i) a fully automated pre-processing pipeline for custom data sets creation related to specific regional gastronomy, (ii) a transfer learning-based training paradigm to filter out noisy labels through loss ranking, incorporating a Russian Roulette probabilistic approach to mitigate data imbalance problems, and (iii) a method for deploying the resulting model on smartphones for real-time inferences. We assess the performance of our framework on a novel, web-collected data set and the ETH Food-101 real-world noisy data set. We demonstrate the filtering capabilities of our proposed method through embedding visualization of the feature space using the t-SNE dimension reduction scheme. Our filtering scheme is efficient and effectively improves accuracy in all cases, boosting performance by 0.1% for curated data sets and 0.61% for noisy data set like ETH Food-101. Additionally, we introduce MENA-150, a large-scale image data set that we created from scratch with the proposed framework. MENA-150 features 150 unique traditional dishes from the Middle East and North Africa (MENA) region.