Natural Language Understanding (NLU)

Natural Language Understanding (NLU) is a fundamental building block of goal-oriented dialogue systems like Alexa, Siri, Google Assistant, and Cortana (Fig 1). One of the major challenges of NLU is predicting the user's intent and the slots (arguments) of that intent from their query as seen on the figure below. Several NLU resources exist for high resource languages like English, however, African languages, which constitute over 2000 spoken languages globally, are underrepresented in NLU resources and systems. To address this, we extend the ATIS dataset to 3 African languages: Swahili, Luganda, and Kinyarwanda as the first steps towards improving the representation of African Languages in goal-oriented dialogue systems.

PROBLEM STATEMENT

The limited availability of Natural Language Understanding (NLU) resources for Low Resource Languages (LRLs), particularly African languages which hinders the development of goal-oriented conversational AI systems for these languages.

STUDY OBJECTIVES

• Creating re-usable translation pipelines for African languages
• Creating and open source NLU datasets for African Languages to enable further development in the domain
• Train deep learning models under semi-supervised and unsupervised experiments
• Publish a research paper

DATASET

We used the ATIS dataset, comprising of 4978 training and 893 test samples. We translated the utterance into Swahili and Kinyarwanda using Google Translation API, and hired native translators to correct them. Only the test set was annotated for intents and slots in Kinyarwanda and Swahili using a custom annotation tool(scan annotation tool which the GitHub link is available in the uploaded datasets).