pavanpankaj/Web-Crawl-African

Web Crawl African # WebCrawl African WebCrawl African is a collection of African Multilingual parallel corpora comprising of 6.95 lakhs (approx) sentence pairs, covering 15 African languages plus English and 73 language pairs. African languages covered include Afrikaans - afr, Lingala - lin, Swati - ssw, Amharic - amh, Luganda - lug, Tswana - tsn, Chichewa - nya, Hausa - hau, Oroma - orm, Xhosa - xho, Igbo - ibo, Xitsonga - tso, Yoruba - yor, Swahili - swh, Zulu - zul. This corpora is submitted as part of Large Scale Multilingual African Shared task Organized by WMT 2022. Thanks to Organizers for hosting the shared task. It covers variety of domains political, stories, religious and songs. Corpora have sentences covering both formal and informal writing styles. Futher details please go through the papers 1.aclanthology.org 2.aclanthology.org # Dataset Format **African language, English parallel corpus**: Parallel corpus is located at data/African_English_parallel_corpora **African language, African language parallel corpus**: Parallel corpus is located at data/African_African_parallel_corpora Naming convention for src-lang, tgt-lang parallel corpus: Source language file : webcrawl-african-{src-lang}-{tgt-lang}.{src-lang} Target langauge file : webcrawl-african-{src-lang}-{tgt-lang}.{tgt-lang} Example: For Afrikaans, Amharic parallel corpus Source language file: webcrawl-african-afr-amh.afr Target language file: webcrawl-african-afr-amh.amh # Statistics: Table 1: Statistics of African-English parallel corpus {column : source-language} (Value is in multiple of thousand's(K)) African Language | #English | --- | --- | afr | 62.2 | amh | 4.6 | nya | 1.4 | hau | 5.6 | ibo | 1.1 | orm | 7.0 | swh | 64.5 | ssw | 14.4 | tsn | 25.9 | lin | 1.1 | lug | 3.6 | xho | 46.2 | tso | 24.4 | yor | 6.3 | zul | 50.9 | Total | 319.7 | Table 2: Statistics of African-African parallel corpus from {column: source-language}, {row: target-language} …