fmadore/IWAC-Hugging-Face

Python pipeline mirroring the Islam West Africa Collection from Omeka S into Hugging Face datasets: semantic embeddings, LDA topic models, lexical metrics, Islamic-calendar dates and a multi-model sentiment panel # IWAC Hugging Face Pipeline Python pipeline that mirrors the Islam West Africa Collection (IWAC) from its Omeka S archive into versioned Hugging Face datasets. ## Context The Islam West Africa Collection is an open-access digital database documenting Islam and Muslim communities in Benin, Burkina Faso, Côte d'Ivoire, Niger, Nigeria, and Togo since the 1960s. Created by Frédérick Madore and hosted at the Leibniz-Zentrum Moderner Orient (ZMO) in Berlin, it holds over 14,500 items curated in Omeka S. Omeka S is built for curation and public access, not for analysis. This pipeline turns the archive into something a researcher can actually compute over: it reads the Omeka S REST API, flattens each resource class into a tabular subset, enriches it with columns that do not exist in the source — semantic embeddings, lemmatised text, topic assignments, lexical metrics, Islamic-calendar dates, a multi-model sentiment panel — and publishes the result as a Hugging Face dataset that can be loaded in one line. It is the data layer behind the collection's visualisations and its MCP server, and a companion to iwac-ai-pipelines, which handles the LLM-assisted curation happening upstream inside Omeka S. ## The two-repo split Much of the collection's full text is **private on the Omeka S source** — rights-restricted newspaper scans, for instance — while a large share is public. The dataset is therefore split across two Hub repos: | Repo | Visibility | Role | |------|-----------|------| | `fmadore/islam-west-africa-collection-full` | Private | Complete superset, full text for all rows. The canonical target of **every** upload and post-processing script. | | `fmadore/islam-west-africa-collection` | Public | The citable projection. Written **only** by `post-processing/publish_public.py`. | The projection **masks full text per row rather than stripping it wholesale**. `OCR`, `lemma_text`, and `lemma_nostop` survive wherever `OCR_is_public` is true — a flag derived from t …