fmadore/iwac-ai-pipelines

LLM pipelines for the Islam West Africa Collection: OCR, HTR, transcription, NER, summarization and sentiment analysis, written back to Omeka S # IWAC AI Pipelines Python workflows for processing the Islam West Africa Collection (IWAC) using Large Language Models. ## Context The Islam West Africa Collection is an open-access digital database documenting Islam and Muslim communities in Benin, Burkina Faso, Côte d'Ivoire, Niger, Nigeria, and Togo since the 1960s. Created by Frédérick Madore and hosted at the Leibniz-Zentrum Moderner Orient (ZMO) in Berlin, the collection contains over 14,500 items and 28 million words of text. At this scale, traditional manual processing—metadata tagging, OCR correction, entity identification—is no longer viable. These pipelines use LLMs from Google Gemini, OpenAI, and Mistral, plus open-weights models (Qwen, DeepSeek) reached through OpenRouter, to automate labor-intensive tasks that would otherwise leave much of the corpus inaccessible. ## What These Tools Do | Pipeline | Purpose | |----------|---------| | **OCR Extraction** | Extract text from PDF scans using Gemini vision or Mistral Document AI | | **OCR Correction** | Fix errors in machine-generated text, including ALTO XML with coordinate preservation | | **Named Entity Recognition** | Extract people, places, organizations with authority reconciliation | | **Summarization** | Generate bilingual French/English summaries for document discovery | | **Audio Transcription** | Transcribe interviews and oral histories using Gemini or Voxtral (with speaker diarization) | | **Video Processing** | Summarize or transcribe video with visual descriptions | | **Handwritten Text Recognition** | Read manuscripts in French, Arabic, or mixed languages | | **Magazine Article Extraction** | Index individual articles within digitized periodicals | | **Sentiment Analysis** | Evaluate centrality, subjectivity, and polarity of Islam/Muslim representations with a four-model panel | | **Reference Indexing** | Assign controlled subject and spatial keywords to scholarly references using Claude, with authority reconciliation | ## Limitat …