hatemamine/STCALIR-Semi-Synthetic-Test-Collection-for-Algerian-Legal-IR
Domain:
natural language processingRecord type:
datasetsoftware
Creator:
hat
Host:
STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval
# STCIR — Semi-Synthetic Test Collection for IR
**STCIR** is an end-to-end, language-adaptive framework for building semi-synthetic IR test collections.
It supports both standard benchmark evaluation (Mr. TyDi, mMARCO, MS MARCO) and custom domain corpora
(e.g., Algerian legal text), with full checkpointing, pre-built run reuse, and LLM-assisted annotation.
---
## Key Features
| Feature | Details |
|---|---|
| **Languages** | Arabic · English (registry-driven, extensible) |
| **Retrieval Stage 1** | BM25 (rank-bm25) + 6 bi-encoders (SentenceTransformer + FAISS) → RRF fusion → top-1000 pool |
| **Retrieval Stage 2** | 5–6 cross-encoders → RRF fusion → top-10 candidates |
| **Pre-built runs** | Download Stage-1 and Stage-2 runs from HuggingFace — skip GPU encoding entirely |
| **Annotation** | Human (Flask UI) or automatic (Gemma 3 4B LLM) |
| **Evaluation** | MRR@k · nDCG@k · MAP · Recall@k · Hit@k · P@k · Kendall's τ · Spearman's ρ |
| **Checkpointing** | Every phase writes a marker file; interrupted runs resume from the last completed phase |
| **Batch mode** | Run all three benchmarks sequentially with one command (`batch_run.ipynb`) |
| **Domain mode** | Chunk a raw corpus → sample passages → create topics → full pipeline |
---
## Repository Structure
```
STCIR/
├── main.ipynb # Interactive single-dataset pipeline (10 cells)
├── batch_run.ipynb # Unattended batch runner (all datasets)
├── configs/
│ ├── mrtydi_arabic.yaml
│ ├── mmarco_arabic.yaml
│ ├── algerian_legal.yaml
│ ├── mrtydi_english.yaml
│ └── msmarco_english.yaml
├── stcir/
│ ├── config.py # STCIRConfig (dataclass, YAML-serialisable)
│ ├── registry.py # Model registry, dataset maps, prebuilt folder map
│ ├── pipeline/
│ │ └── runner.py # PipelineRunner — programmatic phase executor
│ ├── corpus/ # Corpus loader + token-aware sliding-window chunker
│ ├── topics/ # Topic loader (HuggingFace primary / ir_dat …