marvinmouroum/africa-ethno-fusion
Domain:
geospatialRecord type:
dataset
Creator:
mar
Host:
Fuse open datasets mapping ethnic groups to places in Africa into one GeoPandas dataset (Murdock, GREG, GeoEPR, D-PLACE, Glottolog, Joshua Project)
# africa-ethno-fusion
Fuse the major open datasets that map **which ethnic groups live where in Africa**
into one queryable star schema (pandas / GeoPandas), then export to
GeoParquet, GeoPackage, or CSV.
🗺️ **Live interactive map:** marvinmouroum.github.io
— Africa partitioned into territory polygons coloured by the dominant ethnicity,
with 1st/2nd/3rd-most-common toggles and a per-ethnicity locator.
## Why a star schema (and not one big table)
The datasets do **not** share a universal key. They link through a *chain* of
partial keys, so a flat join would silently drop or duplicate most rows. Instead
we keep every source record as its own row and express relationships explicitly:
```
Murdock polygon ─(EA "CODE")→ Ethnographic Atlas society ─(glottocode)→ Glottolog ─(ISO 639-3)→ Joshua Project
(territory ~1900) (cultural traits) (language) (people groups, today)
GREG (1964) ┐
├─ carry NO language/glotto codes → only joinable by NAME (fuzzy, lower confidence)
GeoEPR ────┘
```
### Tables
| table | type | grain | purpose |
|---|---|---|---|
| **`groups`** | GeoDataFrame (EPSG:4326) | one ethnic-group/society record per source | the spine: geometry + identity + crosswalk keys (+ `canonical_id`) |
| **`traits`** | DataFrame (tidy/long) | society × variable | the ethnographic "situation" (mostly Ethnographic Atlas) |
| **`links`** | DataFrame | record ↔ record | crosswalk with method + confidence |
| **`canonical`** | GeoDataFrame | **one resolved entity per real-world group** | the unified "one data source" layer (see below) |
| **`review_candidates`** | DataFrame | fuzzy pair | name matches that were *not* auto-merged, for review |
See `africa_ethno_fusion/schema.py` and `entity.py` for the exact columns.
### The canonical layer (entity resolution)
`canonical` collapses the per-source records into one row per real-world group by
running connected-components over `links`:
* **Exact-code merge …