ligon/LSMS_Library: v0.9.0

v0.9.0 A data-correctness release. Several tables were silently returning fewer rows — or different rows — than the surveys contain. If you have cached data, this release rebuilds it once (see below). ⚠️ Your cache rebuilds once on first use Two changes each invalidate every cached parquet, and they ship together deliberately so the rebuild happens once rather than across two releases: LSMS_CACHE_SCHEMA 4 → 5 (#655) Wave.grab_data's @build_transform fingerprint changes (#627) No action needed; the first build after upgrading is a cold one. Silent data loss, fixed to_parquet was deleting real people (#655, closes #645). It stringified object-dtype columns and then tried to recover nulls by string matching — converting the literal values 'None', 'nan' and '' to NULL. None is this library's canonical label for "no education", and _finalize_result's dropna(how='all') then deleted the row entirely. | country | before | after | recovered | |---|---:|---:|---:| | Guatemala | 20,678 | 29,527 | +8,849 (30%) | | Tajikistan | 54,462 | 59,790 | +5,328 | | Ethiopia | 62,939 | 63,181 | +242 | Guatemala returned the same wrong answer cold and warm, so no cache-clearing would have revealed it. The canonical label is renamed None → No education, with the old spelling retained as an accepted variant. assets.Value is additive (#629). Nigeria's wave-2 asset module is a per-unit roster — a household owning five hoes gets five rows — and the collapse kept only the first. ₦294,594,970 recovered, 25.6% of that wave's reported asset value. Registering a table as additive also used to silence its grain audit wholesale; it now re-audits the columns the sum does not reconcile, so the residual Age loss is still reported loudly. The dfs: merge was manufacturing duplicate rows (#627, GH #323 Site 4). An outer merge with no cardinality guard produced rows that every downstream collapse then destroyed. Fixing the merges cleared 4.68M phantom rows — Mali 4,324,668 (#641), Malawi 355,042 and Guinea-Bissau 59 (#653). Cluster identity and grain (GH #323) Sites 1, 2 and 4 of the duplicate-index class are addressed, along with country fixes for Uganda, Tanzania, Malawi, Benin, Togo, Ethiopia, Niger, Nigeria, Albania, Pakistan, India and Guinea-Bissau. Notable: Uganda people_last7days was serving visitor counts to about half of households in 2018-19/2019-20 — the long-form category key was undeclared, so first() picked whichever row the file listed first. 6,185 destroyed rows. Tanzania cluster_features read Region/District/Rural from the household cover page — the address where the household was interviewed — while the NPS tracks movers and carries the original cluster forward. Contested cells fall 2,106 → 385. Malawi 2004-05's cluster key was an EA sequence number within its Traditional Authority: 110 codes for 564 real EAs. Fixing it restored 20 → 26 districts and recovered IHS2's urban stratum, which had been erased entirely. Uganda crop_production gains a condition index level, so dry and fresh harvests of the same crop are no longer summed together. Schema enforcement Declared spellings are now enforced (#605, closes #602); six countries were shipping non-canonical values. GhanaLSS split households were silently merged (#604, closes #548). discover_waves() searched only collection='lsms' (#599, closes #597). Documentation A survey of 111 waves' population/universe statements, quoted verbatim with locators and with gaps recorded as gaps (#654), plus per-country universe/exclusion/oversample records in 38 CONTENTS.org files (#658). 32 countries carry documented exclusions; 34 carry oversamples or sub-populations. The Tanzania NPS back-casting is now recorded as a country-level fact: split-off households are written back into earlier waves, so one round-1 household id can carry up to 11 panel ids (#652). The DVC rule is stated in its general form — never invoke the dvc CLI, for any operation — after three places in the repo were found advising otherwise (#631). Known open GH #323 remains open: Nigeria's 2012Q3/2013Q1 cartesian cells are unowned, and the groupby().first() key-soundness inventory (#637) is partially reviewed. GH #603 (how the library should represent a sample's population) is a design question with evidence now gathered but no proposal.