abulkhair1994/Kweli

# Kweli - Impact Learners Knowledge Graph ETL pipeline and analytics agent to transform Impact Learners CSV data (1.7M rows, 2.5GB) into a Neo4j knowledge graph with natural language querying. **Kweli** means "truth" in Swahili - your truthful guide to Impact Learners data. ## Features - **ETL Pipeline**: Transform CSV data into Neo4j knowledge graph - **Analytics Agent**: Natural language queries powered by LangGraph + Claude/GPT - **Test-Driven Development**: 200+ tests with comprehensive coverage - **HYBRID Graph Architecture**: Property references + nodes to avoid supernodes - **Temporal State Tracking**: SCD Type 2 pattern for learning/professional states - **Resume Capability**: Checkpoint system to resume interrupted ETL runs - **Performance Optimized**: Chunked processing, batch operations, Polars for CSV - **Rich Progress Tracking**: Real-time progress bars with rate metrics - **Data Quality Validation**: Pydantic models with edge case handling - **Modular Code**: All files (s:LearningState) WHERE r.validFrom = date()) RETURN s.state ``` ### Edge Case Handling - **-99 values**: Sentinel for missing data → converted to NULL - **1970-01-01 dates**: Invalid date marker → converted to NULL - **Empty JSON arrays "[]"**: Properly distinguished from missing data - **N/A company names**: Filtered out in analytics queries ## Performance - **CSV Reading**: Polars with 10K row chunks - **Neo4j Writes**: UNWIND batch operations (1K records/batch) - **Memory**: Streaming architecture, low memory footprint - **Expected Rate**: ~1000-2000 rows/second - **Total Time**: ~15-30 minutes for 1.7M rows ## Troubleshooting ### Neo4j Connection Failed ```bash # Check Neo4j is running docker compose -f docker/docker-compose.yml ps # View logs docker compose -f docker/docker-compose.yml logs neo4j # Restart docker compose -f docker/docker-compose.yml restart neo4j ``` ### ETL Interrupted ```bash # Check checkpoint kweli-etl checkpoint-status # Resume kweli-etl run …