1mad-elmakaoui/hcp-social-signals-2025

A Dagster pipeline for topic-scoped social listening. For a given socio-economic theme it collects public posts,Written during a summer internship (June to July 2025) at the Haut-Commissariat au Plan, Morocco. cleans and language-tags them, scores sentiment, extracts salient terms, and generates context-aware bilingual hashtags with an LLM. # HCP Social Signals A Dagster pipeline for topic-scoped social listening. For a given socio-economic theme it collects public posts, cleans and language-tags them, scores sentiment, extracts salient terms, and generates context-aware bilingual hashtags with an LLM. Written during a summer internship (June to July 2025) at the Haut-Commissariat au Plan, Morocco. ## Quick start ```bash pip install -r requirements-dev.txt python fixtures/_build_posts.py python seed_demo.py dagster dev ``` The UI is at localhost. `HCP_ENV` defaults to `dev`, which runs offline against fixtures and a stub model. No API key and no network access are needed. Set `HCP_ENV=prod` and `ANTHROPIC_API_KEY` to use live feeds and the real model. The dashboard is optional: ```bash pip install -r requirements-dashboard.txt streamlit run dashboard/app.py ``` ## Architecture Dagster sits behind the application, not inside the request path. If the front end needs hashtags for a query in under two seconds, that is a direct API call. Dagster handles the batch side: scheduled collection, sentiment recomputation, backfills, and the cached serving table that the app and the dashboard read. ``` raw_posts ──► clean_posts ──┬──► keyword_signals ──┐ │ ├──► hashtags ──► topic_report └──► post_sentiment ───┘ ``` There are six assets, all partitioned by topic and date. | Asset | What it does | Notes | | --- | --- | --- | | `raw_posts` | Collects public posts and articles for one theme on one day | Runs at 04:00 daily | | `clean_posts` | Dedup, normalization, language tagging | Counts repeats instead of discarding them | | `keyword_signals` | TF-IDF over unigrams and bigrams | Trilingual tokenizer | | `post_sentiment` | Per-post polarity | Lexicon in dev, LLM in prod | | `hashtags` | 8 to 15 bilingual hashtags | The only paid step; skipped under 10 posts | | `topic_report` | One denormalized row per topic-day | The only table the UI queries | ### Why topic x date Th …