Numba1ne/careersng-ai

A chat assistant that turns 20 years of CareersNG.com forum content into an instantly queryable knowledge base for careers, jobs, and professional advice in Nigeria. # CareersNG AI Chat RAG-powered chat assistant that turns 20 years of CareersNG.com forum content into an instantly queryable knowledge base for careers, jobs, and professional advice in Nigeria. --- ## Overview CareersNG.com is a Nigerian careers and jobs forum with ~20 years of community content: job listings, career advice, salary discussions, and professional networking. This project: - **Crawls** publicly accessible forum pages (Scrapy + Playwright) - **Processes** raw HTML into clean text, chunks it, and embeds it - **Indexes** chunks in a vector database (ChromaDB) - **Serves** a chat API and Streamlit UI that answer questions using retrieved forum content and an LLM (OpenAI, Anthropic, or Ollama) Answers are grounded in forum content, with source links. Guardrails keep use on-topic and limit message length to control cost. --- ## Target Audience - **Developers** integrating career Q&A into apps or workflows - **Researchers** studying Nigerian job markets or forum discourse - **End users** (via the Streamlit UI) seeking career and job advice from historical forum wisdom - **DevOps** deploying the stack with Docker --- ## Prerequisites - **Python** 3.11+ - **API keys**: At least one of: - OpenAI API key (for `LLM_PROVIDER=openai`) - Anthropic API key (for `LLM_PROVIDER=anthropic`) - **Optional**: OpenAI key for embeddings (`EMBEDDING_PROVIDER=openai`); otherwise local sentence-transformers are used - **Hardware**: ~4GB RAM for local embeddings; more for large crawls - **OS**: Windows, macOS, or Linux --- ## Installation 1. **Clone the repository** ```bash git clone github.com cd careersng-ai ``` 2. **Create a virtual environment (recommended)** ```bash python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate ``` 3. **Install dependencies** ```bash pip install -r requirements.txt python -m playwright install chromium ``` 4. **Environment setup** (see Environment Setup) ```bash cp .env.ex …