manifold-machines/synglot

Synthetic data generation and translation tools for low-resource languages. # Synglot: Comprehensive Synthetic Data Generation and Translation Toolkit > [!NOTE] > This project is currently under development. Many features may break and the generation functionality is not fully implemented. Features and APIs may change without notice. Use with caution. You're welcome to contribute! `Synglot` is a Python NLP toolkit for bulk machine translation, synthetic data generation, and multilingual dataset management. The goal is to make it extremely simple to bulk translate datasets in new languages, as well as synthetically generate new ones, in order to empower NLP research in low-resource languages. ## Roadmap - implement further throttling to the OpenAI translator to avoid enqueued token limits, as well as TPD limits. - add sentence splitting to the NLLB backend for higher quality. - integrate the PDF scripts into the core Synglot library - refactor and implement real generation functionality - ✅ add OpenRouter support for more providers ## Key Features - **Unified Translation**: Multi-backend translation supporting MarianMT, NLLB, OpenAI, Google Translate, and OpenRouter APIs - **Generation**: Synthetic data generation with HuggingFace and OpenAI backends - **Powerful Dataset Management**: Comprehensive dataset handling with pandas-like operations - **Batch Processing**: Process many examples at once with optimized performance (especially NLLB) - **CLI Interface**: Full command-line interface for all operations ## Installation ### Basic Installation ```bash git clone github.com cd synglot # Install uv package manager if needed curl -LsSf astral.sh | sh uv venv source .venv/bin/activate uv sync ``` ### Backend Dependencies #### For OpenAI Support ```bash pip install openai export OPENAI_API_KEY="your-openai-api-key" ``` #### For Google Translate Support ```bash pip install google-cloud-translate export GOOGLE_CLOUD_PROJECT_ID="your-project-id" # Set up authentication (see C …