roelofrossouw/simply-cpp-afr2eng

Translate from Afrikaans to English # simply-cpp-afr2eng Afrikaans → English translation in C++, with no external services. Translation runs fully offline on the CPU using Meta's NLLB-200 distilled 600M model executed by CTranslate2. A ready-to-use int8-quantized copy of the model is included in `nllb-600M-ct2/`. ## Mechanism The pipeline in `sc::af2eng` (see `include/af2eng.h`, `src/af2eng.cpp`) is: 1. **Tokenize** — each input segment is encoded into subword pieces with SentencePiece (`sentencepiece.bpe.model`), then wrapped with the NLLB source-language token (`afr_Latn`) and an ` ` end marker. 2. **Translate** — the token sequences are passed to a `ctranslate2::Translator` in one batched forward pass, with the target-language token (`eng_Latn`) supplied as a decoding prefix. Inference runs on CPU with int8 compute and configurable beam search (default beam size 2, max 512 decoded tokens). 3. **Detokenize** — the language and sentinel tokens are stripped and the output pieces are decoded back into plain text. Empty or whitespace-only segments bypass the model and pass through unchanged, so line structure is preserved. NLLB is a plain-text sentence model: feed it sentences or short paragraphs, not markup (markdown tables, heading markers, anchors, etc.). The header-only `include/lang_guess.h` provides `sc::guess_language()`, a fast deterministic stopword-based Afrikaans/English discriminator — useful for skipping translation of text that is already English. It is reliable on a paragraph or more of prose and returns a signed confidence score in [-1, 1]. ## Usage ```cpp #include "af2eng.h" const sc::af2eng translator({.model_path = "../nllb-600M-ct2"}); // Single segment std::string en = translator.translate("Hierdie is 'n toets."); // Many segments in one batched call (much faster than a loop); // output order matches input order std::vector out = translator.translate_batch({"Een.", "Twee."}); // Whole text, line by line, preserving blank lines std::string doc = translator.translate_tex …