HAKSOAT/wazobia-embed

The code behind the Improving BGE-M3 Multilingual Dense Embeddings for Nigerian Low Resource Languages paper. # 🌍 Improving BGE-M3 Multilingual Dense Embeddings for Nigerian Low Resource Languages This project extends the BGE-M3 on more data for Nigerian native languages: Yoruba, Igbo and Hausa. ## ⚙️ Setting up the Environment This is a Python project based on the `uv` package manager, so you need to run its installation script if you do not already have it installed. The repo can be set up by running: ``` git clone github.com cd wazobia-embed uv sync ``` ## 🚀 Running the Model The model weights are currently on Huggingface at abdulmatinomotoso/bge-finetuned. You can then use it to generate embeddings via: ``` import torch from FlagEmbedding import BGEM3FlagModel device = "cpu" if torch.cuda.is_available(): device = "cuda" elif torch.backends.mps.is_available(): device = "mps" half_precision = False # or True if you really want # Load the abdulmatinomotoso/bge-finetuned model model = BGEM3FlagModel('abdulmatinomotoso/bge-finetuned', use_fp16=half_precision, devices=[device]) # Example text to generate embeddings for documents = [ "Eyi jẹ́ gbolohun àpẹẹrẹ", "Eyi kì í ṣe gbolohun àpẹẹrẹ", "Nke a bụ nkebisiokwu atụ", "Nke a abụghị nkebisiokwu atụ", "Wannan jimla ce ta misali", "Wannan ba jimla ce ta misali ba" ] query = "Where is the example?" # Generate embeddings sparse_embeddings = False # The sparse embeddings are not useful as is, they will require some work to make the align with the entire model. multivec_embeddings = False # The multivector embeddings are still useful despite training only for dense, but this sample uses only dense embeddings dense_embeddings = True doc_embeddings = model.encode(documents, return_sparse=sparse_embeddings, return_dense=dense_embeddings, return_colbert_vecs=multivec_embeddings)["dense_vecs"] query_embeddings = model.encode([query], return_sparse=sparse_embeddings, return_dense=dense_embeddings, return_colbert_vecs=multivec_embeddings)["dense_vecs"] similarity_scores = query_embeddings …