BounharAbdelaziz/MorDern-Bert

Sentence Transformer model finetuned from ModernBERT-base for Moroccan Darija. # Moroccan Darija SentenceTransformer This repository contains a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the al-atlas-moroccan-darija-pretraining-dataset. It maps sentences and paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Features - **Fast and efficient sentence embeddings** for Moroccan Darija. - **Finetuned from ModernBERT-base**, optimized for dialectal Arabic. - **Supports multiple NLP tasks** including search, classification, and clustering. ## Model Details - **Model Type:** Sentence Transformer - **Base Model:** answerdotai/ModernBERT-base - **Maximum Sequence Length:** 8196 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** al-atlas-moroccan-darija-pretraining-dataset - **Weights:** Available on the Hugging Face hub ## Installation Clone the repository and install the required dependencies: ```bash git clone github.com cd moroccan-darija-embeddings pip install -r requirements.txt ``` ## Usage ### Loading Pre-trained Embeddings You can load the trained model using `sentence-transformers`: ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("path/to/moroccan_darija_model") embedding = model.encode("جملة بالدارجة") ``` ## Roadmap - ✅ Sentence Transformer model finetuned for Moroccan Darija - ⏳ Further optimization and finetuning - ⏳ Evaluation on downstream NLP tasks ## Currently Running ### Abdelaziz: - Training ModernBERT from scratch on `atlasia/AL-Atlas-Moroccan-Darija-Pretraining-Dataset` - Training ModernBERT from scratch on `wikipedia-ar` - Finetuning Arabic BERT on `atlasia/AL-Atlas-Moroccan-Darija-Pretraining-Dataset` - Finetuning Multilingual BERT on `atlasia/AL-Atlas-Moroccan-Darija-Pretraining-Dataset` ### Abdelja …