abdouaziz/Semantic-Aware-Cross-Lingual-Speech-Translation-Representation-for-Wolof
Domaine:
natural language processingType de record:
model
Créateur:
abd
Hôte:
# Semantic-Aware Cross-Lingual Speech Representation for Wolof
PhD research project that aligns **Wolof speech representations** with **French text semantics** using dual supervision from two frozen text encoders.
## Architecture
```
loss = sigmoid(α) · L1(e_sp_pooled, nllb_oi) + (1 − sigmoid(α)) · L2(e_sp_pooled, qwen_oi)
α — learnable scalar (initialised at 0.5)
L1, L2 — cosine embedding loss (target = 1, i.e. maximise similarity)
```
**Trainable parameters:** `SpeechEncoder`, `W_sem` (768→1024), `LayerNorm`, `W_qwen` (D→1024), `α`
**Frozen parameters:** NLLB-200 encoder, Qwen3-Embedding encoder
## Installation
```bash
# 1. Create and activate environment
python -m venv .venv && source .venv/bin/activate # or: conda create -n wolof python=3.11
# 2. Install dependencies
pip install -r requirements.txt
# 3. Login to Weights & Biases (skip if using --no_wandb)
wandb login
python modeling.py
--dataset abdouaziz/spontaneous-s2st-frwo
--src_lang wol_Latn --tgt_lang fra_Latn
--speech_model facebook/wav2vec2-base
--nllb_model facebook/nllb-200-distilled-600M
--qwen_model Qwen/Qwen3-Embedding-0.6B
--batch 4 --epochs 3 --lr 1e-4
--weight_decay 1e-2
--grad_clip 1.0
--log_steps 10
--save_steps 100
--output_dir checkpoints/cpu --num_workers 0 --no_wandb
```
## CLI Reference
| Argument | Default | Description |
|----------|---------|-------------|
| `--batch` | `4` | Batch size |
| `--epochs` | `3` | Number of training epochs |
| `--lr` | `1e-4` | AdamW learning rate |
| `--weight_decay` | `1e-2` | AdamW weight decay |
| `--grad_clip` | `1.0` | Gradient norm clipping |
| `--log_steps` | `10` | Log to W&B every N steps |
| `--save_steps` | `100` | Save step checkpoint every N steps |
| `--output_dir` | `checkpoints` | Directory for saved checkpoints |
| `--speech_model` | `facebook/wav2vec2-base` | HuggingFace speech model |
| `--nllb_model` | `facebook/nllb-200-distilled-600M` | NLLB translation model |
| `--qwen_model` | `Qwen/Qwen3-Embedding-0.6B` | Qwen e …