ANAVHEOBA/african-transaction-foundation-model

# african-transaction-foundation-model This project shows how to build a decoder-only foundation model for financial transaction data on NVIDIA GPUs. The workflow starts with raw transaction records, converts them into domain-specific token sequences, pretrains a causal language model, extracts sequence embeddings, and evaluates those embeddings on fraud detection. The example is organized as a notebook-first pipeline backed by RAPIDS for preprocessing and NVIDIA NeMo AutoModel for training. ## What This Project Covers - GPU-accelerated baseline modeling on the TabFormer transaction dataset - A modular tokenizer pipeline for heterogeneous financial transaction fields - Decoder-only pretraining with NeMo AutoModel - Embedding extraction from transaction sequences - Fraud detection with XGBoost using raw features, learned embeddings, and both combined ## Notebook Workflow Run the notebooks in order. | # | Notebook | Purpose | |---|---|---| | 1 | `01_dataset_baseline.ipynb` | Load the dataset, create time-based train/validation/test splits, and build an XGBoost fraud baseline. | | 2 | `02_seq_preproc_tokenization.ipynb` | Convert transaction rows into domain-specific token sequences with the custom tokenizer pipeline. | | 3 | `03_foundation_model_training.ipynb` | Pretrain a decoder-only transaction model with NeMo AutoModel using causal language modeling. | | 4 | `04_inference_embedding_extraction.ipynb` | Load the pretrained model, run inference, extract sequence embeddings, and visualize them with UMAP. | | 5 | `05_xgboost_fraud_detection.ipynb` | Compare fraud detection performance across raw features, embeddings, and combined features. | ## Environment | Component | Recommendation | |---|---| | GPU | 1x NVIDIA A100 80 GB or H100 | | System RAM | 32 GB or more | | OS | Ubuntu 22.04 or newer | | Container Runtime | Docker with NVIDIA Container Toolkit | | Base Container | `nvcr.io` or newer | | Python | 3.10+ inside the container | | …