Handling Imbalanced Financial Data Using Cost-Sensitive Deep Learning for AML

Traditional rule-based anti-money laundering (AML) monitoring systems generate large numbers of falsepositive alerts and are rigid when new laundering strategies emerge. This paper presents a hybrid AML framework that combines seven engineered rule-based risk signals with variational-autoencoder (VAE) behavioural embeddings and graph neural network (GNN) transaction-network embeddings. The framework was evaluated on 54,258 real-world SWIFTbased cross-border payment records from an East African commercial bank. After cleaning, currency conversion, log transformation, deep representation learning, feature fusion, correlation filtering, and mutual-information selection, the fused feature space was reduced from 23 to 9 informative attributes. Isolation Forest, Local Outlier Factor, and One-Class Support Vector Machine (OCSVM) were compared under a semi-supervised novelty-detection protocol. OCSVM achieved 99.63% precision within the top 5% of ranked alerts, while LOF obtained the highest ROC-AUC of 0.8459. The framework identified 536 novel anomalies not captured by rule-based heuristics and rejected 1,275 rule-generated alerts. SHAP explanations expose the influence of rule, VAE, and GNN features, while institutional validation showed approximately 1,000 transactions per second on standard Intel Core i7/16 GB hardware.