Optimizing Anomaly Detection in 5G and Beyond Networks using Reinforcement Learning

Kenya has experienced significant growth in internet use and adoption, evidenced by the rapid deployment of 5G networks. Cyberthreats' sophistication, coupled with 5G network complexity, has increased the risk of security problems. Anomaly detection tools developed for earlier generations of networks are ineffective for 5G and beyond networks, and cannot learn and adapt from interactions with their environment. This study investigates current anomaly detection techniques in 5G networks in local Network Facility Providers in Kenya, evaluates their performance, and develops a reinforcement learning-based model using the 5G-NIDD dataset for improved anomaly detection. A structured questionnaire was distributed online to 29 network management professionals across 11 Network Facility Providers in Kenya, using convenience and purposive sampling. The responses were analyzed using descriptive statistics and thematic analysis to achieve study objectives. The results revealed that most local providers used commercial solutions for anomaly detection, which are rule-based or signature-driven. A Deep Q-Network reinforcement learning model was designed and trained to classify benign and eight types of network attacks using the 5G NIDD dataset. Experimental results showed significant improvement in detection performance, achieving an overall accuracy of 75.41% after refinement. This study confirms the potential of reinforcement learning to address critical limitations in existing solutions and provides a promising direction for enhancing the security resilience of 5G and beyond networks