mkusiappiah/dscd614-group09-sac2

DSCD 614 Reinforcement Learning, Group 9, Project SAC-2: Continuous Mountain Navigation with Soft Actor-Critic. University of Ghana, 2025/2026. # SAC-2 — Continuous Mountain Navigation Group 9 · DSCD 614 Reinforcement Learning · University of Ghana · 2025/2026 Soft Actor-Critic on `MountainCarContinuous-v0`, evaluated against three baselines under a fixed protocol. | | | |---|---| | Project option | **SAC-2**, Continuous Mountain Navigation | | Algorithm | Soft Actor-Critic (Stable-Baselines3 2.7.0) | | Environment | Gymnasium 1.2.0, `MountainCarContinuous-v0` | | Required baseline | Heuristic controller (energy pumping); DDPG and a random controller added | | Training seeds | 0, 1, 2, 3, 4 | | Runs | 58 (12 configurations) | | Held-out evaluation | 100 episodes, seeds 90000–90099, exploration disabled | ## Headline result **The pre-registered agent failed to beat its baseline, and that is the interesting part.** `sac_main`, the RL Baselines3 Zoo tuned configuration, returns **54.50 ± 56.46** across five seeds against **91.99 ± 0.00** for the energy-pumping heuristic, reaching the flag on 60% of held-out episodes against 100%. It did not fail to learn. All five seeds cross return 90 during training; two then collapse and never recover, finishing at −0.08 and −14.14 while the other three finish at 95.84, 96.19 and 94.69. Splitting "never learned it" from "learned it and lost it" is what makes the result diagnosable. Five other SAC configurations do beat the heuristic, topping out at **95.71 ± 0.65** for `sac_shaped`, and the whole margin is control effort: both succeed on every episode, but SAC spends 42.9 in Σa² against the heuristic's 80.1. Three single-factor findings: - **gSDE is load-bearing.** Swapping it for Gaussian action noise takes the agent from 5/5 seeds solved to **0/5**. The failed policy converges on a ≈ 0, which this reward function makes a genuine local optimum worth exactly 0. - **Potential-based shaping fully rescues that failure** and triples sample efficiency, reaching return 90 at 2,500 steps against 7,500. The potential is policy-invariant by construction, so it costs nothing …