Circuit Localization in a Tiny Language Model: Geographic Routing and Representational Depth in Qwen2-0.5B

Abstract This paper investigates how the Qwen2-0.5B language model encodes and routes geographic associations. Using mechanistic interpretability techniques, including activation patching, causal ablation, and head-level path patching, we localize the circuits responsible for geographic fact recall.

Key Findings:

• Mechanistic Bottleneck: We identify a critical "Contextual Integration Hub" at Layer 17. Specifically, patching out Layer 17 Head 7 alone causes a 103% damage to the model's geographic prediction, flipping the logit preference to a corrupted target.

• Representational Depth Disparity: We find a significant difference in how the model processes Western vs. MENA (Middle East and North Africa) geography. Western concepts resolve significantly earlier (mean Layer 18), while MENA concepts are consistently gated to the deepest layers of the model (Layer 21), regardless of prompt specificity.

• Routing Convergence: High-specificity cues (landmarks/currencies) trigger earlier circuit activation than ambiguous cultural cues, suggesting a multi-mode routing architecture based on cue strength.

Methodology: Experiments were conducted using the TransformerLens library. The study includes:

1. Multi-trigger convergence tests (Economic vs. Landmark vs. Cultural cues).

2. A large-scale geographic bias audit (36 concepts).

3. Layerwise causal ablation sweeps.

4. Precision head-level path patching for circuit localization.

Code and Reproducibility: The full codebase, including Jupyter notebooks for all experiments (core experiments and diagnostics), is available on GitHub: github.com