Layer Wise IndoBERT Representation Evolution for Indonesian Misinformation Detection

Abstract Given the rapid and widespread spread of misinformation on internet, automated hoax detection has become a critical task. For low-resource languages such as Bahasa Indonesia, misinformation often relies on subtle semantic shifts and contextual manipulation, making it particularly difficult to detect using surface-level lexical cues alone. This challenge is further compounded by the limited availability of large, high-quality annotated datasets and linguistic resources, which constrain the development of robust and generalizable detection models. Although transformer-based models—most notably IndoBERT—have demonstrated strong performance in Bahasa Indonesia fake news detection, existing studies largely treat these models as black boxes, offering limited insight into how hoax-related signals are encoded across layers. In this paper, we conduct a systematic layer-wise investigation of fine-tuned IndoBERT models for hoax news detection in Bahasa Indonesia by employing similarity-based analysis to understand how task-relevant features emerge and evolve throughout the network. Through a comprehensive analysis of hidden states, [CLS]-based representations, and mean-pooled representations, we identify a consistent three-phase pattern of representational evolution: early-layer stability, middle-layer transformation, and upper-layer convergence. The results show that the most substantial representational changes occur in the intermediate layers, suggesting that critical task-related features are progressively formed during this stage and subsequently consolidated in higher layers. Furthermore, we observe a clear functional distinction between [CLS] representations, which exhibit sharper task-oriented specialisation, and mean-pooled representations, which retain smoother distributed semantic continuity. Our results reveal a clear and consistent representational pattern in which early layers primarily capture surface-level lexical information, middle layers perform the most substantial semantic restructuring, and upper layers progressively converge toward task-aligned representations. Across all IndoBERT variants, the most significant representational transformations occur in the middle-to-upper layers, suggesting that hoax-related features emerge and are progressively refined at these depths. We further observe that [CLS]-based representations become increasingly specialised for classification-oriented summarisation, whereas mean-pooled embeddings exhibit smoother and more gradual semantic transitions across layers. These findings provide empirical insights into the internal organisation and evolution of IndoBERT representations, revealing how task-relevant features progressively emerge across layers. They further highlight the functional differentiation of transformer layers in low-resource misinformation detection. Beyond performance gains, this analysis points toward the potential development of more efficient, interpretable, and diagnostically grounded transformer-based models for combating misinformation.