Token-Level Code-Switching Detection in Bengali English Conversational Data Using Multilingual Transformer Models

Code-switching (CS), switching between two or more languages within a single conversation, is common in multilingual environments, including both Bangladesh and India. Detection of code-switches at the token level is important for downstream NLP applications like machine translation, speech recognition, sentiment analysis, etc. It is extremely difficult to perform token-level CS detection on low-resource language pairs, like Bengali-English, because we don't have annotated data, and Romanized informal Bengali has very high lexical similarity with English. In this work, we use multilingual transformer models to detect token-level CS. Specifically, we approach token-level CS detection as a sequence labeling task and compare fine-tuning strategies across multiple architectures. We present a preprocessing pipeline that targets Bengali-English social media and conversational data, dealing with problems such as transliteration, variation in spelling, variable token boundaries, etc. We experiment across multiple informal conversation datasets to assess model robustness and evaluate performance across architectures. We also highlight XLM-R's advantage in exploiting cross-lingual contextual dependencies. An error analysis of the failure cases, such as named entities, loan words, and intra-word CS, is performed. Overall, transformer models outperform the classical approaches by a significant margin, providing a good baseline for Bengali-English CS detection.