Logo Lanfrica
  • Accueil
  • Atlas
  • Analyses
  • Documentation
  • Sign in

© 2026 Lanfrica. Tous droits réservés. Tous les droits d'auteur des ressources affichées sur le site Web Lanfrica appartiennent aux détenteurs de droits d'auteur d'origine, sauf indication contraire explicite.

Akshara: Script-Aware, Grapheme-Cluster-Correct Array Operations for Indic Text

Domaine:

natural language processing

Type de record:

software
Créateur:
Ahl
Éditeur:
Zenodo
Hôte:avatar
pandas, Polars, and Apache Arrow represent string columns as sequences of Unicode codepoints. For the nine Brahmic abugida scripts (Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam), a single user-perceived unit — an akshara, or orthographic syllable — is frequently encoded as a cluster of several codepoints, causing len() and slicing to silently corrupt conjuncts. This report introduces Akshara, a pure Python/NumPy library providing script-aware, grapheme-cluster-correct array operations — linear-time akshara segmentation, vectorized batch operations, and corpus pipeline integration — validated against 9,108 real FLORES-200 sentences across all nine target scripts with zero failures.

Visit

doi.org

Tasks

text normalization

Tags

tokenization Indic scripts grapheme clustering Brahmic scripts natural language processing low-resource languages Devanagari corpus preprocessing Unicode computational linguistics

Licenses

Creative Commons Attribution 4.0 Internationalhttps://creativecommons.org/licenses/by/4.0/legalcodeCopyright (C) 2026 Prashant Ahlawathttp://rightsstatements.org/vocab/InC/1.0/