Mmasibidi/sesotho-corpus-analysis
Domaine:
natural language processingType de record:
dataset
Créateur:
Mma
Hôte:
Sesotho-corpus-analysis. Reproducable computational workflow for corpus-based analysis of Sesotho.
# Sesotho Corpus Analysis
## Project Overview
This project develops a reproducible computational workflow for the
analysis and preprocessing of a Sesotho monolingual corpus.
The project uses the Autshumato Monolingual Corpus (Sesotho) v2.1 as
the primary corpus.
The initial focus is on understanding the structure and linguistic
characteristics of the corpus before developing a final preprocessing
strategy.
## Corpus
**Corpus:** Autshumato Monolingual Corpus (Sesotho) v2.1
**Raw corpus file:**
`Autshumato.MonolingualCorpus(Sesotho).v2.1.st.txt`
The raw corpus is stored locally and is not committed to this
repository.
## Computational Workflow
The project follows a staged and reproducible workflow:
```text
01_load_corpus.py
↓
02_inspect_corpus.py
↓
03_profile_corpus.py
↓
04_explore_tokens.py
↓
05_preprocess_corpus.py
↓
Future analysis stages
Markdown
### 01. Load Corpus
`01_load_corpus.py`
Checks that the corpus file exists and establishes the input file for
the computational workflow.
### 02. Inspect Corpus
`02_inspect_corpus.py`
Reports basic corpus metadata without modifying the original corpus.
Current observations include:
- File size
- Number of lines
- Sample corpus lines
### 03. Profile Corpus
`03_profile_corpus.py`
Provides basic structural statistics about the corpus, including:
- Total number of lines
- Empty and non-empty lines
- Total whitespace-delimited tokens
- Number of unique tokens
- Shortest line
- Longest line
### 04. Explore Tokens
`04_explore_tokens.py`
Investigates the structure of tokens before preprocessing.
The exploration includes:
- Non-alphabetic tokens
- Punctuation
- Hyphenated tokens
- Apostrophe-containing tokens
- Quotation marks
- Capitalisation
- Character frequencies
- Sentence punctuation
This stage is exploratory and does not modify the raw corpus.
### 05. Preprocess Corpus
`05_preprocess_corpus.py`
Creates a derived processed version of the corpus.
The current preprocessing approach is delibera …