Marwa-ruwaithi/arabic-ai-text-detection_new

# Arabic AI-Generated Text Detection ## Where I Used Apache Spark | File | How I used it | |------|---------------| | `data_preparation.py` | Spark UDFs to clean Arabic text in parallel | | `mapreduce_jobs.py` | Spark RDDs with `flatMap` and `reduceByKey` | | `feature_engineering.py` | Spark UDFs for features + Spark MLlib for TF-IDF | | `modeling.py` | Spark MLlib to train the classifiers | | `streaming_pipeline.py` | Spark Structured Streaming | | `scalability_benchmark.py` | Tests Spark with different partition counts | --- ## Where I Used Apache Hadoop | Where | How I used it | |-------|---------------| | Input | Can read from HDFS paths like `hdfs:///...` | | Output | Writes Parquet files to HDFS | | Run | `spark-submit --master yarn` on a YARN cluster | | MapReduce output | Saved as Hadoop part-files | --- ## Where I Used MapReduce I wrote 3 MapReduce jobs in `mapreduce_jobs.py`: | Job | Map | Reduce | |-----|-----|--------| | Word Count | `(text) → [(word, 1), ...]` | sum counts per word | | Bigram Count | `(text) → [((w₁, w₂), 1), ...]` | sum counts per bigram | | Hapax Ratio (2-stage) | Stage A = Word Count; Stage B re-keys to "hapax" or "total" | sum per category | --- ## Project structure ``` arabic-ai-text-detection/ ├── data/ │ ├── raw/ Original parquet from Hugging Face │ └── processed/ Cleaned + balanced + features parquets ├── models/ Saved Spark MLlib models ├── notebooks/ Jupyter notebooks for EDA ├── reports/figures/ All plots used in the report ├── src/ Python source code │ ├── utils.py Helpers (Spark session, Arabic NLP) │ ├── data_acquisition.py Phase 1: download from Hugging Face │ ├── data_preparation.py Phase 2: Spark UDFs for cleaning │ ├── mapreduce_jobs.py Phase 2b: MapReduce jobs (PySpark RDDs) │ ├── feature_engineering.py Phase 3a: Spark UDFs + MLlib TF-IDF │ ├── modeling.py …