gabriel-de-jesus/labadain-crawler

A data collection pipeline for low-resource languages # Labadain Crawler ## Overview Labadain crawler is a data collection pipeline for low-resource languages designed to automate and optimize the process of constructing comprehensive textual corpora from the web. The system is built on top of the Apache Nutch framework and incorporates language-specific components such as a tokenizer and a language identification model. ## Requirements ### Technical requirements - [ ] Apache Nutch. - [ ] Apache Solr. ### Language specific requirements - [ ] An initial text corpus containing the target language. - [ ] A tokenizer. - [ ] A language identification model. ## Getting started - [ ] Create a project folder and name it `crawler-home`. - [ ] Navigate into the project folder, create a virtual environment and activate it. - [ ] Clone the pipeline's source codes: ``` $ git clone github.com ``` - [ ] Install the dependencies specified in the `requirements` file. ``` pip install -r requirements.txt ``` ## Apache Nutch and Solr Setup To set up Apache Nutch and Solr, follow these steps: - [ ] **Download Apache Nutch and Solr:** download the Apache Nutch and Solr packages and save them into the **crawler-home** directory. Ensure that you download the appropriate version compatible with your system. - [ ] **Configure Apache Nutch:** refer to the Nutch installation and configuration tutorial for detailed instructions on how to configuring Nutch. Follow the tutorial, but skip the **Crawl your first website** section and proceed directly to the **Setup Solr for search** section. - [ ] **Rename the Nutch and Solr packages:** after downloading and extracting the packages, rename the Nutch package directory from `apache-nutch-1.x` to `nutch` and the Solr package directory from `apache-solr-9.x` to `solr`. - [ ] **Verify the Nutch and Solr configurations:** after completing the configuration steps, verify that both Nutch and Solr are working correctly. You can do this by foll …