yuwenbin860/tunisia-procurement-scraper

# tunisia-procurement-scraper > Proof-of-concept probe for extracting tender award results from > marchespublics.gov.tn > (Tunisian public procurement portal, HAICOP system). This repo currently contains a **verified probe** — a single end-to-end request that confirms the fetch flow and reports the real record count. It is the foundation for the full scraper described in the Upwork brief. ## Run it ```bash git clone github.com cd tunisia-procurement-scraper python probe.py ``` Only dependency is `requests` (`pip install requests`). Output: ``` [warm] GET .../fr/resultats -> 200, 349066 bytes [fetch] GET ?draw=1&start=0&length=2 -> 200 recordsTotal = 55440 recordsFiltered= 55440 rows returned = 2 ``` A `sample_response.json` is written next to the script as auditable evidence. ## Core mechanism — what I learned by reading the page source The client's brief says the results list "is loaded via AJAX POST with a Laravel CSRF token." Reading the page's inline JavaScript reveals the actual flow, which is slightly different and simpler than assumed: | Step | What happens | Why it matters | |---|---|---| | 1. GET `/fr/resultats` | Server returns the HTML page **and** sets a session cookie `XSRF-TOKEN` (Laravel's encrypted CSRF cookie). | One warm-up GET in a `requests.Session()` is enough — the cookie jar carries the CSRF state. No manual ` ` token parsing needed for reads. | | 2. GET `/fr/resultats?draw=&start=&length=` | The same URL, with jQuery **DataTables** server-side params, returns **JSON** (not POST). | DataTables `serverSide: true, ajax: '/fr/resultats'` defaults to GET. The brief's "POST" assumption is a common misread of DataTables. | | 3. Response | Standard DataTables payload: `{recordsTotal, recordsFiltered, data:[...]}`. | Pagination is just `start += length`. `recordsTotal` is the authoritative full count. | The DataTables config found in the page source (`serverSide: true`, `ajax: '.../fr/ …