slacker498/document-pipeline

Document processing dashboard built with Antigravity during Build With AI Accra 2026 # Serverless Document Processing Pipeline This repository contains the code and deployment script for a serverless, event-driven document processing pipeline on Google Cloud. ## Architecture 1. **Ingestion:** Users upload files to a Cloud Storage bucket. 2. **Trigger:** Object creation in the bucket triggers a Pub/Sub message. 3. **Processor:** A Python-based Cloud Run service receives the Pub/Sub push message, downloads the file, processes it (simulated OCR & metadata extraction), and writes to BigQuery. 4. **Storage:** Extracted metadata is streamed into a BigQuery table. ## Prerequisites - Google Cloud SDK (gcloud) installed and authenticated. - A Google Cloud Project with billing enabled. ## Deployment 1. Ensure you have set your active project: ```bash gcloud config set project YOUR_PROJECT_ID ``` 2. Run the deployment script: ```bash chmod +x deploy.sh ./deploy.sh ``` *The script handles enabling APIs, creating the Storage bucket, BigQuery dataset/table, Pub/Sub topic/subscription, and deploying the Cloud Run service.* ## Testing the Pipeline 1. Upload a file to the created bucket (the bucket name will be output by `deploy.sh`, typically `doc-pipeline-upload-YOUR_PROJECT_ID`): ```bash echo "This is a test document." > test.txt gsutil cp test.txt gs:// / ``` 2. Check the Cloud Run logs to ensure the file was processed successfully. You can do this in the Google Cloud Console or via CLI: ```bash gcloud run logs read document-processor --region us-central1 ``` 3. Query the BigQuery table to see the extracted metadata: ```bash bq query --nouse_legacy_sql 'SELECT * FROM `YOUR_PROJECT_ID.document_pipeline.document_metadata` LIMIT 10' ```