Project Overview
This is a legal-document scraping and storage pipeline built to turn a vast, unstructured body of court records into something you can actually query. A Scrapy crawler walks High Court websites; the raw documents are then OCR'd and their metadata extracted, cleaned, and loaded into a relational SQL database — 1.4M+ documents in total — for rapid querying and downstream analysis.
At that scale, reliability matters as much as the scraping itself. The daily ETL run is containerized with Docker, scheduled on Azure, and instrumented with logging, monitoring, and automated alerts so the pipeline keeps feeding the database unattended.