Scrapers fail quietly. The job exits zero, the table fills with nulls, and three weeks later a dashboard is wrong. Everything here is about making failure loud and recovery cheap.
Validate the shape, not just the status code
A 200 response proves the server answered, not that the selector matched. Parse into a typed schema and fail the run when required fields are missing or types shift.
Track extraction rate as a first-class metric. A drop from 98% to 71% is a layout change, and you want to hear about it the same day.
Be a good client
Read robots.txt and honour it. Rate limit per host, identify yourself in the user agent, and cache aggressively so you fetch each page once. Respectful scrapers get blocked far less often, which makes them cheaper to run.
Where a public API or data export exists, use it. It is more stable than any selector you will write.
- One concurrent request per host by default, backoff on 429
- Conditional requests with ETag and If-Modified-Since
- Store the raw response so re-parsing never means re-fetching
Make runs idempotent
Key every record on a stable identifier and upsert. A retried run should converge to the same state, which means a crashed job can simply be run again instead of triggering a manual cleanup.
Keep a run ledger: start time, pages fetched, records written, failures. It answers 'what changed' without a forensic exercise.
Alert on absence
The dangerous failure is the one that produces no error. Alert when a scheduled run does not report in, when volume deviates from the trailing median, and when the extraction rate drops below its floor.
Want us to run this for you?
We do this work on client systems every week. Tell us where you are and we will say what we would do first.
Book a Call