Data Engineering

The State of Web Crawling: Inside a 66-Million-Host Database

Statistics dashboard of a large-scale web crawler database.
Data Engineering

An hourly snapshot from our crawler dashboard: 66.3 million hosts, 27.4 million indexed, DNS, TLS and robots.txt audits, and the error codes that map the dead web. more

Why We Recommend Preemptible/Spot VMs for Web Crawling

Server racks providing cloud computing infrastructure.
Data Engineering

Crawl 66 million pages at $2 per million on preemptible VMs, and learn the one limitation that decides whether the trade-off fits. more