Crawling

The State of Web Crawling: Inside a 66-Million-Host Database

Statistics dashboard of a large-scale web crawler database.
Data Engineering

An hourly snapshot from our crawler dashboard: 66.3 million hosts, 27.4 million indexed, DNS, TLS and robots.txt audits, and the error codes that map the dead web. more

Cloudflare gives bot operators a workspace in BotBase

BotBase dashboard with submission stages, identity checks and bot status.
Cybersecurity ,

BotBase for Operators lets you submit a crawler, watch the review status, edit the record, and pass automated identity checks. more

Why We Recommend Preemptible/Spot VMs for Web Crawling

Server racks providing cloud computing infrastructure.
Data Engineering

Crawl 66 million pages at $2 per million on preemptible VMs, and learn the one limitation that decides whether the trade-off fits. more