Data Engineering

PatchTST, Quantiles and Probabilistic Forecasts on Minute Candles

Minute candles split into patches, a transformer and a quantile forecast fan.
Data Engineering ,

Our research team trains patch transformers on closed one-minute candles and reports quantiles, coverage and CRPS instead of one predicted price. more

The State of Web Crawling: Inside a 66-Million-Host Database

Statistics dashboard of a large-scale web crawler database.
Data Engineering

An hourly snapshot from our crawler dashboard: 66.3 million hosts, 27.4 million indexed, DNS, TLS and robots.txt audits, and the error codes that map the dead web. more

Why We Recommend Preemptible/Spot VMs for Web Crawling

Server racks providing cloud computing infrastructure.
Data Engineering

Crawl 66 million pages at $2 per million on preemptible VMs, and learn the one limitation that decides whether the trade-off fits. more