Project documentation

Plain-English definitions of common crawler, Python, infrastructure, cloud, and design-pattern terms used across our engineering documentation. more

Guidelines for productive engineering work, escalation, learning, documentation, and team communication. more

Workstation setup and local development instructions for contributors to the crawler project. more

Use Scrapy signals as an Observer pattern, with working examples and a rule for when pipelines or durable messaging fit better. more