Engineering Glossary
This glossary explains common terms used in crawler, Python, infrastructure, cloud, and design-pattern documentation. It is written for people who are learning the project vocabulary and need short, practical explanations.
Crawler
A crawler is software that automatically visits websites, follows links, and discovers pages or data.
In a web-search or data-ingestion project, the crawler answers: “Which pages should we visit next?”
Coupling
Coupling describes how much one piece of code knows about and depends on another piece of code.
High coupling makes code harder to change because modifying one component often forces changes in other components. Low coupling makes components easier to replace, test, and reuse.
Related: Decoupling, Observer
Decoupling
Decoupling means separating components so they can change independently.
In the Observer pattern, a subject emits an event and observers react to that event without the subject knowing each observer directly.
DNS
DNS, or Domain Name System, converts domain names such as example.com into IP addresses so browsers, servers, and other devices can locate services on the internet.
It is often described as the internet’s address book.
Docker
Docker packages an application together with the libraries, dependencies, and system configuration it needs to run.
The package is called a container. A container makes an application easier to run consistently on a developer laptop, CI server, or production host.
Related: Kubernetes
Git
Git is a version-control system. It records file changes over time so developers can review history, create branches, and collaborate without overwriting each other’s work.
GitLab
GitLab is a platform for hosting Git repositories, issues, merge requests, CI pipelines, and project documentation.
In the onboarding process, GitLab is where developers clone the project, review code, and track work.
GCP
GCP stands for Google Cloud Platform. It is Google’s cloud-computing platform for running applications, storing data, using managed databases, deploying containers, and renting infrastructure over the internet.
Kubernetes
Kubernetes, often shortened to K8s, manages containers across one or many servers.
Docker runs containers. Kubernetes schedules, scales, restarts, and connects many containers in a larger system.
Related: Docker
MongoDB
MongoDB is a NoSQL database that stores data as flexible JSON-like documents instead of rows and columns.
A MongoDB document representing a user might look like this:
{
"name": "Alice",
"age": 25,
"email": "alice@example.com",
"skills": ["Python", "Docker"]
}MongoDB organizes data like this:
MongoDB
-> Database
-> Collection
-> DocumentMyPy
MyPy is a static type checker for Python. It checks Python code for type-related mistakes before the code runs.
For example, it can catch code that passes a string to a function that expects an integer.
Related: Ruff
Observer
Observer is a behavioral design pattern used when one object needs to notify other objects that something has changed.
The object producing the event is often called the subject or publisher. The objects reacting to the event are called observers or subscribers.
Read the practical guide: Observer Pattern with Scrapy
Related: Coupling, Decoupling
OpenTofu
OpenTofu is an open-source Infrastructure as Code tool. It lets you describe infrastructure such as servers, networks, databases, and cloud resources in configuration files, then create and manage that infrastructure automatically.
OpenTofu started as a fork of Terraform after HashiCorp changed Terraform’s licensing in 2023. It is now a Linux Foundation project and keeps the Terraform-style workflow familiar to Terraform users.
Related: Terraform
OVHcloud
OVHcloud, often shortened to OVH, is a French cloud-computing and hosting provider. It offers servers, cloud infrastructure, storage, networking, and related services.
It can be used as an alternative to platforms such as GCP, AWS, or Microsoft Azure.
Python
Python is a general-purpose programming language commonly used for scripting, automation, data processing, web crawling, and backend development.
In this project, Python is used for crawler and data-processing work.
Ruff
Ruff is a fast Python linter and formatter. It helps find Python problems and keep formatting consistent.
Lint code:
ruff check .Format code:
ruff format .Related: MyPy
Scrapy
Scrapy is an open-source Python framework for building web crawlers and web scrapers.
It helps visit pages, follow links, extract information, process items, and store collected data.
Related: Crawler, Spider, Scrapy Middleware
Scrapy Middleware
Scrapy middleware intercepts and modifies requests or responses as they move through the Scrapy engine.
Middleware is useful for cross-cutting behavior such as custom headers, retries, proxies, throttling, or response cleanup.
Spider
A spider is Scrapy’s crawler class. It defines where crawling starts and how responses are parsed.
In Scrapy, a spider usually contains start_urls, parse() methods, and logic for yielding new requests or scraped items.
SSH
SSH, or Secure Shell, is a secure protocol for connecting to remote systems and authenticating access to Git repositories.
Developers usually use SSH keys instead of passwords. A public key is uploaded to a service such as GitLab, while the private key stays on the developer’s machine.
Related: GitLab
Terraform
Terraform is an Infrastructure as Code tool. It lets you define and manage cloud infrastructure using configuration files instead of clicking through a cloud provider’s control panel.
You describe the desired infrastructure in code. Terraform then plans and applies the required changes.
Related: OpenTofu
Venv
venv is Python’s built-in tool for creating a virtual environment.
A virtual environment is a separate Python workspace for one project. It lets you install packages without affecting other Python projects on the same computer.