PCEUROPA Research Bot
The PCEUROPA bot is a non-commercial research crawler. It does not sell collected data and is not operated as a profitable service. Its purpose is to improve our understanding of how the public web and its infrastructure evolve.
Research goals
The project gathers and analyzes publicly observable information to study:
- adoption and use of large language models and related AI technologies;
- name server and DNS provider distribution;
- DNS configuration and changes over time;
- TLS deployment, certificate configuration, and protocol adoption;
- public web technology and infrastructure trends.
Current aggregate results are available on the crawler statistics page.
The resulting research can help developers, maintainers, students, and researchers create open-source tools, validate scientific hypotheses, reproduce measurements, and make technical decisions using current data.
Non-commercial purpose
This crawler exists for engineering and scientific research. The project is not designed for advertising, lead generation, resale of data, or profiling individuals. Its focus is aggregate technical information about publicly reachable systems and technology adoption.
Responsible crawling
Our goal is to collect only the information needed for technical research while minimizing load on visited services. Crawling should use identifiable requests, reasonable request rates, and standard web controls where applicable.
Requests from the PCEUROPA bot originate from the addresses published in bot-ips.json.
We inform host owners about potential improvements identified during crawling. A report for each host is available at:
https://crawler.pceuropa.net/hosts/<hostname>For example, see the report for r2-concept.be.
Website operators who have questions about a request from the PCEUROPA bot, or who want to report an issue, can contact info@pceuropa.net. Please include the affected hostname, request time, and relevant server log entries.
Project documentation
Browse the crawler's indexed pages.