Git Stars Git Stars

#crawlers Repositories

Discover 3 active open source GitHub repositories tagged with #crawlers, accumulating 6,067 total stars (averaging 2,022 stars per project). Leading this topic is ai-robots-txt/ai.robots.txt with 4,265 stars.

3
Total Repositories
6,067
Total Stars
2,022
Avg Stars/Repo

3 repositories total

1
ai-robots-txt/ai.robots.txt

A list of AI agents and robots to block.

ai-robots-txt
⭐ 4,265 Python 👤 ai-robots-txt Organization 🕐 Updated 2026-11-16 📅 Created 2024-03-27
2
omrilotan/isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

omrilotan
⭐ 1,168 TypeScript 👤 omrilotan User 🕐 Updated 2026-11-17 📅 Created 2015-07-24
3
scrapfly/Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

scrapfly
⭐ 636 JavaScript 👤 scrapfly Organization 🕐 Updated 2026-11-17 📅 Created 2025-09-03

Frequently Asked Questions

1

What is the most popular #crawlers repository on GitHub?

The top ranked open source #crawlers project on GitHub is ai-robots-txt/ai.robots.txt with 4,265 stars. A list of AI agents and robots to block.

2

Which programming languages are most used in #crawlers repositories?

Among the top ranked #crawlers projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript.

3

How is the #crawlers repository ranking calculated?

We track 3 active open source repositories tagged with #crawlers (500+ stars), totaling 6,067 stars with an average of 2,022 stars per repository. Rankings update daily from the GitHub API.