#crawling Repositories
Discover 39 active open source GitHub repositories tagged with #crawling, accumulating 335,438 total stars (averaging 8,600 stars per project). Leading this topic is D4Vinci/Scrapling with 104,760 stars.
39 repositories total
๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโjoin here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
CrawleeโA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Turn any website into a structured API. Extract, automate, search and monitor the web.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
CrawleeโA web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
List of libraries, tools and APIs for web scraping and data processing.
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
๐ ๐จ๐ณ China Statutory Holidays Data Automatically Fetched Daily from State Council Announcements
Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
Run a high-fidelity browser-based web archiving crawler in a single Docker container
This is the sample code from <6๊ฐ์ ์น ์ ๋ฌด๋ฅผ ํ๋ฃจ ๋ง์ ๋๋ด๋ ์ ๋ฌด ์๋ํ(์๋ฅ์ถํ์ฌ, 2020)>. It is an example designed for people who have never learned Python, and it provides examples in various fields related to task automation, ranging from Excel to design, macros, and crawling.
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
๐ต๏ธโโ๏ธ LinkedIn profile scraper returning structured profile data in JSON.
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.
Python 3 script to dump/scrape/extract company employees from LinkedIn API
Frequently Asked Questions
What is the most popular #crawling repository on GitHub?
The top ranked open source #crawling project on GitHub is D4Vinci/Scrapling with 104,760 stars. ๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโjoin here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Which programming languages are most used in #crawling repositories?
Among the top ranked #crawling projects on GitHub, the most common programming languages are Python, TypeScript, Go, JavaScript.
How is the #crawling repository ranking calculated?
We track 39 active open source repositories tagged with #crawling (500+ stars), totaling 335,438 stars with an average of 8,600 stars per repository. Rankings update daily from the GitHub API.