#web-scraping Repositories
Discover 381 active open source GitHub repositories tagged with #web-scraping, accumulating 2,147,590 total stars (averaging 5,636 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.
382 repositories total
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Command-line program to download videos from YouTube.com and other video sites
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโjoin here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝComment Crawler
Legado 3.0 Book Reader with powerful controls & full functions โค๏ธ Reading 3.0, Reading is a tool that allows custom sources to read online content, providing network literature enthusiasts with a convenient, fast, and comfortable trial reading experience.
The best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoringโwith MCP included! All for free, or enjoy our SaaS plan!
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
CrawleeโA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust
๐ Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader โ asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.
Turn any website into a structured API. Extract, automate, search and monitor the web.
Air-gapped, privacy-focused, open-source NotebookLM alternative. An AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
๐ 12306 Ticket Buying Assistant, supports clusters, multiple accounts, multi-task ticket buying, and Web page management
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.
Fast and reliable browser automation for end-to-end testing, scraping, and stealth. Bypasses bot detection with CDP Mode. Integrates with pytest, Playwright, and MCP.
A practical Douyin downloader for both single-item and profile batch downloads, with progress display, retries, SQLite deduplication, and browser fallback support. A Douyin batch download tool supporting videos, image albums, collections, and music (original audio).
Stealth headless browser for AI agents โ bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.
High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
CrawleeโA web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
FreeAskInternet is a completely free, PRIVATE and LOCALLY running search aggregator & answer generate using MULTI LLMs, without GPU needed. The user can ask a question and the system will make a multi engine search and combine the search result to LLM and generate the answer based on search results. It's all FREE to use.
Xiaohongshu (RED) crawler data collection, Xiaohongshu reverse engineering, private messaging, live streaming, and full-domain Xiaohongshu operation solutions
This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHubโperiod. ๐ช
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews, email and more for each place
List of libraries, tools and APIs for web scraping and data processing.
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
Barley automatic ticket grabbing, supports selection of people, cities, date sessions, and prices
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing.
An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl
Declarative data automation language and Go runtime for structured extraction workflows.
EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.
Frequently Asked Questions
What is the most popular #web-scraping repository on GitHub?
The top ranked open source #web-scraping project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Which programming languages are most used in #web-scraping repositories?
Among the top ranked #web-scraping projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Go.
How is the #web-scraping repository ranking calculated?
We track 381 active open source repositories tagged with #web-scraping (500+ stars), totaling 2,147,590 stars with an average of 5,636 stars per repository. Rankings update daily from the GitHub API.