#scraping Repositories
Discover 109 active open source GitHub repositories tagged with #scraping, accumulating 768,143 total stars (averaging 7,047 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.
109 repositories total
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโjoin here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
CrawleeโA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Stealth headless browser for AI agents โ bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
CrawleeโA web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
List of libraries, tools and APIs for web scraping and data processing.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
๐ Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
Swiss-army tool for scraping and extracting data from online assets, made for hackers
Collection of useful data science topics along with articles, videos, and code
Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor, and OpenCode.
AnyCrawl ๐: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
An open source fingerprint browser based on Ungoogled Chromium. Fingerprint browser Privacy browser
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript
OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will grow every time will research, there is a research, science and technology, tutorial. Please use it wisely.
List of anti-detect and humanizing tools and browsers, including captcha solvers and sms-activation.
Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.
Frequently Asked Questions
What is the most popular #scraping repository on GitHub?
The top ranked open source #scraping project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Which programming languages are most used in #scraping repositories?
Among the top ranked #scraping projects on GitHub, the most common programming languages are Python, TypeScript, Go, JavaScript.
How is the #scraping repository ranking calculated?
We track 109 active open source repositories tagged with #scraping (500+ stars), totaling 768,143 stars with an average of 7,047 stars per repository. Rankings update daily from the GitHub API.