#web-scraping Repositories
382 repositories total
AI Map is an AI-powered website mapping tool by Oxylabs AI Studio that uses natural language prompts to intelligently discover and extract relevant URLs from any website.
โค๏ธ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
๐ฑ๏ธ Generate human-like mouse movements with puppeteer or on any 2D plane
Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP ๐
Undetected Python version of the Playwright testing and automation library.
A simple Chrome (and Firefox) Extension that converts Web Novels (and other web pages) into an EPUB.
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
Search for rental housing on a map: Tired of exhausting yourself searching for rentals all over town? Try a different approach and search for rentals directly on the map! [WeChat Official Account & Mini Program: ไบบ็ๅ ้คๆๅ]
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
Collect structured responses from a ChatGPT scraper by sending a prompt with valid ChatGPT scraping API credentials. Enable live search, inject HTML context, and automate intelligent scraper ChatGPT workflows in just a few parameters.
WebAI2API: Web AI to OpenAI API tool based on Camoufox. Supports LMArena/Gemini etc., multi-window concurrency & account isolation. | Web AI to OpenAI API via Camoufox. Supports LMArena/Gemini and more, multi-window concurrency & account isolation.
Use Web Scraper API to extract data from Google Finance, including stock titles, pricing, and price changes in percentages.
Puppeteer for Python that anti-bot systems canโt detect, so there are no CAPTCHAs: the same API, anti-detection stealth headless Firefox, an undetected fingerprint, bypassing bot detection, web scraping, and browser automation.
๐ MCP server for fetch deepwiki.com and get latest knowledge in Cursor and other Code Editors
Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.
Calibre new douban metadata source plugin. Douban no longer provides book APIs to the public, so it can only use web crawling to obtain data. This is a calibre Douban plugin based on web crawling.
๐ฅ AI-powered web monitoring platform. Create automated scouts that search the web and send email alerts when they find what you're looking for.
WebLoop: Scriptable, headless WebKit with a Go API. Like PhantomJS, but for Go.
Flyscrape is a command-line web scraping tool designed for those without advanced programming skills.
Dianping Crawler (crawlable on the whole site, solves dynamic font encryption, not OCR). Continuous update
๐ฅ AI-powered data enrichment tool that transforms emails into rich datasets with company profiles, funding data, tech stacks, and more using Firecrawl and multi-agent AI
Kspider is a crawler platform that allows you to define crawler workflows graphically, enabling the implementation of a crawler workflow without code. Kspider is not limited to web crawling, but can also be used for WEB automation testing. More features await your exploration.
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
An automated price tracker that uses bright data, playwright, react and flask.
Source-level Chromium fingerprinting engine, rated as ordinary Chrome by detection sites. A source-level Chromium fingerprint kernel that passes bot-detection tests. Not JS injectionโC++ patches compiled into the binary.
HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.
A web extension to clip information from web page. Save it to your local machine to avoid information invalidation. Not bored registration, Not charged.