#web-scraping Repositories
382 repositories total
Playwright Model Context Protocol Server - Tool to automate Browsers and APIs in Claude Desktop, Cline, Cursor IDE and More ๐
The go-to web for your AI coding agent โ local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Undetected version of the Playwright testing and automation library.
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
The simplified HTTP request client 'request' with Promise support. Powered by Bluebird.
Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do
An agentic extractor for messaging platforms. Noria receives messages and URLs via chat, scrapes web data to evaluate content against custom scoring matrices (e.g., jobs, scholarships), and sends a structured summary with a calculated match score directly to the assigned messenger number.
search and download music from NetEase Cloud Music, QQ Music, KuGou Music, Baidu Music, Xiami Music, MiGu Music, etc.
(unofficial) Googletrans: Free and Unlimited Google translate API for Python. Translates totally free of charge.
The ultimate home for Open Source and Cleaned apps. OrionStore offers instant access to YouTube Morphe, YT Music Morphe, and essential open-source tools without the clutter. No servers, no tracking, just a beautiful, modern gateway to the apps you love.
Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor, and OpenCode.
Superstar Learning Cloud / Super Star Er Ya / Pan Ya Superstar Fully Automatic Unattended Task Point Completion
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Multi-platform content monitoring ยท collection ยท reposting. One web dashboard to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok / Bilibili / Instagram / Xiaohongshu overseas (rednote)
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
TikTok reverse engineering, TikTok crawler, all TikTok APIs, private messages, live room monitoring
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.
Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Free anti fingerprint browser, fingerprint browser, privacy browser, anti recognition browser, anti identification browser, anti association browser, free web3 airdrop dedicated fingerprint browser https://virtualbrowser.cc/?src=github
A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
๐ฒ An agent for sourcing, curating, and scheduling social media posts with human-in-the-loop.
A python program that turns an LLM, running on Ollama, into an automated researcher, which will with a single query determine focus areas to investigate, do websearches and scrape content from various relevant websites and do research for you all on its own! And more, not limited to but including saving the findings for you!
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
A high-level API to control headless Chrome or Chromium over the DevTools Protocol. It is the Rust equivalent of Puppeteer, a Node library maintained by the Chrome DevTools team.
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
Huya, Douyu, Douyin, BiliBili, TikTok, Twitch๐ฅHot๐ฅSmart Live Video Editing and Publishing AI Robot, Automation๐ค, Fully Intelligentโ(Smart Slice Generation, Titles, Covers, Descriptions), Visualization๐, Platform Hotness Monitoring๐ก, Rich Plugins with Free Expansion๐น, Fast Deploymentโก, Automated Video Account Creation and Publishing๐, DIY Support๐ฎ
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.