#web-scraping Repositories
382 repositories total
Taobao, Tmall Half-price Flash Sale, Grab TVs, Grab Maotai, Crush Scalpers
AIMedia is an integrated software that automatically captures hot topics, creates articles with AI, and publishes them automatically. Supports Toutiao, Xiaohongshu, WeChat Official Accounts, and more.
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating, and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex, and Cursor.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data โ all from Rust. CLI, REST API, and MCP server.
An HTML DOM parser. It allows you to manipulate HTML. Find tags on an HTML page with selectors just like jQuery.
Export books from WeChat Reading to formats like epub, pdf, mobi, etc.
The undetected self-hosted browser automation platform. Powered by Camoufox (Firefox) for 0% detection rates. Built for speed, privacy, and scalability.
Call RPC (Remote Procedure Call) browser methods, avoiding manual code writing and environment setup
Tmall TaoBao iMaotai iMatoTai JD Jingdong rush purchase, JD Moutai rush Windows version, ready to use out of the box without environment configuration. Development imminent (open source protocol under Apache License)
Open source alternative to Gemini Deep Research. Generate reports with AI based on search results.
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
Automatically collected IPv4 hotel TV live streams, with automatic playback speed testing and daily automatic updates. Includes CCTV national and satellite channels, as well as some local channels, for smooth playback. Can also be run on OpenWrt or Synology's Docker.
A fast tool to convert any website into LLM-ready markdown data. Built by https://supermemory.ai
A simple, agile, distributed Java crawler framework supporting SpringBoot; An agile, distributed crawler framework.
Intelligent parser for the "Chitai-Gorod" catalog. High-speed gathering and structuring of data based on any specified parameters: from ISBN to discount dynamics.
net/http.Client like HTTP Client with options to select specific client TLS Fingerprints to use for requests.
Douyin Live Web Version Danmaku Data Scraping (2025 Latest Version)
THIS REPOSITORY IS JUST A MIRROR! The main development repository is https://codeberg.org/Freedium-cfd/web
SURF - Advanced Go HTTP client with Chrome/Firefox browser impersonation, HTTP/3 with QUIC fingerprinting, JA3/JA4 TLS emulation, and anti-bot bypass for web automation and scraping.
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
qstock is developed by the โPython Financial Quantโ public account, aiming to build a personal quantitative investment research analysis package, currently including modules for data acquisition (data), visualization (plot), stock selection (stock), and quantitative backtesting (strategy backtest). qstock will provide users with a concise data interface and normalized financial market data. The visualization module provides users with a simple interface for web-based interactive graphics; the stock selection module provides Tonghuashunโs stock selection data and custom stock selection, including RPS, MM trend, financial indicators, cash flow model, etc.; the backtest module provides everyone with a vectorized (based on pandas) and event-driven basic framework and model. Follow the โPython Financial Quantโ WeChat public account to get more application information.
Follow blogs, wikis, YouTube channels, as well as accounts on Twitter, Instagram, etc. from a single page.
Learn how to build your own Google Jobs scraper that simultaneously scrapes Google Jobs for multiple search queries and geo-locations with Python and Oxylabsโ Google Jobs Scraper API. https://oxylabs.io/blog/how-to-scrape-google-jobs
Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 158 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.
Baidu Netdisk, Quark Netdisk, and Aliyun Drive movie and TV resource sharing. Daily updates of the latest TV series, variety shows, and movie resources.
In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
Implementation of text selection, character selection, selection, and touch CAPTCHA recognition, trained based on PyTorch
Twitter scraper and X scraper for Python: scrape tweets, profiles, followers, following, and trends without the official API. A Twitter scraping library with multi-account pooling, proxy support, and async capabilities, or a hosted version on Apify.