#web-scraping Repositories
382 repositories total
Self-hosted scraping engine โ bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
A JavaScript library for generating random user agents with data that's updated daily.
A self-hosted API that takes a URL and returns a file with browser screenshots.
Provide scrapers for all major Israeli banks and credit card companies
In this repository you will find sample code files for each day of the course "Python for OSINT. A 21-day course for beginners".
Xiaohongshu xs pure calculation, reverse engineering of pure calculation logic for fields such as Xiaohongshu x-s, x-s-common, xsc.
Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.
660+ muapi-hosted generative media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping) โ one YAML file per entry, browsable by capability.
Kernel-level antidetect browser with the Playwright API you already write. Python + Node SDKs, MCP-ready, unlimited local profiles. Linux x64 + arm64, macOS Intel + Apple Silicon, Windows x64.
The last modification time of this position can be seen at a glance: green is within 2 weeks, dark orange is within 1.5 months, and red is over 1.5 months
A Weibo data-based public opinion analysis project, including Weibo crawler, LDA topic analysis, and sentiment analysis.
A next-generation HTTP stealth proxy which perfectly cloaks requests as the Chrome browser across all layers of the stack.
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
A Rust library to extract useful data from HTML documents, suitable for web scraping.
Ranking Google Scholar search results based on the number of citations
Code for "WebVoyager: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models"
Geektime Course Downloader, supports downloading Geektime Columns/Video Courses/Daily Lessons/Big Factory Practicals/Bootcamp Videos
This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios.
Allows you to bulk download all your Kindle eBook in a more automated fashion. This tool allows you to create backup copies of the books you've already purchased.
ChatGPT Team (Business) promo code scanner โ batch discovery, validation, price collection, supports 34 codes across 17 countries, up to 71% off | ChatGPT Business promo code scanner โ batch discovery, validation, price collection, 34 codes across 17 countries, up to 71% off
A Chrome extension for web sensitive information detection and fingerprinting.
Dorks Eye Google Hacking Dork Scraping and Searching Script. Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Author: Jolanda de Koff