#web-scraping Repositories
382 repositories total
Just a Simple HTML DOM library fork. (http://simplehtmldom.sourceforge.net/)
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
Turn an entire GitHub Repo into a single organized .txt file to use with LLM's (ChatGPT, Claude, Gemini, etc)
๐ฅ Instantly create AI chatbots for any website with RAG-powered search, streaming responses, and OpenAI-compatible API endpoints
knowledge-grab is a desktop application built on Tauri and Vue 3, designed to facilitate users in downloading various educational resources from the National Primary and Secondary School Smart Education Platform (basic.smartedu.cn).
the framework/ sdk that lets you build browser controlling agents in 3 lines of code. join chat @ https://discord.gg/umgnyQU2K8
Free Proxy List โ ๐ HTTP, HTTPS, SOCKS4 & SOCKS5 | Updated every 5 minutes | Strict SSL, zero MITM, multi-country
A skill for AI agents: search the web with SearXNG, browse with Camofox, bypass protections with CloakBrowser. Anti-hallucination by design. Self-hosted, free, unlimited.
Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
Self-hosted URL- and file-to-Markdown service for humans and AI agents โ web pages, documents, images, audio, YouTube. PWA + REST + MCP + Claude Code skill, Reddit-aware, refreshable share links.
Search engine redirect script, which allows for convenient switching between multiple engines
a free and open-source browser sidebar plugin that offers a cost-free alternative to products like Sider, Monica, and Merlin.
Reverse engineering TikTok web signatures generates valid TikTok signatures for API requests.
Agent Skills for browser fingerprint testing: run liarjs from Claude Code, Codex, Cursor or Copilot to score a browser against itself - 40 consistency checks over canvas, WebGL, WebGPU, audio, fonts and WebRTC, cross-checked against the TLS/HTTP layer. Playwright and Puppeteer harness QA, CI regression gates, headless detection triage.
Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.
All-in-one web scraping API for real-time, large-scale data extraction โ proxies, CAPTCHA management, JS rendering, and parsing in a single request returning HTML or structured JSON.
๐ฏ Bilibili comment data visualization analysis software -- content creators can use it to guide their topic selection and understand their fan base.