#scraping Repositories
109 repositories total
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
Collect structured responses from a ChatGPT scraper by sending a prompt with valid ChatGPT scraping API credentials. Enable live search, inject HTML context, and automate intelligent scraper ChatGPT workflows in just a few parameters.
๐ฐ Brazilian official gazettes accessible to all | ๐ฐ Brazilian government gazettes, accessible to everyone.
Puppeteer for Python that anti-bot systems canโt detect, so there are no CAPTCHAs: the same API, anti-detection stealth headless Firefox, an undetected fingerprint, bypassing bot detection, web scraping, and browser automation.
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
A completely revamped and redesigned fork with OnlyFans and Fansly support, reimagined from scratch based on the original onlyfans-scraper.
Self-hosted scraping engine โ bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
๐ฐ Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
Automatically archive links to videos, images, and social media content from Google Sheets (and more).
โ๏ธ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
Open Source Bulk Auto Gmail Creator Bot with Selenium & Seleniumwire ( Python ). Feel free to contact me with Django/Flask, ML, AI, GPT, Automation, Scraping.
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
๐ Python tool to turn Notion.so pages into lightweight, customizable static websites
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
A high-performance proxy rotation engine with automated IP management and real-time health monitoring
๐ต๏ธโโ๏ธ LinkedIn profile scraper returning structured profile data in JSON.
๐๐ A curated library of research papers and presentations for counter-detection and web privacy enthusiasts.
Kemono pawchive Downloader is a fast, PyQt5 app for archiving content from a wide array of sites, including Kemono, Coomer, Bunkr, Erome, Saint2.su, nhentai, and Discord. It features a Creator Browser, update checker, and supports multi-threaded, multi-part downloads. Sessions can pause, resume, or recover filters.
Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
This Python application is an OSINT (Open Source Intelligence) tool called "Ominis OSINT - Web Hunter." It performs online information gathering by querying Google for search results related to a user-inputted query. The tool extracts relevant information such as titles, URLs, and potential mentions of the query in the results.
Async Python client for collecting public data from X (Twitter) โ search, profiles, threads, replies, and media. Built-in account rotation, adaptive rate limiting, and export to JSON/CSV/SQLite.
:card_index: Extract social media profiles and more with regular expressions
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
Python 3 script to dump/scrape/extract company employees from LinkedIn API
The simplest demo of chrome automation by python and selenium in AWS Lambda
A tool that allows you to scrape media from any Onlyfans account and more