#crawl Repositories
Discover 11 active open source GitHub repositories tagged with #crawl, accumulating 27,381 total stars (averaging 2,489 stars per project). Leading this topic is kangvcar/InfoSpider with 8,275 stars.
11 repositories total
INFO-SPIDER is a crawler toolbox๐งฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.
novel-plus is a feature-rich, multi-platform (PC, WAP, app, and mini program) novel CMS. It includes features such as novel recommendations, search, rankings, reading, bookshelves, reviews, reading lists, web scraping, a member center, a writersโ section, top-ups and subscriptions, and news publishing.
AnyCrawl ๐: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
JS reverse engineering, cracking JS anti-scraping encryption parameters, already cracked Geetest slider w (2022.2.19), QQ Music sign (2022.2.13), Pinduoduo anti_content, Boss Zhipin zp_token, Zhihu x-zse-96, Kugou kg_mid/dfid, Vipshop mars_cid, China Judgments Online (updated 2020-06-30), Taobao password, Tianan Insurance login, Bilibili login, Fang.com login, WPS login, Weibo login, Youdao Translate, Netease login, WeChat Official Account login, Air China Network login, Jmubiao login, Student Information Management System login, Gengying Finance login, Chongqing Science and Technology Resource Sharing Platform login, Netease Cloud Music download, one-click video link parsing, Cailianshe login.
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
Frequently Asked Questions
What is the most popular #crawl repository on GitHub?
The top ranked open source #crawl project on GitHub is kangvcar/InfoSpider with 8,275 stars. INFO-SPIDER is a crawler toolbox๐งฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.
Which programming languages are most used in #crawl repositories?
Among the top ranked #crawl projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Java.
How is the #crawl repository ranking calculated?
We track 11 active open source repositories tagged with #crawl (500+ stars), totaling 27,381 stars with an average of 2,489 stars per repository. Rankings update daily from the GitHub API.