Top #crawler Tools & Software

firecrawl

github

🔥 The API to search, scrape, and interact with the web for AI

scrapy

github

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapling

github

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

lux

github

👾 Fast and simple video download library and CLI tool written in Go

colly

github

Elegant Scraper and Crawler Framework for Golang

Scrapegraph-ai

github

Python scraper based on AI

crawlee

github

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

katana

github

A next-generation crawling and spidering framework.

maxun

github

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

webmagic

github

A scalable web crawler framework for Java.

crawlee-python

github

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

autoscraper

github

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

pydoll

github

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

TorBot

github

Dark Web OSINT Tool

cariddi

github

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

NGCBot

github

一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知，安全客，奇安信攻防社区】，👯Kfc文案，⚡漏洞查询，⚡手机号归属地查询，⚡知识库查询，🎉星座查询，⚡天气查询，🌱摸鱼日历，⚡微步威胁情报查询， 🐛视频，⚡图片，👯帮助菜单。📫 支持积分功能，⚡支持自动拉人，，🌱自动群发，👯Ai回复（国内主流AI模型，扣子，FastGpt，Dify全面支持！），⚡视频号解析，😄自定义程度丰富，小白也可轻松上手！