Top #crawling Tools & Software

scrapy

github

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapling

github

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

colly

github

Elegant Scraper and Crawler Framework for Golang

crawlee

github

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

maxun

github

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

crawlee-python

github

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

rod

github

A Chrome DevTools Protocol driver for web automation and scraping.

cariddi

github

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more