crawl-websites-at-scale

Scrape websites at scale using Scrapy, a Python web crawling and scraping framework. Use when: (1) Crawling multiple pages or entire sites, (2) Extracting structured data from HTML/XML, or (3) Building automated data pipelines from web sources.

v1.0.0New

Signing

SignedSLSA L2

Signed by: skills-hub.ai distributor
Method: Distributor-signed by skills-hub.aiCryptographically signed by the skills-hub.ai distributor key at publish time.
Signed: May 3, 2026, 2:23 AM

Install this skill

Run this command in your terminal. No account required — it auto-detects your AI tool and installs the skill file.

npx @skills-hub-ai/cli install open-skills-crawl-websites-at-scale

Or download directly:

Browse all CLI commands →

Setup by platform

Claude Code

~/.claude/skills/<skill>/SKILL.md

Setup guide →

Install

One-click setup for your editor

Run in your project root

npx @skills-hub-ai/cli install open-skills-crawl-websites-at-scale --target claude-code

crawl-websites-at-scale

Signing

Install

Instructions

Security

Reviews (0)