Security

llm-crawler-access-check

by buildwithclaude

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

v1.0.0New

Unsigned, install at your own risk

Unverified

This skill has no cryptographic signature attached. We can't verify the contents match what the publisher intended.

Install this skill

Run this command in your terminal. No account required — it auto-detects your AI tool and installs the skill file.

npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check
Or download directly:
Browse all CLI commands →

Setup by platform

Claude Code

~/.claude/skills/<skill>/SKILL.md

Setup guide →

Install

One-click setup for your editor

Run in your project root

npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check --target claude-code
0

Instructions

This skill doesn’t include stateful context yet, instructions only. Learn about stateful skills.

Security

Loading security scan...

Reviews (0)

View full changelog & diffs →

Related skills

Browse all →
notion-automationAutomate Notion tasks via Rube MCP (Composio): pages, databases, blocks, comments, users. Always search tools first for current schemas.1 installstravel-agent-skill私人旅行管家 — 从出发地到目的地的完整行程规划+攻略导出。 输入出发地、目的地、天数、预算、风格偏好，自动输出闭环行程， 包含交通推荐、酒店推荐、美食路线、每日预算，并可选生成攻略。 当用户提到「做攻略」「旅行规划」「旅游计划」「行程安排」时使用。1 installscf-proxyDeploy a free VLESS proxy/VPN node on Cloudflare Pages using edgetunnel. Automates code download, UUID generation, Pages deployment, free domain registration (DNSExit), DNS configuration, custom domain binding, and client setup for Shadowrocket/v2rayN/Clash. Uses Cloudflare Pages (not Workers) because Pages supports CNAME-based custom domains from any DNS provider, avoiding the need to host DNS on Cloudflare.1 installsrender-automationAutomate Render tasks via Rube MCP (Composio): services, deployments, projects. Always search tools first for current schemas.1 installsops-ecomShopify store command center. Orders, inventory, fulfillment, analytics, and store health. Works with any Shopify store via Admin API.1 installsproject-milestone-tracker项目管理里程碑跟踪器（Project Milestone Tracker）——当用户要求代理在代码库/项目会话中规划、跟踪里程碑、记录承诺或事件、生成进度报告时使用。核心：三层里程碑规则（文件依据/客户提供/待确认）+ 原话+时间戳事件记录。Project milestone tracker: plan and track project milestones, record events with original text + timestamp, generate status reports, using a three-tier milestone rule (contract-sourced / stakeholder-stated / pending-confirmation).1 installs

More from buildwithclaude

View source →
json-canvasCreate and edit JSON Canvas files (.canvas) with nodes, edges, groups, and connections. Use when working with .canvas files, creating visual canvases, mind maps, flowcharts, or when the user mentions Canvas files in Obsidian.3 installsecommerce-material-studio电商素材工坊（中英双语）。用户需要生成电商主图、详情图、场景图、产品图合成、品牌叠加时使用。自动识别品类→匹配风格→场景感知合成→统一文字→自动质检→多平台适配→标准化交付。E-commerce product image studio: category detection, style matching, scene-aware compositing, text overlay, quality check, multi-platform adaptation, batch delivery.3 installscode-healthScans the codebase for dead code, tech debt, outdated dependencies, and code quality issues. Delegates to the Centinela (QA) agent.2 installsshopify-automationAutomate Shopify tasks via Rube MCP (Composio): products, orders, customers, inventory, collections. Always search tools first for current schemas.1 installssupabase-automationAutomate Supabase database queries, table management, project administration, storage, edge functions, and SQL execution via Rube MCP (Composio). Always search tools first for current schemas.1 installstelegram-automationAutomate Telegram tasks via Rube MCP (Composio): send messages, manage chats, share photos/documents, and handle bot commands. Always search tools first for current schemas.1 installs

More Security skills

Browse category →
secureFull-stack security posture assessment with 0-100 risk scoring. Scans dependency vulnerabilities (npm audit, pip-audit, cargo audit, govulncheck), dangerous code patterns (SQL injection, eval, command injection, ReDoS, innerHTML, XSS vectors), authentication gaps (missing auth middleware.16 installspentestStatic-analysis penetration test that hunts for exploitable vulnerabilities with proof-of-concept payloads and fix code.15 installsowaspSystematic audit against the OWASP 2021 Top 10 web application security risks with severity-rated, file-level findings.12 installscheck-vantaFetches dependency vulnerabilities from Vanta, Snyk, Dependabot, or GitHub Security Advisories, creates a tracking issue in Jira/Linear/GitHub Issues, then fixes, commits, pushes, and opens PRs for each affected repo. Trigger on: vulnerabilities, security scan, Vanta, CVE, dependency audit, Snyk, Dependabot.4 installshipaaDeep HIPAA Security Rule technical audit mapping code-level findings to 45 CFR sections. Triggers: building or auditing healthcare apps, EHR integrations, FHIR APIs, telehealth platforms.4 installsgdprGDPR and CCPA/CPRA privacy compliance audit for codebases. Inventories PII fields (email, phone, SSN, IP, device ID, geolocation, biometrics, behavioral data), maps data collection points (forms, APIs, cookies, analytics, error tracking), audits consent mechanisms (cookie banners, opt-in.3 installs

Frequently asked questions about llm-crawler-access-check

What does the llm-crawler-access-check skill do?

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations. It's a reusable SKILL.md instruction set that loads into your AI coding assistant on demand, no prompt engineering, no copy-pasting every session.

How do I install the llm-crawler-access-check skill?

Run `npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check` from your terminal. The CLI writes the SKILL.md to the correct location for your AI tool (e.g. ~/.claude/skills/buildwithclaude-llm-crawler-access-check/ for Claude Code or ~/.cursor/skills/ for Cursor with --target cursor) and adds it to your project's .skills.json lockfile.

Which AI tools does llm-crawler-access-check work with?

llm-crawler-access-check runs in Claude Code. It follows the open Agent Skills standard (SKILL.md), so the same skill works in every supported tool without modification.

Is the llm-crawler-access-check skill free?

Yes. Every skill on skills-hub.ai is free and open-source. There are no premium tiers, paywalls, or usage limits. You only pay for whatever AI assistant you're already using.

How do I use llm-crawler-access-check after installing it?

In Claude Code, type `/buildwithclaude-llm-crawler-access-check` (or whatever slash command the skill registers) and the AI follows the skill's instructions immediately. You can also reference it by name in natural language, your AI loads the skill into context when relevant.

Can I share the llm-crawler-access-check skill with my team?

Yes. Commit your project's .skills.json lockfile and teammates run `npx @skills-hub-ai/cli install` (no args) to install every skill at the exact version you pinned. Organization-scoped installs work via skills-hub.ai organizations.