llm-crawler-access-check

by buildwithclaude

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

v1.0.0New

#buildwithclaude

Unsigned, install at your own risk

Unverified

This skill has no cryptographic signature attached. We can't verify the contents match what the publisher intended.

Install this skill

Run this command in your terminal. No account required — it auto-detects your AI tool and installs the skill file.

npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check

Or download directly:

Browse all CLI commands →

Setup by platform

Claude Code

~/.claude/skills/<skill>/SKILL.md

Setup guide →

Install

One-click setup for your editor

Run in your project root

npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check --target claude-code

Instructions

This skill doesn’t include stateful context yet, instructions only. Learn about stateful skills.

Security

Loading security scan...

Reviews (0)

View full changelog & diffs →

Browse all →

gsd:autonomousRun all remaining phases autonomously — discuss→plan→execute per phase0 installs resemble-detectDeepfake detection and media safety — detect AI-generated audio, images, video, and text, trace synthesis sources, apply watermarks, verify speaker identity, and analyze media intelligence using Resemble AI0 installs oiloil-ui-ux-guideModern, clean UI/UX guidance + review skill. Use when you need actionable UX/UI recommendations, design principles, or a design review checklist for new features or existing systems (web/app). Focus on CRAP (Contrast/Repetition/Alignment/Proximity) plus task-first UX, information architecture, feedback & system status, consistency, affordances, error prevention/recovery, and cognitive load. Enforce a modern minimal style (clean, spacious, typography-led), reduce unnecessary copy, forbid emoji as icons, and recommend intuitive refined icons from a consistent icon set.0 installs gsd:plan-milestone-gapsCreate phases to close all gaps identified by milestone audit0 installs fund-operationsCompute fund KPIs (TVPI, DPI, IRR, MOIC), model carried interest and management fees, and generate LP quarterly update narratives. Triggered by: "/venture-capital-intelligence:fund-operations", "calculate fund KPIs", "what is my fund TVPI", "IRR calculation", "compute MOIC", "LP report", "quarterly update draft", "carried interest calculation", "management fee calculation", "fund performance report", "write my LP update", "how is my fund performing", "what is my DPI", "fund returns analysis", "model my carry", "how much carry do I earn", "portfolio performance summary", "generate investor update". Claude Code only. Requires Python 3.x.0 installs gsd:ultraplan-phase[BETA] Offload plan phase to Claude Code's ultraplan cloud — drafts remotely while terminal stays free, review in browser with inline comments, import back via /gsd:import. Claude Code only.0 installs

More from buildwithclaude

View source →

cohesivityBackend and infra for a project via Cohesivity (cohesivity.ai). Provisions Postgres, hosting and deploys, auth and social login, realtime websockets, an agent-native email inbox, object and vector storage, Redis, cron, and AI model APIs (OpenAI, Anthropic, Deepgram, Exa) through one HTTP API. Use when a valid .cohesivity file exists, a Cohesivity MCP is available, the user names Cohesivity, or the project needs a backend or any of these services and no competing provider was chosen.0 installs ai-search-visibility-auditAudit whether a website can be found, crawled, and cited by AI answer engines such as ChatGPT Search, Perplexity, Google AI Overviews, and Microsoft Copilot. Use when someone asks why their brand is missing from AI answers, whether AI crawlers can read their site, how to get cited by ChatGPT or Perplexity, or asks for a GEO or AEO (generative / answer engine optimization) review. Produces a citation baseline across buyer-intent prompts, a crawler-access check, a citability review of named pages, and a ranked fix list. Not for keyword rank tracking, paid search, or pages behind a login.0 installs work-statusShows WORK progress and TASK status. Use when the user asks about WORK list, WORK progress, TASK status, or pipeline status (e.g., "WORK list", "WORK-01 progress", "show status").0 installs zendesk-automationAutomate Zendesk tasks via Rube MCP (Composio): tickets, users, organizations, replies. Always search tools first for current schemas.0 installs ops-firesProduction incidents dashboard. Reads ECS health, Sentry errors, CI failures. Offers to dispatch fix agents for active fires.0 installs project-milestone-tracker项目管理里程碑跟踪器（Project Milestone Tracker）——当用户要求代理在代码库/项目会话中规划、跟踪里程碑、记录承诺或事件、生成进度报告时使用。核心：三层里程碑规则（文件依据/客户提供/待确认）+ 原话+时间戳事件记录。Project milestone tracker: plan and track project milestones, record events with original text + timestamp, generate status reports, using a three-tier milestone rule (contract-sourced / stakeholder-stated / pending-confirmation).0 installs

More Security skills

Browse category →

pentestStatic-analysis penetration test that hunts for exploitable vulnerabilities with proof-of-concept payloads and fix code. Covers SQL and NoSQL injection (string concatenation, raw queries, operator injection), XSS (reflected, stored, DOM-based, template injection, dangerouslySetInnerHTML), authentication bypass (missing auth middleware, JWT algorithm confusion, predictable tokens, OAuth state CSRF), authorization flaws (IDOR, mass assignment, horizontal/vertical privilege escalation), path traversal and file inclusion (unsanitized file paths, upload validation, LFI), command injection (exec, system, subprocess with user input), CSRF and SSRF (missing SameSite, user-supplied URLs, open redirects), hardcoded secrets (AWS keys, private keys, JWT secrets, connection strings, .env in git), and insecure deserialization (pickle, yaml.load, XXE, ObjectInputStream). Maps full attack surface with route inventory. Use for pre-release security validation, finding exploitable bugs, or generating pen6 installs secureFull-stack security posture assessment with 0-100 risk scoring. Scans dependency vulnerabilities (npm audit, pip-audit, cargo audit, govulncheck), dangerous code patterns (SQL injection, eval, command injection, ReDoS, innerHTML, XSS vectors), authentication gaps (missing auth middleware, CSRF, hardcoded JWT secrets, insecure session flags), insecure crypto (MD5/SHA1 password hashing, Math.random for tokens, hardcoded encryption keys), configuration issues (exposed .env files, debug mode, permissive CORS, missing security headers CSP/HSTS, Docker root containers, default credentials), and data handling problems (PII in logs, missing input validation, file upload exploits, missing rate limiting). Produces a prioritized risk report and routes to specialized skills (pentest, owasp, gdpr, encryption, soc2). Use as a first-pass security triage before deeper audits or before shipping to production.3 installs owaspSystematic audit against the OWASP 2021 Top 10 web application security risks with severity-rated, file-level findings. Checks A01 Broken Access Control (IDOR, path traversal, CORS, privilege escalation), A02 Cryptographic Failures (weak algorithms, exposed secrets, missing TLS), A03 Injection (SQL, NoSQL, command, XSS, LDAP, XPath, template injection), A04 Insecure Design (missing rate limiting, business logic flaws, race conditions), A05 Security Misconfiguration (debug mode, default credentials, missing security headers, CSP), A06 Vulnerable Components (dependency CVEs, outdated frameworks, EOL runtimes), A07 Auth Failures (weak passwords, session fixation, missing MFA), A08 Data Integrity (insecure deserialization, CI/CD integrity, dependency confusion), A09 Logging Failures (missing security events, PII in logs, log injection), A10 SSRF (user-supplied URLs, cloud metadata access, DNS rebinding). Use for web app security audits, pre-release security checks, or compliance evidence g2 installs game-securityGame-specific security review covering cheat prevention, exploit surfaces, and server authority. Audits client-side authority vulnerabilities (damage, health, currency, cooldown, movement speed), memory manipulation risks (value scanning, anti-debug, DLL injection), network security (packet tampering, replay attacks, speed hacking, MITM, DDoS resilience), save file integrity (encryption, checksums, cloud save abuse, config tampering, wallhacks), API and backend security (auth bypass, receipt validation, double-spend, botting, user enumeration), and anti-cheat architecture (server authority, statistical anomaly detection, client integrity, ban systems, shadow banning). Use for multiplayer, competitive, F2P economy, leaderboard, or single-player games with progression systems.2 installs check-vantaFetches dependency vulnerabilities from Vanta, Snyk, Dependabot, or GitHub Security Advisories, creates a tracking issue in Jira/Linear/GitHub Issues, then fixes, commits, pushes, and opens PRs for each affected repo. Trigger on: vulnerabilities, security scan, Vanta, CVE, dependency audit, Snyk, Dependabot.2 installs gdprGDPR and CCPA/CPRA privacy compliance audit for codebases. Inventories PII fields (email, phone, SSN, IP, device ID, geolocation, biometrics, behavioral data), maps data collection points (forms, APIs, cookies, analytics, error tracking), audits consent mechanisms (cookie banners, opt-in, pre-checked boxes, consent withdrawal), verifies data subject rights implementation (right to access, erasure, rectification, portability, opt-out, Do Not Sell), traces third-party data sharing (Google Analytics, Facebook Pixel, Stripe, SendGrid, Sentry), and checks data retention policies and automated purging. Use when auditing privacy compliance, building data export or deletion endpoints, reviewing cookie consent, or assessing DSAR readiness.1 installs

Frequently asked questions about llm-crawler-access-check

What does the llm-crawler-access-check skill do?

How do I install the llm-crawler-access-check skill?

Run `npx @skills-hub-ai/cli install buildwithclaude-llm-crawler-access-check` from your terminal. The CLI writes the SKILL.md to the correct location for your AI tool (e.g. ~/.claude/skills/buildwithclaude-llm-crawler-access-check/ for Claude Code or ~/.cursor/skills/ for Cursor with --target cursor) and adds it to your project's .skills.json lockfile.

Which AI tools does llm-crawler-access-check work with?

llm-crawler-access-check runs in Claude Code. It follows the open Agent Skills standard (SKILL.md), so the same skill works in every supported tool without modification.

Is the llm-crawler-access-check skill free?

Yes. Every skill on skills-hub.ai is free and open-source. There are no premium tiers, paywalls, or usage limits. You only pay for whatever AI assistant you're already using.

How do I use llm-crawler-access-check after installing it?

In Claude Code, type `/buildwithclaude-llm-crawler-access-check` (or whatever slash command the skill registers) and the AI follows the skill's instructions immediately. You can also reference it by name in natural language, your AI loads the skill into context when relevant.

Can I share the llm-crawler-access-check skill with my team?

Yes. Commit your project's .skills.json lockfile and teammates run `npx @skills-hub-ai/cli install` (no args) to install every skill at the exact version you pinned. Organization-scoped installs work via skills-hub.ai organizations.