Back to skills

AI Crawl

A spec-compliant llms.txt / llms-full.txt, robots.txt rules for GPTBot/ClaudeBot/PerplexityBot, and a wired sitemap route — from the repo's real routes.

npx skills add drmrduck/skills@aicrawl
View SKILL.md on GitHub ↗drmrduck/skills@aicrawl

AI Crawlability

Write the files that let AI models read and cite the site correctly. Everything lands in the repo — nothing to copy-paste.

Steps

  1. Map the site. Read the routes (Next app//pages/, Astro/SvelteKit pages, or the sitemap if one exists) and the top-level content. Note the real sections a model should know about (docs, product, blog, pricing).
  2. Write llms.txt at the web root (public/llms.txt for most frameworks): an # H1 site name, a one-line > summary, then ## Section lists of the best links with a short gloss each. Add llms-full.txt with the expanded copy when the content is small enough to inline.
  3. robots.txt — allow the AI bots the user wants (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot), block the rest, and declare Sitemap:. Prefer a generated route over a static file so it cannot rot.
  4. Sitemap — wire a real sitemap route (Next app/sitemap.ts, etc.) from the routes, not a hand-maintained XML file.

Verify

  • llms.txt, robots.txt and a sitemap all resolve at the root.
  • robots names the AI user-agents explicitly and declares the sitemap.
  • Try it live in the browser first: https://aicrawl.tools.drummerduck.com

Related skills

appicons

Generate + install a complete favicon / app-icon / PWA stack (ico, every PNG size, apple-touch, theme-aware light/dark, per-environment variants, manifest) from an emoji, named icon, logo, or URL — written straight into the project.

faviconpwaicons
ogimage

A dynamic opengraph-image route + per-page og:/twitter: tags so links unfurl correctly in Slack, iMessage, X, and LinkedIn.

ogsocialseo
metatags

Correct <title>/description/canonical/Open Graph + valid JSON-LD schema on every route — audited and patched in place.

seoschemajson-ldmeta
legalpages

Tailored /privacy and /terms pages and a cookie-consent component, matched to what the app actually does.

legalprivacytermscookies