# RealCraft.ca Robots.txt # Generated for SEO optimization # As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference # Block dashboard/admin areas from indexing Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ # Block auth-gated user CRUD (returns login redirects, pollutes GSC as "Page with redirect") Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ # Permit check (every first request opens a session row, so crawlers stay out; the city guides are the indexable pages) Disallow: /permits/check/ Disallow: /fr/permits/check/ # Block pagination and filter noise (we want canonical pages) Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= # Session-setting redirect links (e.g. /province//?next=…) 302 to the # target; crawling them wastes budget on redirects with no indexable content. Disallow: /*?next= # Block internal utility endpoints Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ # HTMX tab fragments (partial HTML, no SEO value, splits authority from canonical /p/{slug}/) Disallow: /__debug__/ Disallow: /static/admin/ # pdf.js vendor files (character maps, fonts, wasm, the module). The asset # manifest in each page head names them by plain path, and only the hashed # copies exist. The viewer loads on a click, so no page render needs them. Disallow: /static/js/vendor/pdfjs/ # Blanket allow goes LAST in the group. Google resolves conflicts by longest # matching path so position is irrelevant to it, but order-based parsers take # the first matching rule: a leading "Allow: /" makes those ignore every # Disallow above. Trailing it keeps both interpretations identical. Allow: / # Sitemap location Sitemap: https://realcraft.ca/sitemap.xml # ============================================ # AI retrieval and answer engines - allowed on public pages # # These crawlers build search indexes or fetch a page on a user's behalf to # answer a question, and they cite the source back to the reader. Being in # their corpus sends real traffic, so they get the same access as Googlebot: # every public page, none of the dashboards or PII-bearing paths. # # Each group repeats the full rule set rather than a subset. Under RFC 9309 a # matched user-agent group REPLACES the wildcard group, it does not merge with # it, so any Disallow omitted here is simply not enforced for that crawler. # ============================================ User-agent: OAI-SearchBot Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / User-agent: ChatGPT-User Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / User-agent: Claude-SearchBot Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / User-agent: Claude-User Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / User-agent: PerplexityBot Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / User-agent: Amazonbot Disallow: /dashboard/ Disallow: /admin/ Disallow: /accounts/ Disallow: /api/ Disallow: /flower/ Disallow: /addresses/ Disallow: /socials/ Disallow: /onboarding/ Disallow: /contact-professional/ Disallow: /professional/contact/ Disallow: /rating/ Disallow: /secure/ Disallow: /permits/check/ Disallow: /fr/permits/check/ Disallow: /*?page= Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?next= Disallow: /health/ Disallow: /cities.js Disallow: /p/*/tab/ Disallow: /__debug__/ Disallow: /static/admin/ Disallow: /static/js/vendor/pdfjs/ Allow: / # ============================================ # AI training corpora - blocked site-wide # # These crawlers harvest content into model training sets and return no # traffic or attribution. Matches Content-Signal: ai-train=no above. # # Applebot-Extended is the training opt-out token only; plain Applebot (Siri # and Spotlight search) is deliberately left to the User-agent: * group. # ============================================ User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / # Google-Extended covers Gemini app / Vertex AI grounding AND Gemini model # training in one token; they cannot be separated, so honouring ai-train=no # means blocking it. This does NOT affect Google Search ranking or AI # Overviews eligibility - both are driven by the Googlebot crawl above. User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / # ============================================ # Block SEO competitor analysis tools # (Same strategy as HomeStars, Jiffy, etc.) # ============================================ User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: PetalBot Disallow: / # Block other aggressive crawlers User-agent: MegaIndex.ru Disallow: / User-agent: YandexBot Disallow: / User-agent: Baiduspider Disallow: /