# robots.txt for nodo.lt - the landing page and the documents that describe this service.
#
# THIS SERVICE EXISTS TO BE READ BY MACHINES. The corpus is published Lithuanian
# legislation, licensed CC BY 4.0 by the bodies that publish it, and the product is that
# an agent can quote it with a date attached. So every rule below says yes, on purpose. A
# defensive template copied from another site would contradict the licence and the point
# of the service at the same time.
#
# The register's own site ships a managed ai-train=no default it never chose, and it has
# confused people about what may be reused. We do not inherit a signal we do not mean.
#
# THIS FILE IS FOR THIS ORIGIN ONLY. robots.txt is per-origin (RFC 9309). The MCP endpoint
# is a different host with the opposite policy - it is an endpoint, not a page, and must
# never be indexed - and it serves its own robots.txt saying so. Saying it here would be
# saying it at the wrong origin, where no crawler of that host will ever read it.
#
# Generated by scripts/generate_agent_artefacts.py - edit the generator, not this file.

User-agent: *
Allow: /

# Content-Signal is Cloudflare's convention (https://contentsignals.org/), not a standard:
# the three labels come from an individual Internet-Draft that expired on 2026-04-04, and
# the robots.txt line that carries them is not in that draft at all. It is one line and it
# states a real policy, so it is here - but it is a declaration of intent, not a licence.
# The licence is the one that governs, and it is CC BY 4.0: reuse is permitted and
# attribution is required.
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# The AI crawlers the readiness checkers look for, named one by one. The wildcard group
# above already allows them; naming them removes any doubt about whether the silence was
# deliberate, and it survives a checker that stops accepting a wildcard.
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Bytespider
User-agent: PerplexityBot
User-agent: cohere-ai
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Omgilibot
User-agent: Diffbot
Allow: /

Sitemap: https://nodo.lt/sitemap.xml

# Agentmap is to /.well-known/ai-catalog.json what Sitemap is to /sitemap.xml: a
# pointer for a reader that starts at robots.txt. Non-standard - it is an Agentic
# Resource Discovery convention, and ARD is a v0.9 draft - but it is one line, it is
# ignored by every parser that does not know it, and it is the only route by which an
# agent reading this file learns that the MCP endpoint on the other host exists.
Agentmap: https://nodo.lt/.well-known/ai-catalog.json