What it is
Robots.txt sits at the root of a domain and implements the Robots Exclusion Protocol, telling crawlers which paths to leave alone. It stops bots from hammering a site, but it will not keep a page out of search results, which is what the noindex tag is for. Many sites now also use it to turn away AI crawlers, which collect pages to train models or answer questions. llms.txt is a newer, separate file that offers AI tools a plain summary of the site instead.
Why it matters
Because it is a list of things the owner would rather robots did not touch, it occasionally names directories nothing links to: admin panels, staging paths, old exports. It also shows which crawlers the site treats differently from the rest.
How this assessment works
Skopeon Labs queries public infrastructure signals and authoritative network protocols directly from external sensor nodes. All evaluations are completely non-invasive and agentless, assessing observable configurations without sending malicious payloads or disrupting service.
Operational Scope & Limitations
This check evaluates public perimeter responses observable at the time of inquiry. Endpoints protected behind third-party reverse proxies or CDNs reflect the edge configuration of the proxy layer.