Resource Access

robots.txt

The robots.txt file allows websites to tell clients which resources they can or cannot access. This file is managed by a webmaster and is located at the root of an origin (e.g. https://example.com/robots.txt). When we encounter a request for a disallowed resource, the request will not be performed and a local error will be used instead.

We regularly retrieve this file and parse it according to the Robots Exclusion Protocol. The file may be cached for a minimum of 24 hours, but other HTTP caching properties will be respected.