The X-Robots-Tag is a type of HTTP header that is used to control the indexing and serving of a webpage by search engines. The X-Robots-Tag is an extension of the Robots meta tag, which is used to provide similar instructions to search engines, but at the page level.
The X-Robots-Tag can be used to specify a variety of instructions for search engines, such as whether the page should be indexed, whether it should be shown in search results, whether it should be cached, and whether it should be followed by the search engine’s crawlers.
To use the X-Robots-Tag, website owners can add the appropriate instructions to the HTTP headers of their pages. For example, to prevent a page from being indexed by search engines, the following X-Robots-Tag can be used:
X-Robots-Tag: noindex
To prevent a page from being shown in search results, but still allow it to be indexed, the following X-Robots-Tag can be used:
X-Robots-Tag: noindex, nofollow
Overall, the X-Robots-Tag is a powerful tool