Specify Sitemap In Robots.txt

Overview

When creating a new website, or updating an existing website, you need to tell google and other search engines where your sitemap is located. You can submit a sitemap to google using Google’s webmaster tools. A more convenient way is to Specify a Sitemap In Robots.txt file. The robots.txt file is located in the root of your web folder. Search engines generally access and adhere to information in your robots.txt. However this is not guaranteed. Specifying the location of your sitemap.xml in your robots.txt file is done by adding a single line to your existing robots.txt file. If you don’t have a robots.txt file you can create a new robots.txt file and then add the line as shown below.

Specify Sitemap In Robots.txt

Sitemap: http://example.com/sitemap_location.xml

It will now tell Google and other search engines where your sitemap is located on your website. The sitemap listed here can be either your full sitemap or even a top level sitemap that links to all your other sitemaps (if you have more than one sitemap.xml file).

Full Robots.txt Example

User-agent: *
Allow: /
Disallow: /example-folder/
Disallow: /another-folder/

Sitemap: http://www.example.com/sitemap.xml.gz