Adding a sitemap to the robots.txt is incredibly easy. Not much can go wrong, however you MUST have the sitemap url as an absolute url (http://www.domain.com/sitemap.xml for example)
Lets take an example robots.txt:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
To add a sitemap, you would add the following:
Sitemap: http://thomasharvey.me/sitemap.xml
So the overall robots.txt would now be the following:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: http://thomasharvey.me/sitemap.xml
See the robots.txt documentation for Google: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
I'm a digital marketing consultant based near Peterborough, England. I'm active on the SEO Chat forums and Moz. I'm always on the look out for things to test and to learn that little bit more.
About me and this blog
I'm a digital marketing consultant, posting as much as I can that will help people learn about SEO and other topics.
Request a free quote
I offer various SEO services that help websites increase their online presence, I offer packages for every single type of website.
Subscribe to our newsletter!
Recent Posts
- Protected: 9jsfd920rj 4th December 2018
- Using the correct image type and the impact on page size 26th October 2018
- How Nick Eubanks is Failing at Error Pages 28th March 2018
