Comments on: Add Sitemaps Autodiscovery in Robots.txt File

By: Adil

Adil — Sat, 26 Jan 2008 20:44:10 +0000

has anyone of you seen any difference after linking ur sitemaps in robots.txt file?

By: Arthur

Arthur — Sat, 21 Apr 2007 15:11:09 +0000

If you have any problems to create XML sitemap files for your sites or to notify search engines about updated sitemap files, try Sitemap Writer Pro. It is a powerfull tool that helps you to create and keep up-to-date sitemaps for your websites. Sitemap Writer Pro has new tools - FTP manager for uploading sitemaps, search engine notification tool (now supports Ask.com and MSN.com), site crawler for adding an URLs into the sitemap and Yahoo Index viewer.

By: apartamentos em portugal

apartamentos em portugal — Thu, 19 Apr 2007 20:26:04 +0000

thanks

By: Ashish Mohta

Ashish Mohta — Sat, 14 Apr 2007 17:20:22 +0000

@Ronald: Byte is right. You have to put

Sitemap: http://www.example.com/sitemap.xml ?

“Sitemap” is the keyword which is understood by crawlers.

By: Mr.Byte

Mr.Byte — Sat, 14 Apr 2007 16:03:19 +0000

@Ronald, The later one… Check out http://www.johntp.com/robots.txt for example.

By: sourcer

sourcer — Sat, 14 Apr 2007 12:08:54 +0000

That is a nice feature, after I have added the robots.txt file on my blog the traffic from google increased, I hope this will also be good for my blog.

Off-topic: John I noticed that you are using Kontera, I will be happy to post some stats or reviews cuz I am trying them right now and I not so satisfied.

By: Mr.Byte

Mr.Byte — Sat, 14 Apr 2007 09:59:29 +0000

@egon & Ashish, I understood now what he meant. Thanks for the clarification.

By: Ronald

Ronald — Sat, 14 Apr 2007 09:53:03 +0000

@Ashish Mohta ; Just put the url http://www.example.com/sitemap.xml or Sitemap: http://www.example.com/sitemap.xml ?

Thanks

By: Ashish Mohta

Ashish Mohta — Sat, 14 Apr 2007 08:07:05 +0000

@MR. Byte Check out JTP links

“Create A Robots.txt File And Increase Your Search Engine Rankings” Its in the related articles.

By: egon

egon — Sat, 14 Apr 2007 07:27:13 +0000

@Mr.Byte: To reduce unnecessary crawler traffic, you use the robots.txt file to tell them which pages not to crawl. Even if those pages are in the sitemap file, whatever crawlers you defined to not crawl those pages, still don’t.

When a bot comes across a page without a sitemap, it has to rely on links from your main page to direct it to all the pages you want indexed. When you have a sitemap, it makes it easier on the bot by saying “Hey Googlebot, here are all the pages on my site, now crawl them unless they’re disallowed in my robots.txt file.” That way Googlebot already knows all of your pages to crawl before it starts following links.