Create A Robots.txt File And Increase Your Search Engine Rankings

The robots.txt is a simple text file used to tell search engine bots which pages on your web site should be crawled and indexed. Neil Patel wrote a post on the Link Building Blog that on his personal blog he created a robots.txt file so he could remove any junk pages and duplicate content from the search engines. After doing this, his web site traffic went up 11.3%.

Here are the things Neil did with his blog:

• removed comment feeds from search results so that no duplicate comment text is indexed
• removed trackback URLs from being indexed because it was causing blank pages to be indexed
• denied search bots access to his blog installation folder (MovableType).

If you are using WordPress, a sample robots.txt file would be:

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/

In the above code, Disallow: /wp- makes sure that the search engine bots will not crawl the WordPress files.

Update: Another example of a robots.txt file for WordPress that I found in WordPress.org:

User-agent: *
# disallow files in /cgi-bin
Disallow: /cgi-bin/
Disallow: /comments/
Disallow: /z/j/
Disallow: /z/c/
# disallow all files ending in .php
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
#disallow all files in /wp- directorys
Disallow: /wp-*/
# disallow all files with ? in url
Disallow: /*?
# disallow any files that are stats related
Disallow: /stats*
Disallow: /about/legal-notice/
Disallow: /about/copyright-policy/
Disallow: /about/terms-and-conditions/
Disallow: /about/feed/
Disallow: /about/trackback/
Disallow: /contact/
Disallow: /tag
Disallow: /docs*
Disallow: /manual*
Disallow: /category/uncategorized*

You can also find an example of a robots.txt file for WordPress here.

If you are not using WordPress, you can substitute the disallow lines with files or folders on your web site that you do not want to be crawled.

After you finish creating the robots.txt file, don’t forget to upload it to your site’s root directory.

I also recommend you to display post excerpts instead of full posts on the homepage to ensure that you are not incurring any search engine penalties for duplicated content (on the homepage and single post pages).

Read posts on your favorite topic

Blogging | Make Money Online | WordPress | SEO | Internet

Software | Google | Tutorials | Tweaks | Others