Blogger introduces robots.txt file
Google Blogger has introduce the robots.txt file in each blogger blog. The structure of the robots.txt file is
User-agent: *Disallow: /search
Some times back when blogger.com upgaraded to new Google blogger, new lable feature was added in blogger blogs. Each lable has its own page. This robot file make sure that lable pages are not crawled and indexed by any search engine bot. This will certainly help search engines to avoid crawling millions of duplicate lable pages.
To check the robots.txt file of your blogger blog, just type the following URL in the adress bar of your browser.
http://www.yourblogname.blogspot.com/robots.txt
(replace yourblogname with your blog name).
July 16th, 2007 at 9:04 pm
I just noticed this too, This is really bad for me as I would like to have people be able to find my search pages in google, e.g. search for “flora nature photography” and come up with my flora search page. Well, there’s nothing I can really do about it. Maybe I’ll change to wordpress…
July 16th, 2007 at 9:07 pm
P.S. My other blog, thebigbyte.blogspot.com is going to lose a lot of traffic from this as well, as most of my traffic to that site came from google searches for the blogger search page.
July 16th, 2007 at 9:44 pm
David, this robots.txt file will not get your blog de-indexed in Google.. It will just disallow search engines to crawl the label pages (category pages). They are jst the duplicate content page. Your original post page and blog home page will still be there in Google.
July 16th, 2007 at 10:41 pm
Yep, I was aware of that, I’m just disappointed that Google search results will no longer include my category pages, as they gather together a range of related items together that users would otherwise have to search for.
This is good news for my site-specific custom Google search though - it will no longer show duplicate items, which I think is the whole point of this change. I would like to be able to choose whether or not this happens though.
July 17th, 2007 at 10:20 am
BTW, Is there a way to edit robots.txt ?
July 17th, 2007 at 11:50 am
No.. you cannot edit robots.txt file in blogger blog. A default robot file is added in every blog.
July 30th, 2007 at 1:36 pm
It has not been added to my blog. it is disallowing web crawlers to crawl my blog.
July 30th, 2007 at 1:41 pm
Rupesh, if you are talking of your blog fundu bytes, the let me tell you that robots.txt file is added in it.
just check the URL fundubytes.blogspot.com/robots.txt
Also if you check the source code of any page of your blog, you will find the line
<META content=’index,follow’ name=’robots’></META>
in the head section.
So I don’t that there will be any problem in indexing of your blog.
July 31st, 2007 at 7:12 am
it is causing problem… my blog is not listed on google anymore…blog’s pagerank has also dropped… google webmaster tools is showing errors..
July 31st, 2007 at 7:36 am
Well Rupesh
First of all please mention the blog URL which you are talking about. For now am assuming that its the one which is linked to your name here (i.e. fundubytes.blogspot.com).
If that is the case, then you can see that it is well indexed in google.
Just use the query site:fundubytes.blogspot.com in google and see the results.
As for pagerank, its a complex algo and may change depending upon backlinks and other factors.
August 10th, 2007 at 12:35 pm
but i have on problem, how can i change the robot.txt file
August 10th, 2007 at 12:39 pm
Maher you cannot edit robots.txt file in blogger. Anyways whats your problem?
feel free to ask for any help.
November 27th, 2007 at 5:56 pm
Hi, hello, privet
munificently
December 15th, 2007 at 9:35 pm
very interesting, but I don’t agree with you
Idetrorce
January 18th, 2008 at 3:45 am
rtyrdt
January 18th, 2008 at 3:48 am
tyur5e46856
January 18th, 2008 at 3:48 am
ytfufio7tryu8
March 18th, 2008 at 4:24 am
Arpit. Does the robot.txt affect SERPs? Cause it did to all my blogger blogs once i found out the errors.
March 24th, 2008 at 4:38 am
I’d prefer reading in my native language, because my knowledge of your languange is no so well.
April 7th, 2008 at 2:22 am
Can I edit my robots.txt file for the blogger’s blog??
August 7th, 2008 at 5:21 am
No the robots.txt is not editable with Blogger. Anyway the crawling of Labels are not necessary if your posts are cache by the Google. Nobody knows what’s the trick to login at http://ftp.blogger.com I think it’s because of robots.txt
August 25th, 2008 at 5:45 am
That Google does not give access to robots.txt is not well. In this sense that the categories in our blogs remain invisible unless we have them shown as tags under the titles of articles or as an additive in the sidebar! I hope think any decision that all be happy! Because this limitation because of Wordpress are a step forward to the Blogger!