Blogger introduces robots.txt file
Google Blogger has introduce the robots.txt file in each blogger blog. The structure of the robots.txt file is
User-agent: *Disallow: /search
Some times back when blogger.com upgaraded to new Google blogger, new lable feature was added in blogger blogs. Each lable has its own page. This robot file make sure that lable pages are not crawled and indexed by any search engine bot. This will certainly help search engines to avoid crawling millions of duplicate lable pages.
To check the robots.txt file of your blogger blog, just type the following URL in the adress bar of your browser.
http://www.yourblogname.blogspot.com/robots.txt
(replace yourblogname with your blog name).
July 16th, 2007 at 9:04 pm
I just noticed this too, This is really bad for me as I would like to have people be able to find my search pages in google, e.g. search for “flora nature photography” and come up with my flora search page. Well, there’s nothing I can really do about it. Maybe I’ll change to wordpress…
July 16th, 2007 at 9:07 pm
P.S. My other blog, thebigbyte.blogspot.com is going to lose a lot of traffic from this as well, as most of my traffic to that site came from google searches for the blogger search page.
July 16th, 2007 at 9:44 pm
David, this robots.txt file will not get your blog de-indexed in Google.. It will just disallow search engines to crawl the label pages (category pages). They are jst the duplicate content page. Your original post page and blog home page will still be there in Google.
July 16th, 2007 at 10:41 pm
Yep, I was aware of that, I’m just disappointed that Google search results will no longer include my category pages, as they gather together a range of related items together that users would otherwise have to search for.
This is good news for my site-specific custom Google search though - it will no longer show duplicate items, which I think is the whole point of this change. I would like to be able to choose whether or not this happens though.
July 17th, 2007 at 10:20 am
BTW, Is there a way to edit robots.txt ?
July 17th, 2007 at 11:50 am
No.. you cannot edit robots.txt file in blogger blog. A default robot file is added in every blog.
July 30th, 2007 at 1:36 pm
It has not been added to my blog. it is disallowing web crawlers to crawl my blog.
July 30th, 2007 at 1:41 pm
Rupesh, if you are talking of your blog fundu bytes, the let me tell you that robots.txt file is added in it.
just check the URL fundubytes.blogspot.com/robots.txt
Also if you check the source code of any page of your blog, you will find the line
<META content=’index,follow’ name=’robots’></META>
in the head section.
So I don’t that there will be any problem in indexing of your blog.
July 31st, 2007 at 7:12 am
it is causing problem… my blog is not listed on google anymore…blog’s pagerank has also dropped… google webmaster tools is showing errors..
July 31st, 2007 at 7:36 am
Well Rupesh
First of all please mention the blog URL which you are talking about. For now am assuming that its the one which is linked to your name here (i.e. fundubytes.blogspot.com).
If that is the case, then you can see that it is well indexed in google.
Just use the query site:fundubytes.blogspot.com in google and see the results.
As for pagerank, its a complex algo and may change depending upon backlinks and other factors.
August 10th, 2007 at 12:35 pm
but i have on problem, how can i change the robot.txt file
August 10th, 2007 at 12:39 pm
Maher you cannot edit robots.txt file in blogger. Anyways whats your problem?
feel free to ask for any help.
November 27th, 2007 at 5:56 pm
Hi, hello, privet
munificently
December 15th, 2007 at 9:35 pm
very interesting, but I don’t agree with you
Idetrorce
January 18th, 2008 at 3:45 am
rtyrdt
January 18th, 2008 at 3:48 am
tyur5e46856
January 18th, 2008 at 3:48 am
ytfufio7tryu8
March 18th, 2008 at 4:24 am
Arpit. Does the robot.txt affect SERPs? Cause it did to all my blogger blogs once i found out the errors.
March 24th, 2008 at 4:38 am
I’d prefer reading in my native language, because my knowledge of your languange is no so well.
April 7th, 2008 at 2:22 am
Can I edit my robots.txt file for the blogger’s blog??
August 7th, 2008 at 5:21 am
No the robots.txt is not editable with Blogger. Anyway the crawling of Labels are not necessary if your posts are cache by the Google. Nobody knows what’s the trick to login at http://ftp.blogger.com I think it’s because of robots.txt
August 25th, 2008 at 5:45 am
That Google does not give access to robots.txt is not well. In this sense that the categories in our blogs remain invisible unless we have them shown as tags under the titles of articles or as an additive in the sidebar! I hope think any decision that all be happy! Because this limitation because of Wordpress are a step forward to the Blogger!
November 6th, 2008 at 3:11 am
I moved my blog to the WordPress and I’ve an own FTP hosting. Now everything is much better
November 21st, 2008 at 9:47 am
This thing sux…makes less publicity of blog
http://blog.kpsofts.com
see my robots.txt file
January 2nd, 2009 at 8:09 pm
i am realy tired to busy with blogger blogs. /search is not permitted bu how can increase my site navigation for the search engines. There is no mean to labelled the post for the SEO. What a pity!
April 26th, 2009 at 2:47 am
thanks for awesome article like this, i trying to chcek it for my blogspot.
June 24th, 2009 at 12:46 pm
thanks for robots.txt information. this is veru usefull
August 30th, 2009 at 4:38 pm
I never ever post but this time I will,Thanks alot for the great blog.
September 10th, 2009 at 7:42 am
Hi! I was surfing and found your blog post… nice! I love your blog.
Cheers! Sandra. R.
September 10th, 2009 at 9:21 am
I love your site.
Love design!!! I just came across your blog and wanted to say that I?ve really enjoyed browsing your blog posts. Sign: ndsam
September 11th, 2009 at 8:49 am
Sign: umsun Hello!!! rcuwwymhyw and 7634ssgfhphzye and 6662Hi! I was surfing and found your blog post? nice! I just came across your blog and wanted to say that Ive really enjoyed it.
September 25th, 2009 at 6:07 am
Same problem for me. I am no more indexed with Google and it is frustrating I cannot do much about it.
October 12th, 2009 at 2:44 pm
This is annoying, as I normally like to have my search pages indexed also, and is also preventing me from being indexed by google.
October 15th, 2009 at 2:51 pm
Thank you. This is what information I need. I need to make robots.txt for my new blogger site.
October 17th, 2009 at 8:35 pm
How Come in webmaster tools, I can not set up a sitemap ?
http://tellme42.blogspot.com/feeds/posts/defaults?
October 17th, 2009 at 8:37 pm
/feeds/posts/default I mean!
October 17th, 2009 at 8:41 pm
my other site, http://tellme22.blogspot.com/, had no problem?
November 25th, 2009 at 2:52 am
[... - http://www.meetarpit.com is another great source of information on this subject,[… -
February 5th, 2010 at 9:29 am
I have just started my blog and i am waiting for its indexing in google and what i see is mt search phrases are not crawled by google bots .Will it effect the indexing that is timing to index my site in google .What i think is google is not fool because they know if this thing effected there blogs ranking then people will shift to there own ftp servers leaving blogger what i think is all is done to avoid indexing of duplicate content .
Lets Hope for indexing of my blog also .
February 7th, 2010 at 6:21 am
I have the same problem , i had an error message while Google spider was crawling my blog. But from the information that are here , i guess i cannot do nothing about it.
February 25th, 2010 at 4:53 am
My blog also having same problem..
http://geologist-jobs.blogspot.com/robots.txt
how to edit robot
May 26th, 2010 at 12:57 am
i have same problem.can i delete it?