Blogger introduces robots.txt file
Google Blogger has introduce the robots.txt file in each blogger blog. The structure of the robots.txt file is
User-agent: *Disallow: /search
Some times back when blogger.com upgaraded to new Google blogger, new lable feature was added in blogger blogs. Each lable has its own page. This robot file make sure that lable pages are not crawled and indexed by any search engine bot. This will certainly help search engines to avoid crawling millions of duplicate lable pages.
To check the robots.txt file of your blogger blog, just type the following URL in the adress bar of your browser.
http://www.yourblogname.blogspot.com/robots.txt
(replace yourblogname with your blog name).
July 16th, 2007 at 9:04 pm
I just noticed this too, This is really bad for me as I would like to have people be able to find my search pages in google, e.g. search for “flora nature photography” and come up with my flora search page. Well, there’s nothing I can really do about it. Maybe I’ll change to wordpress…
July 16th, 2007 at 9:07 pm
P.S. My other blog, thebigbyte.blogspot.com is going to lose a lot of traffic from this as well, as most of my traffic to that site came from google searches for the blogger search page.
July 16th, 2007 at 9:44 pm
David, this robots.txt file will not get your blog de-indexed in Google.. It will just disallow search engines to crawl the label pages (category pages). They are jst the duplicate content page. Your original post page and blog home page will still be there in Google.
July 16th, 2007 at 10:41 pm
Yep, I was aware of that, I’m just disappointed that Google search results will no longer include my category pages, as they gather together a range of related items together that users would otherwise have to search for.
This is good news for my site-specific custom Google search though – it will no longer show duplicate items, which I think is the whole point of this change. I would like to be able to choose whether or not this happens though.
July 17th, 2007 at 10:20 am
BTW, Is there a way to edit robots.txt ?
July 17th, 2007 at 11:50 am
No.. you cannot edit robots.txt file in blogger blog. A default robot file is added in every blog.
July 30th, 2007 at 1:36 pm
It has not been added to my blog. it is disallowing web crawlers to crawl my blog.
July 30th, 2007 at 1:41 pm
Rupesh, if you are talking of your blog fundu bytes, the let me tell you that robots.txt file is added in it.
just check the URL fundubytes.blogspot.com/robots.txt
Also if you check the source code of any page of your blog, you will find the line
<META content=’index,follow’ name=’robots’></META>
in the head section.
So I don’t that there will be any problem in indexing of your blog.
July 31st, 2007 at 7:12 am
it is causing problem… my blog is not listed on google anymore…blog’s pagerank has also dropped… google webmaster tools is showing errors..
July 31st, 2007 at 7:36 am
Well Rupesh
First of all please mention the blog URL which you are talking about. For now am assuming that its the one which is linked to your name here (i.e. fundubytes.blogspot.com).
If that is the case, then you can see that it is well indexed in google.
Just use the query site:fundubytes.blogspot.com in google and see the results.
As for pagerank, its a complex algo and may change depending upon backlinks and other factors.
August 10th, 2007 at 12:35 pm
but i have on problem, how can i change the robot.txt file
August 10th, 2007 at 12:39 pm
Maher you cannot edit robots.txt file in blogger. Anyways whats your problem?
feel free to ask for any help.
November 27th, 2007 at 5:56 pm
Hi, hello, privet
munificently
December 15th, 2007 at 9:35 pm
very interesting, but I don’t agree with you
Idetrorce
January 18th, 2008 at 3:45 am
rtyrdt
January 18th, 2008 at 3:48 am
tyur5e46856
January 18th, 2008 at 3:48 am
ytfufio7tryu8
March 18th, 2008 at 4:24 am
Arpit. Does the robot.txt affect SERPs? Cause it did to all my blogger blogs once i found out the errors.
March 24th, 2008 at 4:38 am
I’d prefer reading in my native language, because my knowledge of your languange is no so well.
April 7th, 2008 at 2:22 am
Can I edit my robots.txt file for the blogger’s blog??
August 7th, 2008 at 5:21 am
No the robots.txt is not editable with Blogger. Anyway the crawling of Labels are not necessary if your posts are cache by the Google. Nobody knows what’s the trick to login at http://ftp.blogger.com I think it’s because of robots.txt
August 25th, 2008 at 5:45 am
That Google does not give access to robots.txt is not well. In this sense that the categories in our blogs remain invisible unless we have them shown as tags under the titles of articles or as an additive in the sidebar! I hope think any decision that all be happy! Because this limitation because of WordPress are a step forward to the Blogger!
November 6th, 2008 at 3:11 am
I moved my blog to the WordPress and I’ve an own FTP hosting. Now everything is much better
November 21st, 2008 at 9:47 am
This thing sux…makes less publicity of blog
http://blog.kpsofts.com
see my robots.txt file
January 2nd, 2009 at 8:09 pm
i am realy tired to busy with blogger blogs. /search is not permitted bu how can increase my site navigation for the search engines. There is no mean to labelled the post for the SEO. What a pity!
April 26th, 2009 at 2:47 am
thanks for awesome article like this, i trying to chcek it for my blogspot.
June 24th, 2009 at 12:46 pm
thanks for robots.txt information. this is veru usefull
August 30th, 2009 at 4:38 pm
I never ever post but this time I will,Thanks alot for the great blog.
September 10th, 2009 at 7:42 am
Hi! I was surfing and found your blog post… nice! I love your blog.
Cheers! Sandra. R.
September 10th, 2009 at 9:21 am
I love your site.
Love design!!! I just came across your blog and wanted to say that I?ve really enjoyed browsing your blog posts. Sign: ndsam
September 11th, 2009 at 8:49 am
Sign: umsun Hello!!! rcuwwymhyw and 7634ssgfhphzye and 6662Hi! I was surfing and found your blog post? nice! I just came across your blog and wanted to say that Ive really enjoyed it.
September 25th, 2009 at 6:07 am
Same problem for me. I am no more indexed with Google and it is frustrating I cannot do much about it.
October 12th, 2009 at 2:44 pm
This is annoying, as I normally like to have my search pages indexed also, and is also preventing me from being indexed by google.
October 15th, 2009 at 2:51 pm
Thank you. This is what information I need. I need to make robots.txt for my new blogger site.
October 17th, 2009 at 8:35 pm
How Come in webmaster tools, I can not set up a sitemap ?
http://tellme42.blogspot.com/feeds/posts/defaults?
October 17th, 2009 at 8:37 pm
/feeds/posts/default I mean!
October 17th, 2009 at 8:41 pm
my other site, http://tellme22.blogspot.com/, had no problem?
November 25th, 2009 at 2:52 am
[... - http://www.meetarpit.com is another great source of information on this subject,[… -
February 5th, 2010 at 9:29 am
I have just started my blog and i am waiting for its indexing in google and what i see is mt search phrases are not crawled by google bots .Will it effect the indexing that is timing to index my site in google .What i think is google is not fool because they know if this thing effected there blogs ranking then people will shift to there own ftp servers leaving blogger what i think is all is done to avoid indexing of duplicate content .
Lets Hope for indexing of my blog also .
February 7th, 2010 at 6:21 am
I have the same problem , i had an error message while Google spider was crawling my blog. But from the information that are here , i guess i cannot do nothing about it.
February 25th, 2010 at 4:53 am
My blog also having same problem..
http://geologist-jobs.blogspot.com/robots.txt
how to edit robot
May 26th, 2010 at 12:57 am
i have same problem.can i delete it?
August 10th, 2010 at 11:35 am
Sign: wdpad Hello!!! zvnbm and 4229efawdkwveb and 7072 : Hi! I was surfing and found your blog post! nice! I just came across your blog and wanted to say that Ive really enjoyed it.
August 16th, 2010 at 10:46 pm
i got error Restricted by robots.txt for 25 pages in webmaster.how to solve this error.plz reply on my mail………..and how to solve duplicate meta tag error?
August 24th, 2010 at 11:39 am
Sign: zdbrw Hello!!! tfsme and 455ucpeymwqzh and 6936 : I love your site.
Love design!!! I just came across your blog and wanted to say that Ive really enjoyed browsing your blog posts.
October 13th, 2010 at 1:37 am
I also got shocked by seeing robot.txt file of blogger. I thought google still has not given robot.txt file. Still we can just see this file we can not edit file. I wondered when webmaster tool shown 29 pages crawl error of my blog. But when I read comment of Arpit. I said thank god ! google does not crawl labels pages because of duplicate content.
November 3rd, 2010 at 1:44 pm
Technorati is blocking my block form being part of their list due to the Robot.txt in blogspot. Is there any way to get around this so Technorati is not blocked by the robots.txt?
November 13th, 2010 at 8:09 am
My page title and snippet of the article not shown by Bing and Yahoo! search although it shows the hope page of key word every article but google works fine.
Is this due to this reason of robots.txt file. or if any other reason, if known please help.
User-agent: *
Disallow: /search
November 13th, 2010 at 8:12 am
errata:
Bing and Yahoo! always index only HomePage on every keyword of my article (written in my blog).
December 3rd, 2010 at 8:55 am
Hello, I don’t normally publish feedback on blogs, as I like to read only. But I find the article that you have written earlier has very insightful info, and I find it very informational. Anyway, I am questioning whether or not you’re open for link exchange, as I hope that we will agree on a mutual hyperlink exchange agreement. Hope to listen to a constructive reply from you, and have an awesome day!
January 3rd, 2011 at 6:03 am
i also have a problem, how can we edit robot.txt file?
January 21st, 2011 at 7:21 am
I didn’t realize that the robot.txt could not be edited in blogger. This is going to force me to change hosts and possibly move to WordPress. Google really messed up here as many people use labels for creating categories on their site, including my market research blog.
March 24th, 2011 at 6:32 pm
i guess there are commands to added as meta…
April 11th, 2011 at 7:52 pm
I was wondering if I could change the robot.txt in blogger and i got my answer here.. it surprises me that blogger is an amazing platform that allows a lot of features.. yet some basic ones seem always to be missing.. makes you wonder why? thanks again for the great blog!
June 7th, 2011 at 4:22 pm
This really sucks. Just getting into my first blog and reading that google is already starting to limit traffic to them.. Well anyway this is my blog:
The Elder Scrolls V blog
June 26th, 2011 at 9:06 pm
however, how do I edit the robots.txt is while I’m using free hosting from Blogger?
July 4th, 2011 at 10:15 am
I Change my blogger main-page into given search category. When anybody come on my blog it give the search result of mine.
July 10th, 2011 at 7:12 pm
Interesting debate about TXT robot or meta tags … Hmm… I was also hoping to find some easy way to change that robot txt for my blogspot blog labels… anyway this post inspired me a lot
July 22nd, 2011 at 6:25 pm
Oky, we can’t delete or alter the robots.txt itself, but are there any workarounds ? Webmaster Tools / GA reports some errors.
July 31st, 2011 at 1:41 pm
Googles this new feature has lot of lack – specially harming bloggers. In my blog too all labels has been blocked by robot.txt, I tried to find the code but didn’t get the simple code to edit the robot.txt by editing the source code of every page. If anyone know about codes, please inform.
Thank you.
August 2nd, 2011 at 5:14 pm
Hi,
Is robot.txt for blog is good or bad? please let me know how to label my blog, so that all the pages get indexed.
August 16th, 2011 at 3:27 am
I found that there are “URL is restricted by Robot.txt” errors in my Webmaster tool. This way traffic to my blog is not increasing. What can be done to remove this errors?
August 31st, 2011 at 6:26 pm
I typed in my blogspot url followed by “robot.txt” but page is is not found.
I see this post is 4 years old now, so maybe that has something to do with it.
I as placing in GSR’s pages 3 and 4 for my target keyword and then, POOF!
Gone from results all together… I stopped looking at page 50.
I’d really like to find a way to control the bot better and be able to tell it where to go and when to come back.
October 3rd, 2011 at 6:38 am
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://junkiemovies.blogspot.com/feeds/posts/default?orderby=updated
this is the problem what should i do ? should i skip blogger and go for word press or should i do something else?
November 8th, 2011 at 2:49 am
hi my site name is tech-syndrome.blogspot.com/robot.txt
SAME problem,,, how should we fix this or should we just move to word press?
November 9th, 2011 at 11:09 pm
Hi,
I checked my robots.txt file. It shows some thing like this
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
When I try to access data in google webmaster it shows robot.txt file blocked.
I am not getting any data in webmaster. What can be the problem and how can I fix it.
Please help.
December 13th, 2011 at 8:22 pm
i am also facing same problem, plz help…
January 15th, 2012 at 12:38 am
thanks.. i had robots issues with my blog.
January 24th, 2012 at 7:18 am
this is very valuable information. Thx a lot to bring this thing in my notice. Its very sad to know that it restricts search engines to crawl all pages. By the way i recently added the sitemap manually to google, yahoo, ask and some others and waiting for the pages to get indexed. I think that by manually adding your sitemap will result in more indexed pages and more traffic. Try to submit sitemap like this http://hitindiansongs.blogspot.com/rss.xml . I am looking forward for the engine
response and
January 26th, 2012 at 4:45 am
[...] buscas no Google devolveram apenas algumas discussões recentes em fóruns nos grupos do google e esta entrada de 14 de Julho num blog relativamente desconhecido, a mesma que me havia sido sugerida pelo Paulo [...]
January 26th, 2012 at 5:49 am
Thanx, this is great.