Most, but not all, bloggers eagerly anticipate the arrival, upon their blog, by the search engine spiders. The spiders come, when the search engines recognise your blog's existence in the Blogosphere, to index your blog.
For those bloggers who don't want the spiders indexing their blogs, Blogger provides the settings, in Settings - Basic, "Add your blog to our listings?" (for internal Blogger spiders, and Blogger links, such as "Next Blog") and "Let search engines find your blog?" (for external spiders).
These settings control the content of the "robots.txt" file in the blog. You cannot directly modify this file, as you have no access to the files and folders in the blog. So, you make the settings changes, and let Blogger maintain "robots.txt" on your behalf.
These settings are at the blog level - one setting affects the entire blog. If you would like only a part of the blog protected, make a second blog (blogs are free), and include one blog in the other. If you would like specific URLs protected, use the URL removal tool in Google Webmaster Tools.
If you use Google Webmaster Tools, maybe to add a sitemap or otherwise analyse or maintain your blogs search engine relationship, you may see some interesting details
Those restrictions are normal. All label searches are restricted, so the search engines won't detect the label searches as containing duplicate content. Your blog shouldn't depend upon label searches for your readers to find each post.
Interestingly, I note that the search engines can have access to blogs that require permission to read. Private blogs can't have blog feeds, but the search engines can still index them. The "robots.txt" file is advisory only; search engines may honour the files directives, or they may ignore the directives.
For those bloggers who don't want the spiders indexing their blogs, Blogger provides the settings, in Settings - Basic, "Add your blog to our listings?" (for internal Blogger spiders, and Blogger links, such as "Next Blog") and "Let search engines find your blog?" (for external spiders).
These settings control the content of the "robots.txt" file in the blog. You cannot directly modify this file, as you have no access to the files and folders in the blog. So, you make the settings changes, and let Blogger maintain "robots.txt" on your behalf.
These settings are at the blog level - one setting affects the entire blog. If you would like only a part of the blog protected, make a second blog (blogs are free), and include one blog in the other. If you would like specific URLs protected, use the URL removal tool in Google Webmaster Tools.
If you use Google Webmaster Tools, maybe to add a sitemap or otherwise analyse or maintain your blogs search engine relationship, you may see some interesting details
URL restricted by robots.txt (http://myblog.blogspot.com/search/label/mylabel)and you'll generally have one of these notations for each label in the blog.
Those restrictions are normal. All label searches are restricted, so the search engines won't detect the label searches as containing duplicate content. Your blog shouldn't depend upon label searches for your readers to find each post.
Interestingly, I note that the search engines can have access to blogs that require permission to read. Private blogs can't have blog feeds, but the search engines can still index them. The "robots.txt" file is advisory only; search engines may honour the files directives, or they may ignore the directives.
Nice blog! We provide top-quality pest control services in Zirakpur. Get pest control services at a reasonable price. Our mission is to safeguard the customer's house and building from pests, which cause many diseases.
ReplyDelete