Robots.txt File to Blogger: How to Add Custom Robots.txt File

0
How to Add Custom Robots.txt File to Blogger
How to Add Custom Robots.txt File to Blogger

If you do not know what a Robots.txt file is? And how to add Custom Robots.txt file to Blogger, then you read this article completely.

This is very important for the SEO of your blog, due to its incorrect use, your blog can be completely removed from the search engine. Follow this guide to know the benefits and use of Robots.txt File.

The direct contact of the robots.txt file is related to the indexing and crawl rate of your blog. Generally all search engines have their own robots or bots that crawl your blog and the links given in it and index it in the search engine. Because of which whatever you update in the blog, the entire update also shows in the search engine, and what crawl will happen in your blog, it control robot.txt.

Bluehost

What is robots.txt

Robots.txt is a text file with some simple code, which gives instructions to web crawlers which pages to crawl in your blog and which do not.

This file is saved in the server of any website or blog so that search engine bots can be managed. It is mainly used to prevent any non-importance webpages (Demo page, Archive, Labels etc.) from crawling.

Semrush Free Trial

In simple language, through robots.txt file, you have the authority of your blog, which page you want to show in the search engine and which page you want to hide.

Robots.txt File to Blogger

The Note-Search engine scans the robots.txt file before crawling any web page.

By default Blogger’s robots.txt file looks like this, as shown in the screenshot. To see the Custom Robots file, after slipping (/) after the homepage URL of your blog, type robots.txt and press Enter.

robots.txt
robots.txt

Now let us see what is the meaning of this Robots.txt code?

User-agent: Mediapartners-Google

The Ultimate Managed Hosting Platform

This code is for Google AdSense robots, which manages what kind of AdSense ads will come on your blog and website.

User-agent: *
This code means all search engine bots can crawl our blog. Asterisk (*) means all robot (googlebot, yahoo etc).

Disallow: / search

This means that wherever the first search keyword comes after your domain name, the search engine will not crawl it.

Example– http://www.rnkhabri.com/search/label/Blogging
This link is the label name (Menu) of our blog. Similarly, all the labels in our blog will not be indexed.

Allow: /
This code helps to index your blog’s homepage by crawl.

Sitemap
Sitemap gives search engine information about all the existing and new posts of our blog. That is, whenever the web crawler scans our robots.txt file, sitemap will get information about all our published posts.

How to Add Custom Robots.txt File Blogger

To add a Custom Robots.txt file to Blogger, first of all login to Blogger and come to Dashboard.

  • Now you go to Settings and click on Search Preferences option.
  • In this page you will find a section Crawlers and indexing, where there will be an option of Custom robots.txt [Disabled]. To enable it, you have to click on Edit and tick Yes.
  • After this, copy the code given below and paste it in the box of Custom robots.txt and click on Save Changes.

Copy Robots.txt code

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: / search
Allow: /
Sitemap: http://example.com/sitemap.xml

Note: Instead of http://example.com/sitemap.xml inside Sitemap, you use sitemap of your blog.

robots.txt file
robots.txt file

I have explained the entire process of adding Custom Robots.txt file to your Blogger here, you can also add robots.txt file to your blog by following it.

If you want to hide a post or page from here, then you should have proper knowledge of robots.txt because the misuse of its blog can completely damage the SEO of your blog. Do not tamper with its settings at all without knowledge.

Read More:

If you have any question related to custom robots txt, then you can ask in the comment and for similar blogging tips you can like our Facebook page.