Prompting you the best.....
Learn how to create a custom robots.txt file for you Blogger/Blogspot blog. You can easily set robots.txt to best settings for getting great SEO results.
Anybody can do blogging — it’s fun! But if you want to get into professional blogging, you would need to learn some technical details related to search engine optimization. New bloggers often begin their blogging journey by posting content on Blogger/Blogspot. Enabling a custom robots.txt file on these platforms is very crucial. Often people ask about what would be the best custom robots.txt settings for their blog. Let’s learn it all today!
Robots.txt is a simple text file that informs web crawlers (also known as spiders or bots) about which parts of a website or blog to be crawled and which parts should not be crawled.
Robots.txt is a text file on the server in that you can write which directories, web pages, or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt is available for Blogger. In Blogger search option is related to Labels. If you are not using labels wisely per post, you should disallow the crawl of the search link. In Blogger, by default, the search link is disallowed to crawl. In this robots.txt, you can also write the location of your sitemap file. A sitemap is a file located on a server that contains all posts’ permalinks of your website or blog. Mostly sitemap is found in XML format, i.e., sitemap.xml.
Presently Blogger is working on sitemap.xml. Now Blogger is reading sitemap entries through the feed. By this method, the most recent 25 posts are submitted to search engines. If you want search engine, bots, only work on the most recent 25 posts, and then you should use robots.txt type 1 given below. If you set robots.txt like this, then Google Adsense bot is allowed to crawl the entire blog for best Adsense performance.
Why Robots.txt File is Important?
Well, the success of any professional blogs usually depends on how Google search engine ranks your blog. We store a number of posts/pages/files/directories in our website structure. Often we don’t want Google to index all these components. For example, you may have a file for internal use — and it is of no use for the search engines. You don’t want this file to appear in search results. Therefore, it is prudent to hide such files from search engines.
Robots.txt file contains directives which all the top search engines honor. Using these directives you can give instructions to web spiders to ignore certain portions of your website/blog.
Robots.txt Type 1
User-agent: Mediapartners-GoogleDisallow: User-agent: *Allow: /Disallow: /search Disallow: /bSitemap: https://www.aruwaab9ja.com.ng/sitemap.xml
Note: You may alter Blogspot’s default robots.txt like above.’
Robots.txt Type 2
User-agent: Mediapartners-GoogleDisallow: User-agent: *Allow: /Disallow: /search Disallow: /bSitemap: https://www.aruwaab9ja.com.ng/feeds/posts/default?orderby=updated
Note: Don’t forget to change the https://www.aruwaab9ja.com.ng with your blog address or a custom domain. If you want search engine bots to crawl the most recent 500 posts, then you should need to use following robots.txt type 2. If already you have more than 500 posts on your blog, then you can add one more sitemap line highlighted in red. Robots.txt
Robots.txt Type 3
User-agent: Mediapartners-GoogleDisallow: User-agent: *Allow: /Disallow: /search Disallow: /bSitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=1&max-results=500Sitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=501&max-results=500
Note: Don’t forget to change the https://www.aruwaab9ja.com.ng with your blog address or a custom domain.
Mathematical expression for Blogger robots.txt sitemap entries:
Sitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=(m*0)+1&max-results=mSitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=(m*1)+1&max-results=mSitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=(m*3)+1&max-results=mSitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m . . .Sitemap: https://www.aruwaab9ja.com.ng/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m
Where m=500 and n=1, 2, 3, 4,…, n. If you have organized post labels in a good format and good experience of search engine optimization (SEO), then you can remove the following line –
Disallow: /search
Most important: if you don’t want to submit any blogger post or page to search engines then you can add them like that: For Post add a line like that –
Disallow: /yyyy/mm/post-name.html
For Page add a line like that –
Disallow: /p/page-name.html
Manage Blogger custom robots.txt
For this, please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes To get a better understanding of this you can take reference of the image given below:
Sitemap
Sitemap is a very important file in your website/blog. This file contains the structure of your website. It helps web crawlers to find their way through your blog. The Sitemap: directive tells the crawler the location of your sitemap file. In case of Blogger/Blogspot — you can leave this line as it is.
This is it! Should you have any questions regarding custom robots.txt file for Blogger/Blogspot, do let me know in the comments section. I will try my best to assist you. Thank you for connecting with Aruwaab9ja
ALWAYS STAY UPDATED WITH THE LATEST NIGERIAN NEWS FROM ARUWAAB9JA.COM.NG
Post a Comment
Post a Comment