Introduction
The robots.txt file tells program crawlers the URLs the crawler will access on your website. this can be used in the main to avoid overloading your website with requests. it’s not a mechanism for keeping an online page out of Google. to avoid wasting an online page from Google, block assortment with no index or password-protect the page.
What is crawler
A crawler may be a program that visits websites and reads their pages and alternative data so as to form entries for a quest engine index. the most important search engines on the net all have such a program, which is additionally called a “spider” or a “bot.” Crawlers’ area units are generally programmed to go to sites that their house owners submit as new or updated. Entire sites or specific pages will be by selection visited and indexed.
How the robots.txt file works
A robots.txt file is a document with no HTML markup code (hence the .txt extension). The robots.txt file is hosted on the online server similar to the other file on the website. In fact, the robots.txt file for any given website will usually be viewed by typewriting the total universal resource locator for the homepage and adding /robots.txt, like https://www.thecodehubs.com/robots.txt. The file is not coupled to anyplace else on the location, thus users are not doubtless to encounter it, however, most internet crawler bots can explore this file initially before traveling the remainder of the location.
Why do we use the robots.txt file?
- It blocks the contents from search engines.
- It tunes access to the positioning of respected robots.
- It is employed in presently developing websites, which require not show in search engines.
- It is wont to create content accessible to specific search engines.
Example of robots.txt file
User-Agent: * Disallow: /tag/ Disallow: */?*=* Disallow: /articles/page/* Disallow: /question/category/* Disallow: /mycred_*/ Disallow: /author/*/page/* Disallow: /login/ Disallow: /privacy-policy/ Sitemap: https://www.thecodehubs.com/sitemap_index.xml
Directive of the robots.txt file
-
User-Agent Directive
The first few lines in every block square measure the ‘user-agent’, that pinpoints a particular larva. The user-agent can match a particular bots name, thus for instance
User-agent: Googlebot
-
Host Directive
Even though some rumors claim Google does support the host directive, only Yandex currently does. By utilizing this directive, a user may choose whether to display the www. before a URL in this block.
Host: thecodehubs.com
-
Disallow Directive
The second line in an exceedingly block of directives is forbidden. you’ll use this to specify that sections of the positioning shouldn’t be accessed by bots. associate empty forbid means that it’s a fight, and therefore the bots will please themselves wherever they are doing and don’t visit.
Disallow: /tag/ Disallow: */?*=* Disallow: /articles/page/* Disallow: /question/category/* Disallow: /mycred_*/ Disallow: /author/*/page/* Disallow: /login/ Disallow: /privacy-policy/
-
Sitemap Directive (XML Sitemaps)
Using the sitemap directive tells search engines wherever to seek out your XML sitemap. However, in all probability, the foremost helpful factor to try and do would be to submit everyone to the search engine’s specific webmaster tools. this is often as a result of you’ll be able to learn loads of valuable data from every concerning your web site. However, if you’re short on time, the sitemap directive may be a viable various.
Sitemap: https://www.thecodehubs.com/sitemap_index.xml
-
Crawl Delay Directive
Yahoo, Bing, and Yandex will be a touch trigger happy once it involves travel, however, they are doing answer the crawl-delay directive, that keeps them unfree for a short while. means you’ll be able to create the search engines wait 10 seconds before travel the positioning or 10 seconds before they re-access the positioning when travel – it’s primarily constant, however slightly completely different looking on the computer program.
Crawl-delay: 10
