What is robots.txt and how do i use it
The robots.txt (robots exclusion standard) file is a file located in the root of a web server which tells web crawlers what they are allowed to crawl, it can be used to protect or hide folders and sections from your website if you want to prevent inclusion in a search engine. This is useful for …