Robots.txt is usually a file that exists on the basis Listing of each website and can be employed to instruct search engines like google and yahoo on which directories/data files of the web site they can crawl and consist of of their index. Search engines like google and yahoo have https://news.rafeeg.ae