To avoid unwanted material from the lookup indexes, site owners can instruct spiders never to crawl particular data files or directories through the conventional robots.txt file in the basis directory of the area. On top of that, a site can be explicitly excluded from a internet search engine's database by https://news.rafeeg.ae