To stay away from undesirable content from the search indexes, website owners can instruct spiders not to crawl sure documents or directories with the common robots.txt file in the basis directory from the domain. Also, a website page may be explicitly excluded from the search engine's databases by using a https://news.rafeeg.ae