txt file, on the other hand, tells engines like google what written content not to index (for instance, specific policy pages you don’t want to seem in look for.) Create the two to speed up the crawling and indexing https://medium.com/@linkdaddy