Develop robots.txt file to authenticate and stop crawler from crawling everything apart from your own home website page. classification league group household classification tier course type buy established kind grade genus species bracket technology division sort description rubric breed race area wide range title manner branch specialty kidney subdivision subgroup https://hellopredict.com