txt file is then parsed and will instruct the robot as to which webpages will not be to generally be crawled. Like a internet search engine crawler may preserve a cached duplicate of the file, it may on occasion crawl internet pages a webmaster won't need to crawl. Webpages normally prevented from being crawled include login-certain internet pages