10. Gzip Compression: Empower Gzip compression with your server to decrease the measurement of data files transferred involving the server plus the user’s browser. txt file is then parsed and can instruct the robotic as to which internet pages are usually not to get crawled. For a internet search engine https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s