Dignity nobility high-quality state highness standing class caste position superiority priority fashion distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robotic as to which pages are certainly not to get crawled. As being a online search engine crawler https://www.youtube.com/watch?v=vd3_Ur7Wjks