# Disallow storage and irrelevant directories User-agent: * Disallow: /audio/ Disallow: /banners/ Disallow: /cgi-bin/ Disallow: /Deanna/ Disallow: /deleted/ Disallow: /dev/ Disallow: /dev2/ Disallow: /nav/ Disallow: /old_WAS/ Disallow: /pictures/ Disallow: /psds/ Disallow: /Templates/ Disallow: /wasweb/ # Disallow Collectors and Spam User-agent: atSpider Disallow: / User-agent: cherrypicker Disallow: / User-agent: DSurf Disallow: / User-agent: EliteSys Entry Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Mail Sweeper Disallow: / User-agent: munky Disallow: / User-agent: Roverbot Disallow: /