User-agent: * Crawl-delay: 120 User-agent: Slurp Crawl-delay: 2 User-Agent: fast Disallow: / User-Agent: ia_archiver Disallow: / User-Agent: ZyBorg Disallow: / User-Agent: slurp Disallow: / User-agent: girafa Disallow: / User-Agent: Openbot Disallow: / User-Agent: nabot Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: mogimogi Disallow: / User-Agent: sitecheck.internetseer.com Disallow: / User-Agent: Charlotte/2.00 (charlotte2@tinami.com; http://charlotte.tinami.com/robot.html) Disallow: / User-Agent: msnbot Disallow: / User-agent: Yeti Disallow: / User-agent: dotbot Disallow: /