User-agent: Sensis Disallow: / User-agent: omgilibot Disallow: / User-agent: Speedy Disallow: / User-agent: BabalooSpider Disallow: / User-agent: Baiduspider Disallow: / User-agent: baiduspider Disallow: / User-agent: baidu Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: boitho Disallow: / User-agent: Twiceler Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: YodaoBot Disallow: / User-agent: Sogou Disallow: / User-agent: ia_archiver Disallow: / User-agent: IRLbot Disallow: / User-agent: Exabot Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: slurp Crawl-delay: 900 User-agent: BecomeBot Disallow: / User-agent: BoardTracker Disallow: / User-agent: Gigabot Disallow: / User-Agent: MJ12bot Disallow: / User-agent: Nutch Disallow: / User-agent: psbot Disallow: / User-Agent: Omgili Disallow: / User-agent: Slurp Disallow: / User-Agent: Speedy Disallow: / User-agent: Sproose Disallow: / User-agent: msnbot Disallow: User-agent: msnbot/1.0 Disallow: User-agent: msnbot-media/1.0 Disallow: User-agent: bingbot Disallow: User-agent: ia_archiver Disallow: /