# obp crawler
User-agent: Nutch
User-agent: NutchCVS
User-agent: ParkSpider
Disallow: /common/
Disallow: /includes/
Disallow: /tmp/
Disallow: /xml/