User-agent: * #Standard Disallow: /cache/ Disallow: /cgi-bin/ Disallow: /errormsg/ Disallow: /file/ Disallow: /mailout/ Disallow: /site/resource/ Disallow: /web/resource/ Disallow: /web-service/ #Custom