# Directions for robots. See this URL: # http://info.webcrawler.com/mak/projects/robots/norobots.html # for a description of the file format. ##### # Here is where we override the default action # ##### # default action - currently it allows access to most of the site # skipping dynamic content or private areas # User-agent: * Disallow: /cgi-bin/ Disallow: /bob/cgi-bin/ Disallow: /mbsle/cgi-bin/ Disallow: /bob/directions/ Disallow: /fch-east/cgi-bin/ Disallow: /fch-west/cgi-bin/ Disallow: /fch-common/cgi-bin/ Disallow: /fch/cgi-bin/ Disallow: /phsst/cgi-bin/ Disallow: /uscho/ # If your robot is polite enough to request only a small number of # files at a time we will add your robot to the list of cool robots. # Send email to webmaster@bu.edu, referring to the /robots.txt # URL.