installing that robots.txt certainly made an immediate difference with the search engines.
besides reducing the total number of active spiders (esp. yahoo), the spiders left are now all engaged in indexing forums, rather than peripheral and unimportant sections of my site.
