There are various places in which search engines can get stuck, such as the calendar (as it increases indefinately); there are some robots.txt entries that you can add such as:
robots.txt
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php/ubb/calendar.html
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php/ubb/showday.html
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile.html
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers.html
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php/ubb/online.html
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php/ubb/search.html
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php/ubb/faq.html
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies.html
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php/ubb/markallread.html
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
Disallow: /forum/ubbthreads.php/ubb/newuser.html
Be sure to edit the path to your forums from your web root; mine for example is /forum but yours may vary.
Several items in this list aren't just bot traps, but places where SE's shouldn't be in the first place as it will just show an access denied error (as a SE is just a guest).
These include:
new user/registration page
Mark all Read
Clear cookies
View privacy
FAQ
search
who's online
show profile (you can remove this setting if you want SE's trolling your member data)
show members
calendar (including show day's events)
Previous Topic
Index
