What is wrong with the content of this robots.txt content as if I take out the disallow for Google bot then it again ignores the disallows for like events, calendar,etc ?
User-agent: *
Disallow: /cgi-bin/
User-agent: Mozilla/5.0 (Twiceler-0.9
http://www.cuill.com/twiceler/robot.html)
Disallow:/
User-agent: Slurp
Crawl-delay: 5
User-agent: dotbot
Disallow: /
User-agent: *
Crawl-Delay: 120
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
Using Google Webmaster tools website there is a throttle adjustment for crawler but I find it does not do much and if you do reduce the requests and increase a delay if you do not renew that function after 90 days then they go right back to flooding your domain with crawler requests
Also via this tools webpage and reviewing the robots.txt they say they ignore the Crawl-Delay: and it's value