Add pages to your robots.txt file. Though, not all spiders obey the rules. Those rouge spiders do whatever they want.
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/templates/
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php/ubb/calendar.html
Disallow: /forums/ubbthreads.php?ubb=printthread
Disallow: /forums/ubbthreads.php/ubb/printthread
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php/ubb/showday.html
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php/ubb/online.html
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php/ubb/search.html
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php/ubb/faq.html
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies.html
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php/ubb/markallread.html
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser
Disallow: /forums/ubbthreads.php/ubb/newuser.html
Disallow: /forums/ubbthreads.php?ubb=contact
Disallow: /forums/ubbthreads.php/ubb/contact
Disallow: /forums/ubbthreads.php/ubb/contact.html
The robots.txt has zero effect on users linking to items or manually typing entries in to their browser.
The robots.txt file should be placed in your domain folder:
https://www.example.com/robots.txtFurther reading at:
http://www.robotstxt.org/robotstxt.html
