UBB.threads PHP Forum Software Community Forums UBB.threads 7 Version 7 Support Search Spiders Viewing My List of Forums!

Hi there,

I went into my community today to find a search spider was viewing my list of forums. Is this okay? I immediately closed the entire board. My board is very new and I'm still adding board rules and moderator training info. But, I left it open yesterday just to see if anyone would even come by.

Since a search spider was noisy-ing around, is this why all users should post a fake email when they post? So the spiders don't send them spam? If so, I will definitely make this part of the board rule training.

Thank you very much.

Ms. Letha - Rye, Co

Good question.

I often see Yahoo! Viewing their "My Home" which according to online.php is the Control Panel.

How can this be?

After checking the "details" area of "Who's Online," it appears clear that this program was developed with the intention of allowing "Search Spiders" so this must be a good-thing; especially since we can see them when they arrive and see which screens they're looking at. I suppose if they are the "good-spiders," it will also be reported to the appropriate search engines. I'm not really sure.

So, it must've been intentionally programmed into ubbthreads community so that the owners, admins, moderators and users can all see this too. This must be okay. Afterall, it's on this website of forums too!

I think the search spiders are not harmful to us; otherwise, why would they be here also.

Are there good search spiders and bad search spiders?

Can they get into my control panel? Thanks,

Not all search spiders are mallicious, in fact, most won't touch anything but text on your site. If you don't want spiders crawling your member data, you should disable the ability for guests to view profiles.

As for "is this normal" yes, anything on the internet that is public to all guests should eventually get crawled by a spider; it's completely normal, they crawl data, it's what they're designed to do... even when you don't tell them to (as they come in from links from other pages or other sites).

There are ways to "blackhole" bad spiders, I made a post on this like a year ago, it's here somewhere... I never really looked into integrating it with the ubb, but it'd at least build a list of bad spiders for you to block manually.

Good spiders: crawl your content, based on criteria you specify (read up on robots.txt)
Bad Spiders: crawl your content, most take what they want (and completely ignore your robots.txt)
Mallitious Spiders: grab email addresses and user data (hence why there is the ability to block guests from accessing profiles).

Thanks Gizmo,

So as long as I only allow guests to read the posts, the spiders can't open any profiles?

A spider is a guest on your forum, if a guest can see it, a spider can as well; so if you log out and try to access a page, if you can without being logged in, a spider can view it as well.