UBB.threads PHP Forum Software Community Forums General Off Topic We're under attack from search engine spiders!!!

650 search engine spiders in the past half hour, wow.

Make that 665. If this keeps up, we'll break our most online record, that was set only 2 days ago

LOL, they sure are crawling. I had 350 the other night

672 Search Spiders right now

680 How can you possibly have this many ?

Aren't you the lucky ones

LOL

693 Search Spiders. here.

repost but more relevant to this topic

nope

704 Search Spiders.

nope

706 Search Spiders.

I've been getting the same thing recently.
Heck of a lot of spider activity.

My site got a new record at 6.04am the other day.

I didn't see the details, but I estimate there were 500+ spiders online.

(My site is 90% UK - who's online at that time?)

TAMU.edu - Search spider that's a new one searching my forums

Yes, I have that one also.
Just one instance now and again though - not exactly a Yahoo competitor.

Fancy a American University creaming info from our site lol

Actually, I've just checked and I have 250(ish) spider - all Yahoo.

What do you think has brought on this sudden frenzied spidering?

The word PORN lol

Following Marks insightful comments on the content of my site (ahem!) I did a search for the word porn.

And yes - there were 2 results...

• One regarding a Digital inflection rigidometer. Users view porn to get 'started'
• The other when a competitor website tried to trash my site by saying that it was affiliated to a porn site. (One of my members was on the committee of a website that became defunct and their forum was overrun with porn spammers.

Check your boards - do you have 'porn' on them?

OMG

I have 8 pages "Max" limit reached lol

But i have topics called NOT PORN
And a game called Porn Star name too lol

Rose is going to be very upset to hear of this Mark.

She thought you were sooooh sweet

Showing your true colours now, mate

Its not me it my users lol

And its all clean lol

Rose will understand, she has too or life just aint worth living lol

The game 3 words after sex don't help either

Maybe you don't know the 3 correct words. YOU'RE RIGHT DEAR

hrm. technically that's 4


(I KEED)

Not really, it's a contraction for you are

So, debatable on many sites (not an issue I care to debate though)

456 se spiders; all but 1 being yahoo lol

I had two pages of it. But only used in the most innocent of ways.

Speaking of search bots, I have a question. I don't really get many search bots - maybe four or five at a time. I've always thought it was because I don't have the following checked in the CP.

Enable Spider-friendly URLs?
Warning: This function allows search engine spiders to crawl the community, which may result in higher bandwidth usage and server load.

Note: The PATH_INFO environmental variable must be available for this feature to function properly.

The reason I don't have this checked, is because I don't want individual posts coming up in search results in Google and other engines. Do I have a correct understanding of this function?

Just wondering.

SE Friendly urls make it "easier" for SE's to troll your board, yes; though not impossible

But am I right in thinking that if I have that box checked, individual posts will appear in search engine results?

All that box will do is help SE's spider pages easier; they can spide without the option.

And if pages are included in google then yes, individual posts will be in there too.

You'll find that even the most mudayne posts will bring in users from search engines lol

hrm. so if i post sex slave midget and then

a real serious post, google will bring em on in..

note to self.. preface all posts with above words

lol, kinda ...

Our most popular incoming thread at UGN?? HotChicks thread... Kinda like UBBCentral's longest thread lol

Well, my users are totally against having individual posts listed in search engines, and so am I really. I guess I'll leave that unchecked.

well, see, it's not just individual posts; it's the entire topics (and as individual posts are a PART of the topic, they're included as well).

BTW, nothings to say that they won't crawl it anyway, unless you have a robots.txt rule to specificaly tell search engines not to crawl your forums.

I've tried searching for posts on Google, and they don't come up. People talk about some real personal stuff on my forums, and I'd really hate to see those come up in search results.

I do have a robots.txt, but I don't restrict the forums.

do you restrict anonymous users? ie: are they forced to sign up to see forum info?

To protect your forums, if you actually don't want SE's crawling things, you should add this to your robots.txt...

SD, anonymous users can read the boards, but they can't post without registering. Why do you ask?

Giz, I allow them to crawl the forums - in fact I want the forums to be listed in the search engines. I just don't want individual posts to be crawled. I thought that by not checking that checkbox, it prevented that. I could be right though - because I haven't seen an individual post from my forums in a search listing since I've had UBB.

nonono... ok...

REGUARDLESS of if you have SE friendly enabled or not, you will get crawled. SE links just help the process along and streamline the crawling of content.

If THREADS are indexed, invididual posts are as well, as posts are a PART of the thread...

The content of which the search engines use to bring you new users is from the individual posts.

If you WANT your site crawled by SE's you should enable SE frinedly URLs to help the process, though by not enabling them you're still allowing se's to crawl, it'll just be more difficult for them to do so...

BTW, some of your posts/threads are in google:
clicky clicky

Yikes!

Ok .. but who will ever use that search term? I took a phrase out of one of those posts, and put it into Google, and it didn't come up in the results list.

So, the REASON for that check box, is to make it easier for bots?

It won't nessessarily return just any term for just any search; it's kind of hard to explain briefly..

Well, the "reason" for the box is to clean up how URL's look, and how url's are fed to the browser/client requesting them; it's just that the "cleanups" work well with search engines; the initial cleanups in previous code where based off of proven techniques...

If you surf around UBBDev about 2-3 years ago you'll see me talking about search engine spiders and a paid search engine hack that I was using at the time; this was just before SE friendly url's where introduced into the feature set (of at least classic; I don't know when they where put into threads as I wasn't a threads6 user).

Thanks Giz .. I understand now.

I've always found SE stuff difficult to understand, although I recently went right through the Google Webmaster tools, doing everything they recommended for ratings and exposure.

I am quite serious about not having individual posts appear in listings, although what I thought was preventing that looks as though it might be incorrect. I might do as you suggest, Giz, and use the robots.txt to prevent the reading of individual forums, but then activate that checkbox. Is that what you'd recommend?

Or use the "invisible post footer" option.

Left click and drag from here > sex slave midget hotchicks < to here.

quality i like that trick