|
Joined: Jun 2006
Posts: 9,242 Likes: 1
Former Developer
|
Former Developer
Joined: Jun 2006
Posts: 9,242 Likes: 1 |
650 search engine spiders in the past half hour, wow.
|
|
|
|
Joined: Jun 2006
Posts: 9,242 Likes: 1
Former Developer
|
Former Developer
Joined: Jun 2006
Posts: 9,242 Likes: 1 |
Make that 665. If this keeps up, we'll break our most online record, that was set only 2 days ago
|
|
|
|
Joined: Jun 2006
Posts: 1,344
veteran
|
veteran
Joined: Jun 2006
Posts: 1,344 |
LOL, they sure are crawling. I had 350 the other night
|
|
|
|
Joined: Nov 2006
Posts: 3,095 Likes: 1
Carpal Tunnel
|
Carpal Tunnel
Joined: Nov 2006
Posts: 3,095 Likes: 1 |
672 Search Spiders right now
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
680 How can you possibly have this many ?
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jun 2006
Posts: 3,837
Carpal Tunnel
|
Carpal Tunnel
Joined: Jun 2006
Posts: 3,837 |
Aren't you the lucky ones LOL
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
693 Search Spiders. here. repost but more relevant to this topic nope 704 Search Spiders. nope 706 Search Spiders.
Last edited by Mark S; 06/21/2007 2:28 PM.
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
I've been getting the same thing recently. Heck of a lot of spider activity.
My site got a new record at 6.04am the other day.
I didn't see the details, but I estimate there were 500+ spiders online.
(My site is 90% UK - who's online at that time?)
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
TAMU.edu - Search spider that's a new one searching my forums
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
Yes, I have that one also. Just one instance now and again though - not exactly a Yahoo competitor.
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
Fancy a American University creaming info from our site lol
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
Actually, I've just checked and I have 250(ish) spider - all Yahoo.
What do you think has brought on this sudden frenzied spidering?
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
Following Marks insightful comments on the content of my site (ahem!) I did a search for the word porn. And yes - there were 2 results... - One regarding a Digital inflection rigidometer. Users view porn to get 'started'
- The other when a competitor website tried to trash my site by saying that it was affiliated to a porn site. (One of my members was on the committee of a website that became defunct and their forum was overrun with porn spammers.
Check your boards - do you have 'porn' on them?
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
OMG
I have 8 pages "Max" limit reached lol
But i have topics called NOT PORN And a game called Porn Star name too lol
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
Rose is going to be very upset to hear of this Mark. She thought you were sooooh sweet Showing your true colours now, mate
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
Its not me it my users lol
And its all clean lol
Rose will understand, she has too or life just aint worth living lol
The game 3 words after sex don't help either
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
|
Joined: Nov 2006
Posts: 3,095 Likes: 1
Carpal Tunnel
|
Carpal Tunnel
Joined: Nov 2006
Posts: 3,095 Likes: 1 |
The game 3 words after sex don't help either Maybe you don't know the 3 correct words. YOU'RE RIGHT DEAR
|
|
|
|
Joined: Apr 2007
Posts: 3,940 Likes: 1
Former Developer
|
Former Developer
Joined: Apr 2007
Posts: 3,940 Likes: 1 |
hrm. technically that's 4
(I KEED)
|
|
|
|
Joined: Nov 2006
Posts: 3,095 Likes: 1
Carpal Tunnel
|
Carpal Tunnel
Joined: Nov 2006
Posts: 3,095 Likes: 1 |
Not really, it's a contraction for you are
So, debatable on many sites (not an issue I care to debate though)
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
456 se spiders; all but 1 being yahoo lol
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
I had two pages of it. But only used in the most innocent of ways. Speaking of search bots, I have a question. I don't really get many search bots - maybe four or five at a time. I've always thought it was because I don't have the following checked in the CP. Enable Spider-friendly URLs? Warning: This function allows search engine spiders to crawl the community, which may result in higher bandwidth usage and server load. Note: The PATH_INFO environmental variable must be available for this feature to function properly. The reason I don't have this checked, is because I don't want individual posts coming up in search results in Google and other engines. Do I have a correct understanding of this function? Just wondering.
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
SE Friendly urls make it "easier" for SE's to troll your board, yes; though not impossible
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
But am I right in thinking that if I have that box checked, individual posts will appear in search engine results?
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
All that box will do is help SE's spider pages easier; they can spide without the option.
And if pages are included in google then yes, individual posts will be in there too.
You'll find that even the most mudayne posts will bring in users from search engines lol
|
|
|
|
Joined: Apr 2007
Posts: 3,940 Likes: 1
Former Developer
|
Former Developer
Joined: Apr 2007
Posts: 3,940 Likes: 1 |
hrm. so if i post sex slave midget and then a real serious post, google will bring em on in.. note to self.. preface all posts with above words
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
lol, kinda ... Our most popular incoming thread at UGN?? HotChicks thread... Kinda like UBBCentral's longest thread lol
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
Well, my users are totally against having individual posts listed in search engines, and so am I really. I guess I'll leave that unchecked.
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
well, see, it's not just individual posts; it's the entire topics (and as individual posts are a PART of the topic, they're included as well).
BTW, nothings to say that they won't crawl it anyway, unless you have a robots.txt rule to specificaly tell search engines not to crawl your forums.
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
I've tried searching for posts on Google, and they don't come up. People talk about some real personal stuff on my forums, and I'd really hate to see those come up in search results.
I do have a robots.txt, but I don't restrict the forums.
|
|
|
|
Joined: Apr 2007
Posts: 3,940 Likes: 1
Former Developer
|
Former Developer
Joined: Apr 2007
Posts: 3,940 Likes: 1 |
do you restrict anonymous users? ie: are they forced to sign up to see forum info?
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
To protect your forums, if you actually don't want SE's crawling things, you should add this to your robots.txt... User-agent: *
Disallow: /forums
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
SD, anonymous users can read the boards, but they can't post without registering. Why do you ask?
Giz, I allow them to crawl the forums - in fact I want the forums to be listed in the search engines. I just don't want individual posts to be crawled. I thought that by not checking that checkbox, it prevented that. I could be right though - because I haven't seen an individual post from my forums in a search listing since I've had UBB.
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
nonono... ok...
REGUARDLESS of if you have SE friendly enabled or not, you will get crawled. SE links just help the process along and streamline the crawling of content.
If THREADS are indexed, invididual posts are as well, as posts are a PART of the thread...
The content of which the search engines use to bring you new users is from the individual posts.
If you WANT your site crawled by SE's you should enable SE frinedly URLs to help the process, though by not enabling them you're still allowing se's to crawl, it'll just be more difficult for them to do so...
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
BTW, some of your posts/threads are in google: clicky clicky
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
Yikes!
Ok .. but who will ever use that search term? I took a phrase out of one of those posts, and put it into Google, and it didn't come up in the results list.
So, the REASON for that check box, is to make it easier for bots?
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
It won't nessessarily return just any term for just any search; it's kind of hard to explain briefly..
Well, the "reason" for the box is to clean up how URL's look, and how url's are fed to the browser/client requesting them; it's just that the "cleanups" work well with search engines; the initial cleanups in previous code where based off of proven techniques...
If you surf around UBBDev about 2-3 years ago you'll see me talking about search engine spiders and a paid search engine hack that I was using at the time; this was just before SE friendly url's where introduced into the feature set (of at least classic; I don't know when they where put into threads as I wasn't a threads6 user).
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
addict
Joined: May 2006
Posts: 579 |
Thanks Giz .. I understand now. I've always found SE stuff difficult to understand, although I recently went right through the Google Webmaster tools, doing everything they recommended for ratings and exposure. I am quite serious about not having individual posts appear in listings, although what I thought was preventing that looks as though it might be incorrect. I might do as you suggest, Giz, and use the robots.txt to prevent the reading of individual forums, but then activate that checkbox. Is that what you'd recommend?
|
|
|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
veteran
Joined: Dec 2006
Posts: 1,235 |
hrm. so if i post sex slave midget and then a real serious post, google will bring em on in.. note to self.. preface all posts with above words Or use the "invisible post footer" option. Left click and drag from here > sex slave midget hotchicks < to here.
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
Joined: Jul 2006
Posts: 4,057 |
quality i like that trick
BOOM !! Version v7.6.1.1 People who inspire me Isaac ME Gizmo
|
|
|
Bots
by Outdoorking - 04/13/2024 5:08 PM
|
|
|
|
|
|
1 members (1 invisible),
702
guests, and
193
robots. |
Key:
Admin,
Global Mod,
Mod
|
|
|
|