Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
2 registered members (Gizmo, mmkk), 57 guests, and 388 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
M4TT
M4TT
Canada, SK
Posts: 51
Joined: March 2015
Show All Member Profiles 
Top Posters(30 Days)
Gizmo 12
M4TT 9
Ruben 8
mmkk 4
FREAK 3
isaac 3
Latest Photos
Chinese Buddhist temple.
My buddha beads.
Rendered Walls
Multi-Screen wallpaper
Stockholm Metro
Previous Thread
Next Thread
Print Thread
WebSpider List for Who's Online #196886
09/09/07 02:05 PM
09/09/07 02:05 PM
Joined: Aug 2006
Posts: 39
T
Thorsten Offline OP
newbie
Thorsten  Offline OP
newbie
T
Joined: Aug 2006
Posts: 39
Can somebody post his/hers webspiderlist from the admin panel?


Moderator Notice - There is an official Spider Listing at UBBDev that is maintained by the UBBDev staff.

Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: WebSpider List for Who's Online [Re: Thorsten] #196887
09/09/07 02:46 PM
09/09/07 02:46 PM
Joined: Jan 2004
Posts: 2,650
driv Offline

driv  Offline


Joined: Jan 2004
Posts: 2,650
Have a look here for starters...

Spider Link


Using version :: 7.6.0
Re: WebSpider List for Who's Online [Re: driv] #196907
09/09/07 09:30 PM
09/09/07 09:30 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Code
Alexa=ia_archiver
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
AllTheWeb=crawler@fast
Excite=ArchitextSpider
Gigabot=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Inktomi=Slurp
MSN=MSNBOT
Sogou=sogou web spider
Entireweb=Speedy Spider
Voila=Voila.fr
Ask.com=Ask Jeeves
Teoma=TeomaAgent
Wisenut=Zyborg
NorthernLight.com=Gulliver
Excite=Architext spider
AltaVista=Mercator
Crawler.de=Crawler
Infoseek=InfoSeek sidewinder
Lycos=Lycos_Spider_(T-Rex)
Search Hippo=Fluffy the Spider
Infoseek=Ultraseek
Looksmart=MantraAgent
Webcrawler.com=WebCrawler
Twiceler=Twiceler-0.9
Naver.com=Yeti/


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #196918
09/10/07 12:22 AM
09/10/07 12:22 AM
Joined: Nov 2006
Posts: 3,390
ntdoc Offline
Carpal Tunnel
ntdoc  Offline
Carpal Tunnel
Joined: Nov 2006
Posts: 3,390
The last one. Yeti/ is that correct?

Re: WebSpider List for Who's Online [Re: ntdoc] #196921
09/10/07 01:25 AM
09/10/07 01:25 AM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Yes, its agent is Yeti/0.1; I wanted to ensure it doesn't trigger on anything with a similar name (and didn't want to have to include the version string), so I left it as Yeti/


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #201956
11/24/07 05:34 PM
11/24/07 05:34 PM
Joined: Dec 2006
Posts: 1,420
UK
ScriptKeeper Offline

veteran
ScriptKeeper  Offline

veteran
Joined: Dec 2006
Posts: 1,420
UK
My updated spider list:
Quote:
Alexa=ia_archiver
Altavista=Scooter
Anzwers=AnzwersCrawl
Ask=Teoma
Atomz=Atomz
Boitho=boitho.com
Entireweb=Speedy Spider
Exalead=Exabot
Excite=ArchitextSpider
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=GigaBot
Google=Googlebot
Google-Image=Googlebot-Image
Yahoo!=Yahoo! Slurp
Infoseek=Ultraseek
Inktomi=Slurp
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Microsoft Research=MSRBOT
MSN=MSNBOT
NetSeer=Teemer
noXtrum=noxtrumbot
Searchme=Charlotte
Seznam=SeznamBot
Snap=Snapbot
Voila=VoilaBot
Walhello=appie
WISEnut=ZyBorg

.htaccess blocked list:
Quote:
RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX [OR]
RewriteCond %{HTTP_USER_AGENT} Yeti

Re: WebSpider List for Who's Online [Re: ScriptKeeper] #201959
11/24/07 06:42 PM
11/24/07 06:42 PM
Joined: Jan 2004
Posts: 2,650
driv Offline

driv  Offline


Joined: Jan 2004
Posts: 2,650
Sorry to be thicker than usual mate (perhaps I missed a thread or two) - what's the story with the second .htaccess banned list?

Are they known spammers - or the like?


Using version :: 7.6.0
Re: WebSpider List for Who's Online [Re: driv] #201961
11/24/07 07:33 PM
11/24/07 07:33 PM
Joined: Dec 2006
Posts: 1,420
UK
ScriptKeeper Offline

veteran
ScriptKeeper  Offline

veteran
Joined: Dec 2006
Posts: 1,420
UK
It's just a personal list that I use. Some are spam bots, e-mail bots, etc.

Some, like Yahoo! Slurp China, sogou and Yeti are for Asian search engines. I get a lot of spam posted on my forums from software developers in Japan, China, Beijing, etc. trying to get free advertisement for their products so I block anything that comes snooping round my site from these areas.

Re: WebSpider List for Who's Online [Re: ScriptKeeper] #201966
11/24/07 08:11 PM
11/24/07 08:11 PM
Joined: Jul 2006
Posts: 4,704
Liverpool : England : UK
Mark S Offline
Mark S  Offline

Joined: Jul 2006
Posts: 4,704
Liverpool : England : UK
good stuff smile cheers


Version v7.5.8
People who inspire me Gizmo ID242 SD
Its been a long road. . . .to be waiting
Re: WebSpider List for Who's Online [Re: Mark S] #201970
11/24/07 08:19 PM
11/24/07 08:19 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Yeh, some bad bots won't respect the robots.txt standard; so you have to tell apache where to put 'em wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #201971
11/24/07 08:20 PM
11/24/07 08:20 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
BTW SK, you included your rewrite conditions, what's the rule?


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #201986
11/25/07 12:08 PM
11/25/07 12:08 PM
Joined: Dec 2006
Posts: 1,420
UK
ScriptKeeper Offline

veteran
ScriptKeeper  Offline

veteran
Joined: Dec 2006
Posts: 1,420
UK
RewriteRule .* - [F,L]

Re: WebSpider List for Who's Online [Re: Gizmo] #202660
12/05/07 03:28 PM
12/05/07 03:28 PM
Joined: Jan 2007
Posts: 57
Belgium
F
femistyle Offline
journeyman
femistyle  Offline
journeyman
F
Joined: Jan 2007
Posts: 57
Belgium
Since I implemented that list, I don't have any guests anymore but only logged in users and spiders :-(
Don't think that's completly right.

Nans

Re: WebSpider List for Who's Online [Re: femistyle] #202667
12/05/07 04:20 PM
12/05/07 04:20 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Well, if they match a useragent string, it's a bot... Standard UA's for browsers are completely different than the bot ones wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #202736
12/06/07 08:57 AM
12/06/07 08:57 AM
Joined: Dec 2006
Posts: 1,420
UK
ScriptKeeper Offline

veteran
ScriptKeeper  Offline

veteran
Joined: Dec 2006
Posts: 1,420
UK
No you've definately done something there as some of your Search Spiders are blank.

Make sure that your very last bot in the list has no line break after it. Leave the curser flashing at the end of the line.

e.g. if | is the curser do this:

WISEnut=ZyBorg|

not:

WISEnut=ZyBorg
|

Re: WebSpider List for Who's Online [Re: ScriptKeeper] #203195
12/13/07 10:09 AM
12/13/07 10:09 AM
Joined: Dec 2006
Posts: 1,420
UK
ScriptKeeper Offline

veteran
ScriptKeeper  Offline

veteran
Joined: Dec 2006
Posts: 1,420
UK
New kid on the block: Pagebull Visual Search Engine.

Well, I say new, it was launched on 28th Nov 2006 but it's just started visiting my forums so it's new to me.

Spider Details: Pagebull=Pagebull

Note: On their website is says the spider is pagebullbot but that isn't recognised on the UBB.

Re: WebSpider List for Who's Online [Re: Thorsten] #237049
05/26/10 04:51 PM
05/26/10 04:51 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
In another topic this discussion came up for a updated spider/agent list.

Currently there is not a way for us users to update a list on this site.
And the stock code has not changed for quite some time.

But it would be a nice to have or at least the moderators have a current updated list posted.
I know it is a revolving door keeping up with them but....
My list as of today is:
Quote:
Accelovation=heritrix
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
AllTheWeb=crawler@fast
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind
80legs=008


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Ruben] #237056
05/26/10 08:47 PM
05/26/10 08:47 PM
Joined: Mar 2007
Posts: 532
Massachusetts
SteveS Offline

addict
SteveS  Offline

addict
Joined: Mar 2007
Posts: 532
Massachusetts
Ruben, thank you for the list. I haven't paid much attention to spiders vs. anonymous viewers. I just updated mine.


Steve

UBB.classic from 2000-2003
UBB.threads from 2003-present!
Re: WebSpider List for Who's Online [Re: SteveS] #237063
05/27/10 12:11 PM
05/27/10 12:11 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
In another post about this Harold found Gizmos post. Which I completely forgot about.
http://www.ubbcentral.com/forums/ubbthreads.php/topics/228585/Spider_List.html#Post228585
But nobody has asked him to update the list with there additions.
since like 08/29/2009


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Ruben] #237071
05/27/10 04:01 PM
05/27/10 04:01 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Updated my thread wink...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #237072
05/27/10 04:02 PM
05/27/10 04:02 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
Cool dude.


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Gizmo] #237073
05/27/10 04:12 PM
05/27/10 04:12 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
BTW, I probably started with Gizmos list to begin with way back when so there probably is not a lot of difference between the two of them. After today there is no difference at all.


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Ruben] #237121
06/01/10 03:41 PM
06/01/10 03:41 PM
Joined: Aug 2004
Posts: 466
Conrad Offline
addict
Conrad  Offline
addict
Joined: Aug 2004
Posts: 466
Do you guys add the entire list in the CP? Does this slow down the board in any way?

Re: WebSpider List for Who's Online [Re: Conrad] #237122
06/01/10 03:43 PM
06/01/10 03:43 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
I think the list would need to be many hundreds of lines long to notice any lag and then it would only be on the who is online screen.
It is just a big hassle to keep it updated


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Thorsten] #254101
11/25/13 04:45 PM
11/25/13 04:45 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
For those using my spider list from the FAQ; I'm currently updating the UA names to be the domain names that the entries belong to (where available and applicable) as it'd be a lot more helpful for reporting crawling issues than having users have to scour the internet for a contact.

I'm hoping that this will be done later today, but if not it'll definitely be done by Thanksgiving day.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Thorsten] #254674
01/20/14 12:36 PM
01/20/14 12:36 PM
Joined: Aug 2004
Posts: 466
Conrad Offline
addict
Conrad  Offline
addict
Joined: Aug 2004
Posts: 466
Is the best way forward to copy/use the entire list on one's site? Nothing wrong with having a long and full list, right?

Re: WebSpider List for Who's Online [Re: Conrad] #255094
03/21/14 11:48 AM
03/21/14 11:48 AM
Joined: Aug 2004
Posts: 466
Conrad Offline
addict
Conrad  Offline
addict
Joined: Aug 2004
Posts: 466
Is the best thing to do to just cut&paste the following list into our forum setting for spiders: http://www.ubbwiki.com/article/view/3/spider-listing.html

Is it better to have a long list or a short list of the main ones? Do these search engine spiders slow things down somewhat, or is the effect negligible?

Re: WebSpider List for Who's Online [Re: Thorsten] #255098
03/21/14 01:57 PM
03/21/14 01:57 PM
Joined: Dec 2003
Posts: 5,827
Lutz,FL
Ruben Offline

Ruben  Offline


Joined: Dec 2003
Posts: 5,827
Lutz,FL
without the list of agents.
all visiting spiders will appear as a guest.
The list just identifies the ones that are visiting your site and segregates them from the guest list to the spider list.
It does not add spiders to your site.

Yes you can copy/paste.
You will probably find a majority of them never visit anyway.

It is your choice if you want a complete list or a partial list.


Blue Man Group


There is no such thing as stupid questions. Just stupid answers
Re: WebSpider List for Who's Online [Re: Thorsten] #255100
03/22/14 04:31 AM
03/22/14 04:31 AM
Joined: Aug 2004
Posts: 466
Conrad Offline
addict
Conrad  Offline
addict
Joined: Aug 2004
Posts: 466
Will adding a long list slow things down though, kind of like adding a long censor list will?

Re: WebSpider List for Who's Online [Re: Thorsten] #255101
03/22/14 05:33 AM
03/22/14 05:33 AM
Joined: Apr 2004
Posts: 1,004
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,004
California
I knew that I posted a more recent, more complete list some place before... and here it is smile
http://www.ubbcentral.com/forums/ubbthreads.php/topics/252157#Post252157 (01-31-2013)

The site that's using it has roughly 1400 visitors on it with in a two hour period all day long. No slow downs except for when someone visits the "Who's Online" page, they get a 1 second delay. But I believe that's mostly because the page being displayed is a mile long, filled with URL links pages which everyone else is browsing.

My best advice, copy/paste the list to your site. Keep it up for a few days. If you notice that your site feels laggy (which it probably will not), replace it with one of the older, less complete lists.

Re: WebSpider List for Who's Online [Re: Gizmo] #255102
03/22/14 05:36 AM
03/22/14 05:36 AM
Joined: Apr 2004
Posts: 1,004
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,004
California
Originally Posted by Gizmo
Updated my thread wink...


It's too bad that no one can contribute updates to that thread. Why was it closed/locked?

Re: WebSpider List for Who's Online [Re: Thorsten] #255107
03/22/14 02:17 PM
03/22/14 02:17 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
It's in the FAQ forum, which is the only official FAQ; Rick wanted it "semi official" so comments on some things can get in the way.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: WebSpider List for Who's Online [Re: Gizmo] #255143
03/26/14 07:49 PM
03/26/14 07:49 PM
Joined: Apr 2004
Posts: 1,004
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,004
California
My list has been updated. It now includes all ROBOTS, not just search spiders.
https://www.ubbcentral.com/forums/ub...e:_List_of_Search_Engine_Agen#Post255142


Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Users Unable to Upload Avatar [Not a Bug]
by M4TT. 12/13/17 08:51 AM
Shout Box Sound Effect
by M4TT. 11/29/17 08:28 PM
Ad island
by TGCsanderson. 11/25/17 06:41 PM
Taking to long to connect to DB
by AstroCat. 11/24/17 12:34 PM
Forum Statistics
Forums36
Topics35,015
Posts190,544
Members12,045
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.1
(Snapshot build 20171106)