You are not logged in. [Log In] UBB.threads PHP Forum Software Community » Forums » Managing & Customizing your Board » Admin Zone » WebSpider List for Who's Online
Register User    Portal Page     Forum List        Calendar     Active Topics     Search     FAQ
Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online
1 registered (driv), 35 Guests and 13 Spiders online.
Key: Admin, Global Mod, Mod
Featured Member
Registered: 11/22/06
Posts: 163
Top Posters (30 Days)
Ruben 51
DennyP 24
Gizmo 24
Dunny 15
SteveS 14
AllenAyres 12
dbremer 10
SD 10
drkknght00 9
doug 8
Latest Photos
OK Corral Shoot Out
Testing
Basildon Train Station
Basildon Town Centre looking from the rounderbout
Basildon Town Square
Page 1 of 3 1 2 3 >
Topic Options
#196886 - 09/09/07 03:05 PM WebSpider List for Who's Online
Thorsten Offline
newbie 		Registered: 08/01/06
Posts: 39
Can somebody post his/hers webspiderlist from the admin panel?
Top
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
#196887 - 09/09/07 03:46 PM Re: WebSpider List for Who's Online [Re: Thorsten]
driv Online censored

Pooh-Bah 		Registered: 01/10/04
Posts: 2377
Have a look here for starters...

Spider Link
_________________________
Using version :: 7.5.6
Top
#196907 - 09/09/07 10:30 PM Re: WebSpider List for Who's Online [Re: driv]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
 Code:
Alexa=ia_archiver
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
AllTheWeb=crawler@fast
Excite=ArchitextSpider
Gigabot=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Inktomi=Slurp
MSN=MSNBOT
Sogou=sogou web spider
Entireweb=Speedy Spider
Voila=Voila.fr
Ask.com=Ask Jeeves
Teoma=TeomaAgent
Wisenut=Zyborg
NorthernLight.com=Gulliver
Excite=Architext spider
AltaVista=Mercator
Crawler.de=Crawler
Infoseek=InfoSeek sidewinder
Lycos=Lycos_Spider_(T-Rex)
Search Hippo=Fluffy the Spider
Infoseek=Ultraseek
Looksmart=MantraAgent
Webcrawler.com=WebCrawler
Twiceler=Twiceler-0.9
Naver.com=Yeti/
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#196918 - 09/10/07 01:22 AM Re: WebSpider List for Who's Online [Re: Gizmo]
ntdoc Offline
Registered: 11/09/06
Posts: 3384
The last one. Yeti/ is that correct?
Top
#196921 - 09/10/07 02:25 AM Re: WebSpider List for Who's Online [Re: ntdoc]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
Yes, its agent is Yeti/0.1; I wanted to ensure it doesn't trigger on anything with a similar name (and didn't want to have to include the version string), so I left it as Yeti/
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#201956 - 11/24/07 05:34 PM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
My updated spider list:
 Quote:
Alexa=ia_archiver
Altavista=Scooter
Anzwers=AnzwersCrawl
Ask=Teoma
Atomz=Atomz
Boitho=boitho.com
Entireweb=Speedy Spider
Exalead=Exabot
Excite=ArchitextSpider
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=GigaBot
Google=Googlebot
Google-Image=Googlebot-Image
Yahoo!=Yahoo! Slurp
Infoseek=Ultraseek
Inktomi=Slurp
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Microsoft Research=MSRBOT
MSN=MSNBOT
NetSeer=Teemer
noXtrum=noxtrumbot
Searchme=Charlotte
Seznam=SeznamBot
Snap=Snapbot
Voila=VoilaBot
Walhello=appie
WISEnut=ZyBorg

.htaccess blocked list:
 Quote:
RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX [OR]
RewriteCond %{HTTP_USER_AGENT} Yeti
Top
#201959 - 11/24/07 06:42 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
driv Online censored

Pooh-Bah 		Registered: 01/10/04
Posts: 2377
Sorry to be thicker than usual mate (perhaps I missed a thread or two) - what's the story with the second .htaccess banned list?

Are they known spammers - or the like?
_________________________
Using version :: 7.5.6
Top
#201961 - 11/24/07 07:33 PM Re: WebSpider List for Who's Online [Re: driv]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
It's just a personal list that I use. Some are spam bots, e-mail bots, etc.

Some, like Yahoo! Slurp China, sogou and Yeti are for Asian search engines. I get a lot of spam posted on my forums from software developers in Japan, China, Beijing, etc. trying to get free advertisement for their products so I block anything that comes snooping round my site from these areas.
Top
#201966 - 11/24/07 08:11 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
Mark S Offline
Carpal Tunnel 		Registered: 07/04/06
Posts: 4447
Loc: Liverpool : England : UK
good stuff \:\) cheers
_________________________
Version v7.5.6 smile smile < Threads satisfaction status
People who inspire me Rick Gizmo Ian David jgeoff ntdoc
Oooo i hear 8 is coming? just after 7 my friend.
Top
#201970 - 11/24/07 08:19 PM Re: WebSpider List for Who's Online [Re: Mark S]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
Yeh, some bad bots won't respect the robots.txt standard; so you have to tell apache where to put 'em ;\)
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#201971 - 11/24/07 08:20 PM Re: WebSpider List for Who's Online [Re: Gizmo]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
BTW SK, you included your rewrite conditions, what's the rule?
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#201986 - 11/25/07 12:08 PM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
RewriteRule .* - [F,L]
Top
#202660 - 12/05/07 03:28 PM Re: WebSpider List for Who's Online [Re: Gizmo]
femistyle Offline
journeyman 		Registered: 01/08/07
Posts: 57
Loc: Belgium
Since I implemented that list, I don't have any guests anymore but only logged in users and spiders :-(
Don't think that's completly right.

Nans
Top
#202667 - 12/05/07 04:20 PM Re: WebSpider List for Who's Online [Re: femistyle]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
Well, if they match a useragent string, it's a bot... Standard UA's for browsers are completely different than the bot ones ;\)
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#202736 - 12/06/07 08:57 AM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
No you've definately done something there as some of your Search Spiders are blank.

Make sure that your very last bot in the list has no line break after it. Leave the curser flashing at the end of the line.

e.g. if | is the curser do this:

WISEnut=ZyBorg|

not:

WISEnut=ZyBorg
|
Top
#203195 - 12/13/07 10:09 AM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
New kid on the block: Pagebull Visual Search Engine.

Well, I say new, it was launched on 28th Nov 2006 but it's just started visiting my forums so it's new to me.

Spider Details: Pagebull=Pagebull

Note: On their website is says the spider is pagebullbot but that isn't recognised on the UBB.
Top
#237049 - 05/26/10 05:51 PM Re: WebSpider List for Who's Online [Re: Thorsten]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
In another topic this discussion came up for a updated spider/agent list.

Currently there is not a way for us users to update a list on this site.
And the stock code has not changed for quite some time.

But it would be a nice to have or at least the moderators have a current updated list posted.
I know it is a revolving door keeping up with them but....
My list as of today is:
Quote:
Accelovation=heritrix
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
AllTheWeb=crawler@fast
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind
80legs=008
_________________________
Blue Man Group
Top
#237056 - 05/26/10 09:47 PM Re: WebSpider List for Who's Online [Re: Ruben]
SteveS Online content

addict 		Registered: 03/22/07
Posts: 416
Loc: Massachusetts
Ruben, thank you for the list. I haven't paid much attention to spiders vs. anonymous viewers. I just updated mine.
_________________________
Steve
crownvic.net
UBB.classic from 2000-2003
UBB.threads from 2003-present!
Top
#237063 - 05/27/10 01:11 PM Re: WebSpider List for Who's Online [Re: SteveS]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
In another post about this Harold found Gizmos post. Which I completely forgot about.
http://www.ubbcentral.com/forums/ubbthreads.php/topics/228585/Spider_List.html#Post228585
But nobody has asked him to update the list with there additions.
since like 08/29/2009
_________________________
Blue Man Group
Top
#237071 - 05/27/10 05:01 PM Re: WebSpider List for Who's Online [Re: Ruben]
Gizmo Offline

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
Updated my thread wink...
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#237072 - 05/27/10 05:02 PM Re: WebSpider List for Who's Online [Re: Gizmo]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
Cool dude.
_________________________
Blue Man Group
Top
#237073 - 05/27/10 05:12 PM Re: WebSpider List for Who's Online [Re: Gizmo]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
BTW, I probably started with Gizmos list to begin with way back when so there probably is not a lot of difference between the two of them. After today there is no difference at all.
_________________________
Blue Man Group
Top
#237121 - 06/01/10 04:41 PM Re: WebSpider List for Who's Online [Re: Ruben]
Conrad Offline
addict 		Registered: 08/04/04
Posts: 425
Do you guys add the entire list in the CP? Does this slow down the board in any way?
Top
#237122 - 06/01/10 04:43 PM Re: WebSpider List for Who's Online [Re: Conrad]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
I think the list would need to be many hundreds of lines long to notice any lag and then it would only be on the who is online screen.
It is just a big hassle to keep it updated
_________________________
Blue Man Group
Top
Page 1 of 3 1 2 3 >



Shout Box

Today's Birthdays
No Birthdays
Recent Topics
Due Date Calculator-Calculate When Your Baby is Due
by StewartMyduedate
12:54 AM
Temporary Password email not being received
by
05/24/12 10:02 PM
Ability to "like" individual posts (not Facebook "likes)
by doug
05/23/12 09:03 AM
Island Permissions
by ThreadsUser
05/22/12 03:03 PM
streaming video
by prkrgrp
05/20/12 07:02 PM
Forum Stats
10491 Members
36 Forums
33842 Topics
181709 Posts

Max Online: 978 @ 06/24/07 11:19 PM
Random Image
Board Rules · Mark all read
Contact Us · UBBCentral · Top