Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
Who's Online
7 registered (David Dreezer, Djuma, dslam, Harold, packlite, Pink Jazz, 1 invisible), 28 Guests and 23 Spiders online.
Key: Admin, Global Mod, Mod
Featured Member
Registered: 11/02/08
Posts: 2
Top Posters (30 Days)
Ruben Rocha 179
Gizmo 112
Rick 109
Thelockman 80
driv 43
AllenAyres 35
ntdoc 28
Ian 24
Sirdude 23
ScriptKeeper 20
Latest Photos
My Home System
test photo gallery
Bernese Mountain Dogs
My Daimler
Dorado and shark
Page 1 of 2 1 2 >
Topic Options
Rate This Topic
#196886 - 09/09/07 12:05 PM WebSpider List for Who's Online
Thorsten Offline
newbie

Registered: 08/01/06
Posts: 37
Can somebody post his/hers webspiderlist from the admin panel?

Top
#196887 - 09/09/07 12:46 PM Re: WebSpider List for Who's Online [Re: Thorsten]
driv Offline
Pooh-Bah
****

Registered: 01/10/04
Posts: 1703
Loc: Essex, UK
Have a look here for starters...

Spider Link
_________________________
Oi Oi Saveloy!
(Courtesy of Sd - well known Anglophile...!?!)
My True star rating wink

Top
#196907 - 09/09/07 07:30 PM Re: WebSpider List for Who's Online [Re: driv]
Gizmo Moderator Offline

***

Registered: 06/04/06
Posts: 12006
Loc: Portland, OR; USA
 Code:
Alexa=ia_archiver
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
AllTheWeb=crawler@fast
Excite=ArchitextSpider
Gigabot=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Inktomi=Slurp
MSN=MSNBOT
Sogou=sogou web spider
Entireweb=Speedy Spider
Voila=Voila.fr
Ask.com=Ask Jeeves
Teoma=TeomaAgent
Wisenut=Zyborg
NorthernLight.com=Gulliver
Excite=Architext spider
AltaVista=Mercator
Crawler.de=Crawler
Infoseek=InfoSeek sidewinder
Lycos=Lycos_Spider_(T-Rex)
Search Hippo=Fluffy the Spider
Infoseek=Ultraseek
Looksmart=MantraAgent
Webcrawler.com=WebCrawler
Twiceler=Twiceler-0.9
Naver.com=Yeti/
_________________________
UGN Security, Elite Web Gamers & VNC Web Design Owner
Longtime UBB Supporter, UBB7 Beta Tester & Resident Post-A-Holic

Top
#196918 - 09/09/07 10:22 PM Re: WebSpider List for Who's Online [Re: Gizmo]
ntdoc Offline

***

Registered: 11/08/06
Posts: 3202
The last one. Yeti/ is that correct?

Top
#196921 - 09/09/07 11:25 PM Re: WebSpider List for Who's Online [Re: ntdoc]
Gizmo Moderator Offline

***

Registered: 06/04/06
Posts: 12006
Loc: Portland, OR; USA
Yes, its agent is Yeti/0.1; I wanted to ensure it doesn't trigger on anything with a similar name (and didn't want to have to include the version string), so I left it as Yeti/
_________________________
UGN Security, Elite Web Gamers & VNC Web Design Owner
Longtime UBB Supporter, UBB7 Beta Tester & Resident Post-A-Holic

Top
#201956 - 11/24/07 02:34 PM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline
old hand
***

Registered: 12/09/06
Posts: 1039
Loc: UK
My updated spider list:
 Quote:
Alexa=ia_archiver
Altavista=Scooter
Anzwers=AnzwersCrawl
Ask=Teoma
Atomz=Atomz
Boitho=boitho.com
Entireweb=Speedy Spider
Exalead=Exabot
Excite=ArchitextSpider
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=GigaBot
Google=Googlebot
Google-Image=Googlebot-Image
Yahoo!=Yahoo! Slurp
Infoseek=Ultraseek
Inktomi=Slurp
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Microsoft Research=MSRBOT
MSN=MSNBOT
NetSeer=Teemer
noXtrum=noxtrumbot
Searchme=Charlotte
Seznam=SeznamBot
Snap=Snapbot
Voila=VoilaBot
Walhello=appie
WISEnut=ZyBorg

.htaccess blocked list:
 Quote:
RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX [OR]
RewriteCond %{HTTP_USER_AGENT} Yeti

Top
#201959 - 11/24/07 03:42 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
driv Offline
Pooh-Bah
****

Registered: 01/10/04
Posts: 1703
Loc: Essex, UK
Sorry to be thicker than usual mate (perhaps I missed a thread or two) - what's the story with the second .htaccess banned list?

Are they known spammers - or the like?
_________________________
Oi Oi Saveloy!
(Courtesy of Sd - well known Anglophile...!?!)
My True star rating wink

Top
#201961 - 11/24/07 04:33 PM Re: WebSpider List for Who's Online [Re: driv]
ScriptKeeper Offline
old hand
***

Registered: 12/09/06
Posts: 1039
Loc: UK
It's just a personal list that I use. Some are spam bots, e-mail bots, etc.

Some, like Yahoo! Slurp China, sogou and Yeti are for Asian search engines. I get a lot of spam posted on my forums from software developers in Japan, China, Beijing, etc. trying to get free advertisement for their products so I block anything that comes snooping round my site from these areas.

Top
#201966 - 11/24/07 05:11 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
Mark S Offline
Carpal Tunnel
***

Registered: 07/04/06
Posts: 4044
Loc: Liverpool : England : UK
good stuff \:\) cheers
_________________________
Version v7.2.2 smile smile < Threads satisfaction status
People who inspire me Rick Gizmo Ian David jgeoff ntdoc
To answer the question you must first give a question.

Top
#201970 - 11/24/07 05:19 PM Re: WebSpider List for Who's Online [Re: Mark S]
Gizmo Moderator Offline

***

Registered: 06/04/06
Posts: 12006
Loc: Portland, OR; USA
Yeh, some bad bots won't respect the robots.txt standard; so you have to tell apache where to put 'em ;\)
_________________________
UGN Security, Elite Web Gamers & VNC Web Design Owner
Longtime UBB Supporter, UBB7 Beta Tester & Resident Post-A-Holic

Top
#201971 - 11/24/07 05:20 PM Re: WebSpider List for Who's Online [Re: Gizmo]
Gizmo Moderator Offline

***

Registered: 06/04/06
Posts: 12006
Loc: Portland, OR; USA
BTW SK, you included your rewrite conditions, what's the rule?
_________________________
UGN Security, Elite Web Gamers & VNC Web Design Owner
Longtime UBB Supporter, UBB7 Beta Tester & Resident Post-A-Holic

Top
#201986 - 11/25/07 09:08 AM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline
old hand
***

Registered: 12/09/06
Posts: 1039
Loc: UK
RewriteRule .* - [F,L]

Top
#202660 - 12/05/07 12:28 PM Re: WebSpider List for Who's Online [Re: Gizmo]
femistyle Offline
newbie

Registered: 01/08/07
Posts: 48
Loc: Belgium
Since I implemented that list, I don't have any guests anymore but only logged in users and spiders :-(
Don't think that's completly right.

Nans

Top
#202667 - 12/05/07 01:20 PM Re: WebSpider List for Who's Online [Re: femistyle]
Gizmo Moderator Offline

***

Registered: 06/04/06
Posts: 12006
Loc: Portland, OR; USA
Well, if they match a useragent string, it's a bot... Standard UA's for browsers are completely different than the bot ones ;\)
_________________________
UGN Security, Elite Web Gamers & VNC Web Design Owner
Longtime UBB Supporter, UBB7 Beta Tester & Resident Post-A-Holic

Top
#202736 - 12/06/07 05:57 AM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline
old hand
***

Registered: 12/09/06
Posts: 1039
Loc: UK
No you've definately done something there as some of your Search Spiders are blank.

Make sure that your very last bot in the list has no line break after it. Leave the curser flashing at the end of the line.

e.g. if | is the curser do this:

WISEnut=ZyBorg|

not:

WISEnut=ZyBorg
|

Top
#203195 - 12/13/07 07:09 AM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
ScriptKeeper Offline
old hand
***

Registered: 12/09/06
Posts: 1039
Loc: UK
New kid on the block: Pagebull Visual Search Engine.

Well, I say new, it was launched on 28th Nov 2006 but it's just started visiting my forums so it's new to me.

Spider Details: Pagebull=Pagebull

Note: On their website is says the spider is pagebullbot but that isn't recognised on the UBB.

Top
Page 1 of 2 1 2 >


Moderator:  Gizmo 
Shout Box