You are not logged in. [Log In] UBB.threads PHP Forum Software Community » Forums » Managing & Customizing your Board » Admin Zone » WebSpider List for Who's Online
Register User    Portal Page     Forum List        Calendar     Active Topics     Search     FAQ
Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online
0 registered (), 30 Guests and 8 Spiders online.
Key: Admin, Global Mod, Mod
Featured Member
Registered: 12/20/11
Posts: 35
Top Posters (30 Days)
Ruben 46
Bert 26
Gizmo 18
Rob Provencher 10
Rimex 9
SD 7
sw55 6
Eugene 5
Matthias1976 4
BellaOnline 3
Latest Photos
Uhm...
Mayan End of World
Gas Station Disco Video Shoot
Test Pictures
Audrey Kate
Page 1 of 3 1 2 3 >
Topic Options
#196886 - 09/09/07 02:05 PM WebSpider List for Who's Online
Thorsten Offline
newbie 		Registered: 08/01/06
Posts: 39
Can somebody post his/hers webspiderlist from the admin panel?
Top
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
#196887 - 09/09/07 02:46 PM Re: WebSpider List for Who's Online [Re: Thorsten]
driv Offline

Carpal Tunnel 		Registered: 01/10/04
Posts: 2543
Have a look here for starters...

Spider Link
_________________________
Using version :: 7.5.7 ...sans SFS at the mo' crazy
Top
#196907 - 09/09/07 09:30 PM Re: WebSpider List for Who's Online [Re: driv]
Gizmo Offline

Registered: 06/05/06
Posts: 15455
Loc: Portland, OR; USA
 Code:
Alexa=ia_archiver
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
AllTheWeb=crawler@fast
Excite=ArchitextSpider
Gigabot=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Inktomi=Slurp
MSN=MSNBOT
Sogou=sogou web spider
Entireweb=Speedy Spider
Voila=Voila.fr
Ask.com=Ask Jeeves
Teoma=TeomaAgent
Wisenut=Zyborg
NorthernLight.com=Gulliver
Excite=Architext spider
AltaVista=Mercator
Crawler.de=Crawler
Infoseek=InfoSeek sidewinder
Lycos=Lycos_Spider_(T-Rex)
Search Hippo=Fluffy the Spider
Infoseek=Ultraseek
Looksmart=MantraAgent
Webcrawler.com=WebCrawler
Twiceler=Twiceler-0.9
Naver.com=Yeti/
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime Supporter, Beta Tester & Resident Post-A-Holic.
Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Top
#196918 - 09/10/07 12:22 AM Re: WebSpider List for Who's Online [Re: Gizmo]
ntdoc Offline
Registered: 11/08/06
Posts: 3386
The last one. Yeti/ is that correct?
Top
#196921 - 09/10/07 01:25 AM Re: WebSpider List for Who's Online [Re: ntdoc]
Gizmo Offline

Registered: 06/05/06
Posts: 15455
Loc: Portland, OR; USA
Yes, its agent is Yeti/0.1; I wanted to ensure it doesn't trigger on anything with a similar name (and didn't want to have to include the version string), so I left it as Yeti/
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime Supporter, Beta Tester & Resident Post-A-Holic.
Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Top
#201956 - 11/24/07 04:34 PM Re: WebSpider List for Who's Online [Re: Gizmo]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
My updated spider list:
 Quote:
Alexa=ia_archiver
Altavista=Scooter
Anzwers=AnzwersCrawl
Ask=Teoma
Atomz=Atomz
Boitho=boitho.com
Entireweb=Speedy Spider
Exalead=Exabot
Excite=ArchitextSpider
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=GigaBot
Google=Googlebot
Google-Image=Googlebot-Image
Yahoo!=Yahoo! Slurp
Infoseek=Ultraseek
Inktomi=Slurp
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Microsoft Research=MSRBOT
MSN=MSNBOT
NetSeer=Teemer
noXtrum=noxtrumbot
Searchme=Charlotte
Seznam=SeznamBot
Snap=Snapbot
Voila=VoilaBot
Walhello=appie
WISEnut=ZyBorg

.htaccess blocked list:
 Quote:
RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX [OR]
RewriteCond %{HTTP_USER_AGENT} Yeti
Top
#201959 - 11/24/07 05:42 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
driv Offline

Carpal Tunnel 		Registered: 01/10/04
Posts: 2543
Sorry to be thicker than usual mate (perhaps I missed a thread or two) - what's the story with the second .htaccess banned list?

Are they known spammers - or the like?
_________________________
Using version :: 7.5.7 ...sans SFS at the mo' crazy
Top
#201961 - 11/24/07 06:33 PM Re: WebSpider List for Who's Online [Re: driv]
ScriptKeeper Offline

veteran 		Registered: 12/09/06
Posts: 1420
Loc: UK
It's just a personal list that I use. Some are spam bots, e-mail bots, etc.

Some, like Yahoo! Slurp China, sogou and Yeti are for Asian search engines. I get a lot of spam posted on my forums from software developers in Japan, China, Beijing, etc. trying to get free advertisement for their products so I block anything that comes snooping round my site from these areas.
Top
#201966 - 11/24/07 07:11 PM Re: WebSpider List for Who's Online [Re: ScriptKeeper]
Mark S Offline
Carpal Tunnel 		Registered: 07/04/06
Posts: 4480
Loc: Liverpool : England : UK
good stuff \:\) cheers
_________________________
Version v7.5.6 smile smile < Threads satisfaction status
People who inspire me Rick Gizmo Ian David jgeoff ntdoc
Oooo i hear 8 is coming? just after 7 my friend.
Top
#201970 - 11/24/07 07:19 PM Re: WebSpider List for Who's Online [Re: Mark S]
Gizmo Offline

Registered: 06/05/06
Posts: 15455
Loc: Portland, OR; USA
Yeh, some bad bots won't respect the robots.txt standard; so you have to tell apache where to put 'em ;\)
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime Supporter, Beta Tester & Resident Post-A-Holic.
Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Top
Page 1 of 3 1 2 3 >



Shout Box

Today's Birthdays
No Birthdays
Recent Topics
Time zone setup
by skicomau
05/22/13 12:16 AM
Express hosting.
by Ruben
05/16/13 03:54 PM
Level of detail in new user registration emails
by Mitch P.
05/15/13 10:20 PM
Approving users
by Bert
05/15/13 09:22 PM
Users randomly added to other group
by Bert
05/15/13 09:15 PM
Forum Stats
10967 Members
36 Forums
33958 Topics
183410 Posts

Max Online: 978 @ 06/24/07 10:19 PM
Random Image
Board Rules · Mark all read
Contact Us · UBBCentral · Top