Previous Thread
Next Thread
Print Thread
Hop To
Spider List #228585 08/23/2009 7:19 PM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list. Please use this thread for comments.

Notice - This listing is no longer being maintained here; instead please see our official listing at UBBDev here and on the UBB.Wiki.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:
Code
80legs=008
A6-Indexer=A6-Indexer
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AllTheWeb=crawler@fast
AllTheWeb=FAST-WebCrawler
AllTheWeb=FAST
AltaVista=Mercator
Altavista=Scooter
Amazon.com=AMZNKAssocBot
Anzwers=AnzwersCrawl
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Ask.com=Teoma
Ask.com=AskJeeves
Atomz=Atomz
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho.com=boitho.com
Cityreview=Cityreview Robot
Crawler.de=Crawler
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
EntireWeb.com=Speedy Spider
Exalead=Exabot
Excite=Architext spider
Excite=ArchitextSpider
Facebook=facebookexternalhit
Factbites=Factbot
FreeWebMonitoring=FreeWebMonitoring SiteChecker
Gigablast=GigaBot
Google Adsense=Mediapartners-Google
Google Application Engine=AppEngine-Google
Google Images=Googlebot-Image
Google Mobile=Googlebot-Mobile
Google Sitemaps=Google-Sitemaps/1.0
Google=Googlebot
InfoSeek=InfoSeek sidewinder
InfoSeek=UltraSeek
Inktomi=Slurp
Kalooga=Kalooga
libwww=libwww-perl
LinkDex=linkdexbot
LookSmart=FurlBot
Looksmart=MantraAgent
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
Microsoft Research=MSRBOT
MJ12bot=MJ12bot
MSN Media=msnbot-media
MSN Photo Search=PSbot
MSN=msnbot
Naver.com=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight.com=Gulliver
noXtrum=noxtrumbot
Nutch Bot=Nutch12
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panopta Monitoring=checks.panopta.com
Panscient=panscient.com
Picsearch=psbot
Pintrest=Pinterest
PrivacyFinder=PrivacyFinder
Proximic=proximic
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Hippo=Fluffy the Spider
Search Me=Charlotte
Seznam=SeznamBot
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snap=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TAMU.edu=IRLbot
Teoma=TeomaAgent
TinEye=TinEye
TMCrawler=TMCrawler
Twiceler=Twiceler-0.9
Voila=Voila
Voila=Voila.fr
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
Web Data Centre=WebDataCentreBot
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
WISEnut=ZyBorg
Worio=woriobot
YaCy=yacybot
Yahoo Blogs=Yahoo-blogs/v3.9
Yahoo China=Yahoo! Slurp China
Yahoo Germany=Yahoo! DE Slurp
Yahoo MM=Yahoo-MMcrawler
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Yandex=Yandex
Yanga=Yanga
Yisou=YisouSpider
YodaoBot=YodaoBot
Yoono=yoofind


Last Update: 2011.11.24

(Moderator Note: Please do not update this listing; the "Master List" is considered to be that on the Wiki and as such is taken from there for updates before being run through an alphabetizer and a duplicate checker; as such, changes to this post will be ignored and replaced after updated from the wiki.)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #237070 05/27/2010 4:01 PM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
Updated the above list with additions from this post.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242718 04/20/2011 10:30 PM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
Added Bing to the list.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242751 04/21/2011 5:23 PM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
Someone suggested (through a report) to add Facebook; it's not really a spider so I don't really want to add it... But for anyone who wants it:
Code
Facebook=facebookexternalhit


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #243908 06/14/2011 1:54 PM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
Added:
Code
Magpie=magpie-crawler


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #254093 11/24/2013 3:49 AM
Joined: Jun 2006
Posts: 15,848
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,848
Added several new entries, removed duplicate entries, updated names for several entries, alphabetized listing.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Forum Search
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
Search feature encountering an Error message
by jorb - 11/20/2019 12:06 AM
Followed List v7.7.2 Question
by Ruben - 11/12/2019 12:22 PM
UBB Dev
by JAISP - 11/03/2019 11:01 AM
Exceeded Number of attachments
by rbrtgrmn - 11/02/2019 9:57 PM
Forum New User Registration
by kf6zpl - 10/25/2019 10:45 AM
Who's Online Now
3 registered members (isaac, JAISP, jorb), 54 guests, and 346 spiders.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Amusing Terain Scenics
Amusing Terain Scenics
by isaac, August 19
Sky places
Sky places
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 7.7.4
(Snapshot build 20191023)