This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.
I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list. Please use
this thread for comments.
Notice - This listing is no longer being maintained here; instead please see our official listing at UBBDev
here and on the
UBB.Wiki.
The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents
The List:
80legs=008
A6-Indexer=A6-Indexer
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AllTheWeb=crawler@fast
AllTheWeb=FAST-WebCrawler
AllTheWeb=FAST
AltaVista=Mercator
Altavista=Scooter
Amazon.com=AMZNKAssocBot
Anzwers=AnzwersCrawl
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Ask.com=Teoma
Ask.com=AskJeeves
Atomz=Atomz
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho.com=boitho.com
Cityreview=Cityreview Robot
Crawler.de=Crawler
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
EntireWeb.com=Speedy Spider
Exalead=Exabot
Excite=Architext spider
Excite=ArchitextSpider
Facebook=facebookexternalhit
Factbites=Factbot
FreeWebMonitoring=FreeWebMonitoring SiteChecker
Gigablast=GigaBot
Google Adsense=Mediapartners-Google
Google Application Engine=AppEngine-Google
Google Images=Googlebot-Image
Google Mobile=Googlebot-Mobile
Google Sitemaps=Google-Sitemaps/1.0
Google=Googlebot
InfoSeek=InfoSeek sidewinder
InfoSeek=UltraSeek
Inktomi=Slurp
Kalooga=Kalooga
libwww=libwww-perl
LinkDex=linkdexbot
LookSmart=FurlBot
Looksmart=MantraAgent
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
Microsoft Research=MSRBOT
MJ12bot=MJ12bot
MSN Media=msnbot-media
MSN Photo Search=PSbot
MSN=msnbot
Naver.com=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight.com=Gulliver
noXtrum=noxtrumbot
Nutch Bot=Nutch12
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panopta Monitoring=checks.panopta.com
Panscient=panscient.com
Picsearch=psbot
Pintrest=Pinterest
PrivacyFinder=PrivacyFinder
Proximic=proximic
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Hippo=Fluffy the Spider
Search Me=Charlotte
Seznam=SeznamBot
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snap=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TAMU.edu=IRLbot
Teoma=TeomaAgent
TinEye=TinEye
TMCrawler=TMCrawler
Twiceler=Twiceler-0.9
Voila=Voila
Voila=Voila.fr
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
Web Data Centre=WebDataCentreBot
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
WISEnut=ZyBorg
Worio=woriobot
YaCy=yacybot
Yahoo Blogs=Yahoo-blogs/v3.9
Yahoo China=Yahoo! Slurp China
Yahoo Germany=Yahoo! DE Slurp
Yahoo MM=Yahoo-MMcrawler
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Yandex=Yandex
Yanga=Yanga
Yisou=YisouSpider
YodaoBot=YodaoBot
Yoono=yoofind
Last Update: 2011.11.24
(Moderator Note: Please do not update this listing; the "Master List" is considered to be that on the Wiki and as such is taken from there for updates before being run through an alphabetizer and a duplicate checker; as such, changes to this post will be ignored and replaced after updated from the wiki.)
