Previous Thread
Next Thread
Print Thread
Hop To
Spider List #228585 08/23/2009 8:19 PM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list. Please use this thread for comments.

Notice - This listing is no longer being maintained here; instead please see our official listing at UBBDev here and on the UBB.Wiki.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:
Code
80legs=008
A6-Indexer=A6-Indexer
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AllTheWeb=crawler@fast
AllTheWeb=FAST-WebCrawler
AllTheWeb=FAST
AltaVista=Mercator
Altavista=Scooter
Amazon.com=AMZNKAssocBot
Anzwers=AnzwersCrawl
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Ask.com=Teoma
Ask.com=AskJeeves
Atomz=Atomz
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho.com=boitho.com
Cityreview=Cityreview Robot
Crawler.de=Crawler
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
EntireWeb.com=Speedy Spider
Exalead=Exabot
Excite=Architext spider
Excite=ArchitextSpider
Facebook=facebookexternalhit
Factbites=Factbot
FreeWebMonitoring=FreeWebMonitoring SiteChecker
Gigablast=GigaBot
Google Adsense=Mediapartners-Google
Google Application Engine=AppEngine-Google
Google Images=Googlebot-Image
Google Mobile=Googlebot-Mobile
Google Sitemaps=Google-Sitemaps/1.0
Google=Googlebot
InfoSeek=InfoSeek sidewinder
InfoSeek=UltraSeek
Inktomi=Slurp
Kalooga=Kalooga
libwww=libwww-perl
LinkDex=linkdexbot
LookSmart=FurlBot
Looksmart=MantraAgent
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
Microsoft Research=MSRBOT
MJ12bot=MJ12bot
MSN Media=msnbot-media
MSN Photo Search=PSbot
MSN=msnbot
Naver.com=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight.com=Gulliver
noXtrum=noxtrumbot
Nutch Bot=Nutch12
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panopta Monitoring=checks.panopta.com
Panscient=panscient.com
Picsearch=psbot
Pintrest=Pinterest
PrivacyFinder=PrivacyFinder
Proximic=proximic
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Hippo=Fluffy the Spider
Search Me=Charlotte
Seznam=SeznamBot
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snap=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TAMU.edu=IRLbot
Teoma=TeomaAgent
TinEye=TinEye
TMCrawler=TMCrawler
Twiceler=Twiceler-0.9
Voila=Voila
Voila=Voila.fr
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
Web Data Centre=WebDataCentreBot
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
WISEnut=ZyBorg
Worio=woriobot
YaCy=yacybot
Yahoo Blogs=Yahoo-blogs/v3.9
Yahoo China=Yahoo! Slurp China
Yahoo Germany=Yahoo! DE Slurp
Yahoo MM=Yahoo-MMcrawler
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Yandex=Yandex
Yanga=Yanga
Yisou=YisouSpider
YodaoBot=YodaoBot
Yoono=yoofind


Last Update: 2011.11.24

(Moderator Note: Please do not update this listing; the "Master List" is considered to be that on the Wiki and as such is taken from there for updates before being run through an alphabetizer and a duplicate checker; as such, changes to this post will be ignored and replaced after updated from the wiki.)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #237070 05/27/2010 5:01 PM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
Updated the above list with additions from this post.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242718 04/20/2011 11:30 PM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
Added Bing to the list.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242751 04/21/2011 6:23 PM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
Someone suggested (through a report) to add Facebook; it's not really a spider so I don't really want to add it... But for anyone who wants it:
Code
Facebook=facebookexternalhit


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #243908 06/14/2011 2:54 PM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
Added:
Code
Magpie=magpie-crawler


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #254093 11/24/2013 4:49 AM
Joined: Jun 2006
Posts: 15,851
Gizmo Offline OP
UBB.threads Developer
OP Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,851
Added several new entries, removed duplicate entries, updated names for several entries, alphabetized listing.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Forum Search
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
visual accessibility issues.v7.7.3
by Ruben - 08/21/2019 1:50 PM
Holy grail to fix compatibility bewteen browsers
by Ruben - 08/20/2019 4:25 PM
File Manager and Image Uploads
by Ty Griffin - 08/19/2019 4:48 PM
Minimum Post Character Count
by M4TT - 08/19/2019 1:58 PM
[FIXED for 7.7.4] Strange Merge Issue
by Baldeagle - 08/16/2019 3:37 PM
Who's Online Now
0 registered members (), 48 guests, and 231 spiders.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Photos
Stones
Amusing Terain Scenics
Sky places
Work spaces
Powered by UBB.threads™ PHP Forum Software 7.7.3