Previous Thread
Next Thread
Print Thread
Hop To
#228585 08/23/2009 7:19 PM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list. Please use this thread for comments.

Notice - This listing is no longer being maintained here; instead please see our official listing at UBBDev here and on the UBB.Wiki.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:
Code
80legs=008
A6-Indexer=A6-Indexer
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AllTheWeb=crawler@fast
AllTheWeb=FAST-WebCrawler
AllTheWeb=FAST
AltaVista=Mercator
Altavista=Scooter
Amazon.com=AMZNKAssocBot
Anzwers=AnzwersCrawl
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Ask.com=Teoma
Ask.com=AskJeeves
Atomz=Atomz
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho.com=boitho.com
Cityreview=Cityreview Robot
Crawler.de=Crawler
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
EntireWeb.com=Speedy Spider
Exalead=Exabot
Excite=Architext spider
Excite=ArchitextSpider
Facebook=facebookexternalhit
Factbites=Factbot
FreeWebMonitoring=FreeWebMonitoring SiteChecker
Gigablast=GigaBot
Google Adsense=Mediapartners-Google
Google Application Engine=AppEngine-Google
Google Images=Googlebot-Image
Google Mobile=Googlebot-Mobile
Google Sitemaps=Google-Sitemaps/1.0
Google=Googlebot
InfoSeek=InfoSeek sidewinder
InfoSeek=UltraSeek
Inktomi=Slurp
Kalooga=Kalooga
libwww=libwww-perl
LinkDex=linkdexbot
LookSmart=FurlBot
Looksmart=MantraAgent
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
Microsoft Research=MSRBOT
MJ12bot=MJ12bot
MSN Media=msnbot-media
MSN Photo Search=PSbot
MSN=msnbot
Naver.com=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight.com=Gulliver
noXtrum=noxtrumbot
Nutch Bot=Nutch12
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panopta Monitoring=checks.panopta.com
Panscient=panscient.com
Picsearch=psbot
Pintrest=Pinterest
PrivacyFinder=PrivacyFinder
Proximic=proximic
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Hippo=Fluffy the Spider
Search Me=Charlotte
Seznam=SeznamBot
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snap=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TAMU.edu=IRLbot
Teoma=TeomaAgent
TinEye=TinEye
TMCrawler=TMCrawler
Twiceler=Twiceler-0.9
Voila=Voila
Voila=Voila.fr
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
Web Data Centre=WebDataCentreBot
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
WISEnut=ZyBorg
Worio=woriobot
YaCy=yacybot
Yahoo Blogs=Yahoo-blogs/v3.9
Yahoo China=Yahoo! Slurp China
Yahoo Germany=Yahoo! DE Slurp
Yahoo MM=Yahoo-MMcrawler
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Yandex=Yandex
Yanga=Yanga
Yisou=YisouSpider
YodaoBot=YodaoBot
Yoono=yoofind

Last Update: 2011.11.24

(Moderator Note: Please do not update this listing; the "Master List" is considered to be that on the Wiki and as such is taken from there for updates before being run through an alphabetizer and a duplicate checker; as such, changes to this post will be ignored and replaced after updated from the wiki.)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #237070 05/27/2010 4:01 PM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
Updated the above list with additions from this post.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #242718 04/20/2011 10:30 PM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
Added Bing to the list.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #242751 04/21/2011 5:23 PM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
Someone suggested (through a report) to add Facebook; it's not really a spider so I don't really want to add it... But for anyone who wants it:
Code
Facebook=facebookexternalhit


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #243908 06/14/2011 1:54 PM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
Added:
Code
Magpie=magpie-crawler


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #254093 11/24/2013 3:49 AM
Joined: Jun 2006
Posts: 16,369
Likes: 126
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,369
Likes: 126
Added several new entries, removed duplicate entries, updated names for several entries, alphabetized listing.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
"Recent Posts" box
by SenecaFlyer - 12/13/2024 12:06 PM
Sticky posts and global announcements
by Larry Miller - 12/08/2024 2:30 PM
Profile avatar storage settings
by SenecaFlyer - 12/05/2024 1:24 PM
Not allowing attachment over 2m
by ehill - 12/03/2024 3:16 PM
New Admin Here
by SenecaFlyer - 12/02/2024 4:14 PM
Who's Online Now
3 members (Ruben, SenecaFlyer, 1 invisible), 1,719 guests, and 157 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Ride safe!
Ride safe!
by Morgan, December 7
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Powered by UBB.threads™ PHP Forum Software 8.0.1
(Snapshot build 20240918)