Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
1 registered members (mmkk), 57 guests, and 394 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
Mark S
Mark S
Liverpool : England : UK
Posts: 4,704
Joined: July 2006
Show All Member Profiles 
Top Posters(30 Days)
Gizmo 18
M4TT 12
FREAK 11
Ruben 8
mmkk 4
isaac 4
Latest Photos
Chinese Buddhist temple.
My buddha beads.
Rendered Walls
Multi-Screen wallpaper
Stockholm Metro
Previous Thread
Next Thread
Print Thread
Spider List #228585
08/23/09 07:19 PM
08/23/09 07:19 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list. Please use this thread for comments.

Notice - This listing is no longer being maintained here; instead please see our official listing at UBBDev here and on the UBB.Wiki.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:
Code
80legs=008
A6-Indexer=A6-Indexer
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AllTheWeb=crawler@fast
AllTheWeb=FAST-WebCrawler
AllTheWeb=FAST
AltaVista=Mercator
Altavista=Scooter
Amazon.com=AMZNKAssocBot
Anzwers=AnzwersCrawl
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Ask.com=Teoma
Ask.com=AskJeeves
Atomz=Atomz
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho.com=boitho.com
Cityreview=Cityreview Robot
Crawler.de=Crawler
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
EntireWeb.com=Speedy Spider
Exalead=Exabot
Excite=Architext spider
Excite=ArchitextSpider
Facebook=facebookexternalhit
Factbites=Factbot
FreeWebMonitoring=FreeWebMonitoring SiteChecker
Gigablast=GigaBot
Google Adsense=Mediapartners-Google
Google Application Engine=AppEngine-Google
Google Images=Googlebot-Image
Google Mobile=Googlebot-Mobile
Google Sitemaps=Google-Sitemaps/1.0
Google=Googlebot
InfoSeek=InfoSeek sidewinder
InfoSeek=UltraSeek
Inktomi=Slurp
Kalooga=Kalooga
libwww=libwww-perl
LinkDex=linkdexbot
LookSmart=FurlBot
Looksmart=MantraAgent
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
Microsoft Research=MSRBOT
MJ12bot=MJ12bot
MSN Media=msnbot-media
MSN Photo Search=PSbot
MSN=msnbot
Naver.com=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight.com=Gulliver
noXtrum=noxtrumbot
Nutch Bot=Nutch12
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panopta Monitoring=checks.panopta.com
Panscient=panscient.com
Picsearch=psbot
Pintrest=Pinterest
PrivacyFinder=PrivacyFinder
Proximic=proximic
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Hippo=Fluffy the Spider
Search Me=Charlotte
Seznam=SeznamBot
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snap=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TAMU.edu=IRLbot
Teoma=TeomaAgent
TinEye=TinEye
TMCrawler=TMCrawler
Twiceler=Twiceler-0.9
Voila=Voila
Voila=Voila.fr
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
Web Data Centre=WebDataCentreBot
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
WISEnut=ZyBorg
Worio=woriobot
YaCy=yacybot
Yahoo Blogs=Yahoo-blogs/v3.9
Yahoo China=Yahoo! Slurp China
Yahoo Germany=Yahoo! DE Slurp
Yahoo MM=Yahoo-MMcrawler
Yahoo=Yahoo! Slurp
Yahoo=Yahoo Slurp
Yandex=Yandex
Yanga=Yanga
Yisou=YisouSpider
YodaoBot=YodaoBot
Yoono=yoofind


Last Update: 2011.11.24

(Moderator Note: Please do not update this listing; the "Master List" is considered to be that on the Wiki and as such is taken from there for updates before being run through an alphabetizer and a duplicate checker; as such, changes to this post will be ignored and replaced after updated from the wiki.)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: Spider List [Re: Gizmo] #237070
05/27/10 04:01 PM
05/27/10 04:01 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Updated the above list with additions from this post.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242718
04/20/11 10:30 PM
04/20/11 10:30 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Added Bing to the list.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #242751
04/21/11 05:23 PM
04/21/11 05:23 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Someone suggested (through a report) to add Facebook; it's not really a spider so I don't really want to add it... But for anyone who wants it:
Code
Facebook=facebookexternalhit


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #243908
06/14/11 01:54 PM
06/14/11 01:54 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Added:
Code
Magpie=magpie-crawler


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Spider List [Re: Gizmo] #254093
11/24/13 03:49 AM
11/24/13 03:49 AM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline OP
UBB.threads Developer
Gizmo  Offline OP
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Added several new entries, removed duplicate entries, updated names for several entries, alphabetized listing.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Users Unable to Upload Avatar [Not a Bug]
by M4TT. 12/13/17 08:51 AM
Shout Box Sound Effect
by M4TT. 11/29/17 08:28 PM
Ad island
by TGCsanderson. 11/25/17 06:41 PM
Taking to long to connect to DB
by AstroCat. 11/24/17 12:34 PM
Forum Statistics
Forums36
Topics35,015
Posts190,544
Members12,045
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.1
(Snapshot build 20171106)