Spider List - UBB.threads PHP Forum Software Community

You are not logged in. [Log In] UBB.threads PHP Forum Software Community » Forums » UBB.threads 7 Discussion » UBB.threads FAQ Forum » Spider List

Site Links

Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com

Top Posters (30 Days)

Ruben	68
Gizmo	57
SD	57
gliderdad	33
Dunny	22
Iann128	22
Steve C	20
driv	18
dbremer	16
Stan	15

Latest Photos

Basildon Town Centre looking from the rounderbout

Topic Options

#228585 - 08/23/09 08:19 PM

Spider List

Gizmo

Registered: 06/05/06
Posts: 14904
Loc: Portland, OR; USA

This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list.

This page is located on the UBB.Wiki as well.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:

Code:

Accelovation=heritrix
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
AllTheWeb=crawler@fast
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind
80legs=008
Yahoo=Yahoo! Slurp
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Crawler.de=Crawler
EntireWeb.com=Speedy Spider
Gigabot=Gigabot
Google Sitemaps=Google-Sitemaps/1.0
Inktomi=Slurp
MSN=MSNBOT
Naver.com=Yeti/
NorthernLight.com=Gulliver
Searchme=Charlotte
StackRambler(ru)=StackRambler
TAMU.edu=IRLbot
Twiceler=Twiceler-0.9
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
Yahoo=Yahoo Slurp
Excite=Architext spider
Infoseek=Ultraseek
Gigablast=GigaBot
Google-Image=Googlebot-Image
Snap=Snapbot
WISEnut=ZyBorg

Last Update: 2011.06.14

_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!

Top

Express Hosting

Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."