Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
2 registered members (Mark S, Gizmo), 19 guests, and 23 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
Bad Frog
Bad Frog
Coast of Maine
Posts: 740
Joined: May 2008
Show All Member Profiles 
Top Posters(30 Days)
isaac 106
Gizmo 96
ECNet 49
JAISP 28
Ruben 25
teamzr1 14
mmkk 14
ehm1 12
driv 11
Latest Photos
Some pictures for fun
test for attachment sizes
A gallery with animated GIFs
Lazy fox, drops, and a light house.
Welcome to poet
Previous Thread
Next Thread
Print Thread
#258864 - 01/04/17 11:03 AM Spider List  
Joined: Jan 2012
Posts: 94
dbremer Offline
journeyman
dbremer  Offline
journeyman

Joined: Jan 2012
Posts: 94
Minnesota
Is there a newer spider list than the one from 2013? Or, is this passe' and I am an idiot?


Life is Good on Bremer Pond

Bremer Pond Weather
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
#258865 - 01/04/17 12:15 PM Re: Spider List [Re: dbremer]  
Joined: Feb 2007
Posts: 1,309
JAISP Offline
JAISP  Offline


Joined: Feb 2007
Posts: 1,309
Pennsylvania
this is the list I have been using. I have added to it and updated it some.

Code
008=008dd
192.com=192.com
200PleaseBot=200PleaseBot
360Spider=360Spider
4seohuntBot=4seohuntBot
50.nu=50.nudd
^Nail=^Nail
A6-Indexer=A6-Indexer
abby=abby
Aboundexbot=Aboundexbot
AboutUsBot=AboutUsBot
Abrave Spider=Abrave Spider
Accelobot=Accelobot
AcoonBot=Acoon
AddThis.com=AddThis.com robot
ADmantX=ADmantX
adressendeutschland.de=adressendeutschland.de
AdsBot-Google=AdsBot-Google
AhrefsBot=AhrefsBot
aiHitBot=aiHitBot
akula=akula
alexa site audit=alexa site audit
Alexabot=Alexabot
Amagit.COM=Amagit.COM
amibot=amibot
AMZNKAssocBot=AMZNKAssocBot
AntBot=AntBot
Apercite=Apercite
AportWorm=AportWorm
AraBot=AraBot
arachnode.net=arachnode.net
Arachnophilia=Arachnophilia
archive.org arc=special_archiver
archive.org bot=archive.org_bot
Ask Jeeves=Ask Jeeves
AskQuickly=AskQuickly
Automattic Analytics Crawler=Automattic Analytics Crawler
BabalooSpider=BabalooSpider
backlink-check.de=backlink-check.de
BacklinkCrawler=BacklinkCrawler
Bad-Neighborhood=Bad Neighborhood Header Detector
Bad-Neighborhood=Bad-Neighborhood Link Analyzer
Baiduspider JP=Baiduspider japan
Baiduspider-Image=Baiduspider-image
Baiduspider=Baiduspider
baypup=baypup
BDCbot=BDCbot
BDFetch=BDFetch
BegunAdvertising=BegunAdvertising
bingbot=bingbot
bingbot=bingbot SitemapProbe
BingPreview=BingPreview
bitlybot=bitlybot
biwec=biwec
bixocrawler=adbeat-publisher-description-fetcher
bixocrawler=bixo
bixocrawler=bixocrawler
bixocrawler=bixolabs
bixocrawler=ptd-crawler
bl.uk_lddc_bot=bl.uk_lddc_bot
Blekkobot=Blekkobot
BLEXBot=BLEXBot
BlinkaCrawler=BlinkaCrawler
BlogPulse=BlogPulse
BlogPulse=BlogPulseLive
bnf.fr_bot=bnf.fr_bot
bot-pge.chlooe.com=bot-pge.chlooe.com
bot.wsowner.com=bot.wsowner.com
botmobi=botmobi
BotOnParade=BotOnParade
BrainbruBot=BrainbruBot
Browsershots=Browsershots
BUbiNG=BUbiNG
Butterfly=Butterfly
CamontSpider=CamontSpider
CareerBot=CareerBot
Castabot=Castabot
CatchBot=CatchBot
CCBot=CCBot
CCResearchBot=CCResearchBot
ChangeDetection=ChangeDetection
Charlotte=Charlotte
CirrusExplorer=CirrusExplorer
cityreview=cityreview
classbot=classbot
CligooRobot=CligooRobot
CliqzBot=Cliqz
CloudFlare-AlwaysOnline=CloudFlare-AlwaysOnline
CloudServerMarketSpider=CloudServerMarketSpider
CMS Crawler=CMS Crawler
coccoc=coccoc
COMODOSpider=Comodo
CompSpyBot=CompSpyBot
contentDetection=contentDetection
copyright sheriff=copyright sheriff
CorpusCrawler=CorpusCrawler
Covario-IDS=Covario
crawler for netopian=crawler for netopian
Crawler4j=Crawler4j
CrazyWebCrawler=crazywebcrawler
Crowsnest=Crowsnest
Curious George=Curious George
datagnionbot=datagnionbot
Daumoa=Daumoa
DBLBot=DBLBot
DCPbot=DCPbot
DealGates Bot=DealGates Bot
discoverybot=discobot
discoverybot=discoverybot
DKIMRepBot=DKIMRepBot
dlcbot=dlcbot
dlvr.it=dlvr.it
DNS-Digger-Explorer=DNS-Digger-Explorer
DomainAppender=DomainAppender
DomainDB=DomainDB
DomainTools-HeaderCheck=HTTP Headers Online
DomainTools-LinksCheck=Online Website Link Checker
DomainTools-SitemapGen=Online Sitemap Generator
DotBot=DotBot
dotSemantic=dotSemantic
DripfeedBot=DripfeedBot
drupact=drupact
DuckDuckPreview=DuckDuckPreview
EasouSpider=EasouSpider
EasyBib AutoCite=EasyBib AutoCite
eCairn-Grabber=eCairn-Grabber
eCommerceBot=eCommerceBot
EdisterBot=EdisterBot
Embedly=Embedly
emefgebot=emefgebot
envolk=envolk
Esribot=Esribot
EuripBot=EuripBot
Eurobot=Eurobot
EventGuruBot=EventGuruBot
EveryoneSocialBot=EveryoneSocialBot
EvriNid=EvriNid
Exabot Images=Exabot-Images
Exabot Thumbnails=Exabot-Thumbnails
Exabot=Exabot
Exabot=ExaleadCloudview
Experibot=Experibot
Ezooms=Ezooms
FacebookExternalHit=FacebookExternalHit
facebookPlatform=facebookplatform
factbot=Factbot 1.09
FairShare=FairShare
Falconsbot=Falconsbot
fastbot crawler=fastbot crawler
FauBot=FauBot
FeedCatBot=FeedCatBot
FeedFinder=FeedFinder
Feedly=Feedly
Feedly=FeedlyBot
Fetch-Guess=Fetch
findlinks=findlinks
firmilybot=firmilybot
Flatland Industries Web Spider=flatlandbot
FlightDeckReportsBot=FlightDeckReportsBot
FlipboardProxy=FlipboardProxy
Flocke bot=Flocke bot
FollowSite Bot=FollowSite Bot
Fooooo_Web_Video_Crawl=Fooooo_Web_Video_Crawl
FreeWebMonitoring=FreeWebMonitoring
FyberSpider=FyberSpider
Gaisbot=Gaisbot
GarlikCrawler=GarlikCrawler
GeliyooBot=GeliyooBot
Genieo Web filter=Genieo
GigablastOpenSource=GigablastOpenSource
Gigabot=Gigabot
GingerCrawler=GingerCrawler
Girafabot=Girafabot
gocrawl=gocrawl
GOFORITBOT=GOFORITBOT
gonzo=gonzo
Googlebot AdsBot Mobile=AdsBot-Google-Mobile
Googlebot AdSense=Mediapartners-Google
Googlebot Image=Googlebot-Image
Googlebot Mobile=Googlebot-Mobile
Googlebot Snippet=Googlebot snippet
Googlebot Video=Googlebot-Video
Googlebot Web Preview=Google Web Preview
Googlebot=Googlebot
Grahambot=Grahambot
GrapeshotCrawler=grapeFX
GrapeshotCrawler=GrapeshotCrawler
GurujiBot=GurujiBot
Hailoobot=Hailoobot
HatenaScreenshot=HatenaScreenshot
hawkReader=hawkReader
HeartRails_Capture=HeartRails_Capture
heritrix=heritrix
Holmes=holmes
HolmesBot=HolmesBot
HomeTags=HomeTags
HostTracker.com=HostTracker.com
HostTracker=HostTracker
HuaweiSymantecSpider=HuaweiSymantecSpider
HubSpot Connect=HubSpot Connect
HubSpot Crawler=HubSpot Crawler
HypeStat=HypeStat
ia_archiver=ia_archiver
ICC-Crawler=ICC-Crawler
ichiro=ichiro
iCjobs=iCjobs
IdeelaborPlagiaat=IdeelaborPlagiaat
idmarch=idmarch Automatic.beta
Iframely=Iframely
imbot=imbot
immediatenet thumbnails=immediatenet thumbnails
ImplisenseBot=ImplisenseBot
Impressumscrawler=Impressumscrawler
Influencebot=Influencebot
Infohelfer=Infohelfer
IntegromeDB=IntegromeDB
IstellaBot=IstellaBot
IXEbot=IXEbot
Jabse.com Crawler=Jabse.com Crawler
JadynAveBot=JadynAveBot
JamesBOT=JamesBOT
JikeSpider=JikeSpider
Job Roboter Spider=Job Roboter Spider
JUST-CRAWLER=JUST-CRAWLER
Jyxobot=Jyxobot
Jyxobot=JyxobotRSS
Kalooga=Kalooga
Karneval-Bot=Karneval-Bot
kinshoo=KiNShooboT
KomodiaBot=KomodiaBot
Kraken=Kraken
KrOWLer=KrOWLer
kulturarw=kulturarw3
L.webis=L.webis
Leikibot=Leikibot
LemurWebCrawler=LemurWebCrawler
LexxeBot=LexxeBot
Lijit=Lijit
LinguaBot=LinguaBot
linguatools=linguatools
Linguee Bot=Linguee Bot
Link Valet Online=Link Valet Online
LinkAider=LinkAider
linkdex.com=linkdex.com
linkdexbot=linkdexbot
LinkedInBot=LinkedInBot
LinkWalker=LinkWalker
Lipperhey Spider=Lipperhey Spider
livedoor ScreenShot=livedoor ScreenShot
LivelapBot=LivelapBot
LoadImpactPageAnalyzer=LoadImpactPageAnalyzer
LoadTimeBot=LoadTimeBot
LuminateBot=LuminateBot
magpie-crawler=magpie-crawler
Mail.Ru bot=Mail.Ru
meanpathbot=meanpathbot
MeMoNewsBot=MeMoNewsBot
memoryBot=memoryBot
MetaGeneratorCrawler=MetaGeneratorCrawler
MetaHeadersBot=MetaHeadersBot
MetaJobBot=MetaJobBot
MetamojiCrawler=MetamojiCrawler
Metaspinner=Metaspinner
MetaURI API=MetaURI API
MetaURI=MetaURI
MIA Bot=MIA Bot
MiaDev=MiaDev
MixBot=MixBot
MJ12bot=MJ12bot
MLBot=MLBot
MnoGoSearch=MnoGoSearch
Moatbot=Moatbot
moba-crawler=moba-crawler
MojeekBot=MojeekBot
Motoricerca-Robots.txt-Checker=Motoricerca-Robots.txt-Checker
Mp3Bot=Mp3Bot
MSNBot Media=msnbot-media
MSNBot News=msnbot-NewsBlogs
MSNBot UDiscovery=msnbot-UDiscovery
MSNBot=adidxbot
MSNBot=MSNBot
MSRBOT=MSRBOT
musobot=musobot
Najdi.si=Najdi.si
NalezenCzBot=NalezenCzBot
NaverBot=NaverBot
NaverBot=Yepi
NaverBot=Yeti
NaverBot=Yeti-FeedItemCrawler
NaverBot=Yeti-Mobile
nekstbot=nekstbot
NerdByNature.Bot=NerdByNature.Bot
NerdyBot=NerdyBot
NetcraftSurveyAgent=NetcraftSurveyAgent
netEstate Crawler=netEstate NE Crawler
netEstate Crawler=netEstate RSS crawler
NetResearchServer=nrsbot
Netseer=Netseer crawler
NextGenSearchBot=NextGenSearchBot
Nigma.ru=Nigma.ru
NLNZ_IAHarvester2013=NLNZ_IAHarvester2013
nodestackbot=nodestackbot
Nuhk=Nuhk
Nutch=Nutch
nworm=nwormFeedFinder
Nymesis=Nymesis
oBot=oBot
Ocelli=Ocelli
omgilibot=omgilibot
OoyyoBot=OoyyoBot
Open Web Analytics Bot=Open Web Analytics Bot
OpenCalaisSemanticProxy=OpenCalaisSemanticProxy
OpenindexSpider=OpenindexDeepSpider
OpenindexSpider=OpenindexShalooowSpider
OpenindexSpider=OpenindexSpider
OpenWebSpider=OpenWebSpider
OrgbyBot=OrgbyBot
OsObot=OsObot
ownCloud Server Crawler=ownCloud Server Crawler
Page2RSS=Page2RSS
page_verifier=page_verifier
PagePeeker=PagePeeker
Panscient web crawler=Panscient web crawler
PaperLiBot=PaperLiBot
ParchBot=ParchBot
parsijoo=parsijoo
PayPal IPN=PayPal IPN
Peeplo Screenshot Bot=Peeplo Screenshot Bot
Peepowbot=Peepowbot
peerindex=peerindex
Peew=Peew
PercolateCrawler=percbotspider
PercolateCrawler=PercolateCrawler
pingdom.com_bot=pingdom.com_bot
Pinterest=Pinterest
PiplBot=PiplBot
Pixray-Seeker=Pixray-Seeker
Plukkie=Plukkie
pmoz.info ODP link checker=pmoz.info ODP link checker
Pompos=Pompos
PostPost=PostPost
pr-cy.ru Screenshot Bot=pr-cy.ru
ProCogBot=ProCogBot
ProCogSEOBot=ProCogSEOBot
proximic=proximic
psbot=psbot
psbot=psbot-page
Qirina Hurdler=Qirina Hurdler
Qseero=Qseero
Qualidator.com Bot=Qualidator.com Bot
Qualidator.com SiteAnalyzer=Qualidator.com SiteAnalyzer
QuerySeekerSpider=QuerySeekerSpider
quickobot=quickobot
R6 bot=R6_CommentReader
R6 bot=R6_FeedFetcher
RADaR-Bot=RADaR-Bot
RankurBot=RankurBot
Readability=Readability
Robots_Tester=Robots_Tester
Robozilla=Robozilla
rogerbot=rogerbot
Ronzoobot=Ronzoobot
RSSMicro.com RSS Atom Feed Robot=RSSMicro.com RSS Atom Feed Robot
Ruky-Roboter=Ruky-Roboter
RyzeCrawler=RyzeCrawler
SAI Crawler=SAI Crawler
SanszBot=SanszBot
SBIder=SBIder
SBSearch=SBSearch
Scarlett=Scarlett
SCFCrawler=SCFCrawler
Scooter=Scooter
ScoutJet=ScoutJet
ScoutJet=ScoutJet old
Scrapy=Scrapy
ScreenerBot Crawler=ScreenerBot Crawler
Scrubby=Scrubby
search.KumKie.com=search.KumKie.com
Search17Bot=Search17Bot
SearchmetricsBot=SearchmetricsBot
SecurityResearchBot=SecurityResearchBot
seegnifybot=seebot
seegnifybot=seegnifybot
Semager=Semager
Semantifire=Semantifire1
SemrushBot=SemrushBot
Seobility=Seobility SEO-Check
Seobility=Seobility Urlstat
SEOCentro Keywords=KeywordDensityRobot
SEOCentro MetaTags=MetaTagRobot
SEOCentro SEO Keywords=SEOCentro Page Keyword Analyzer v1.2
SeoCheckBot=SeoCheck
SEODat=SEODat
SEOdiver=SEOdiver
SEOENGBot=SEOENGBot
SEOkicks-Robot=SEOkicks-Robot
Setoozbot=OOZBOT
Setoozbot=SETOOZBOT
SeznamBot=Seznam screenshot-generator
SeznamBot=SeznamBot
SeznamBot=SklikBot
Shareaholicbot=Shareaholicbot
Shelob=Shelob
ShopWiki=ShopWiki
ShowyouBot=ShowyouBot
sistrix=sistrix
SiteCondor=SiteCondor
Sitedomain-Bot=Sitedomain-Bot
Slackbot=Slackbot-LinkExpanding
smart.apnoti.com Robot=smart.apnoti.com Robot
SMTBot=SMTBot
Snapbot=Snapbot
SniffRSS=SniffRSS
socialbm_bot=socialbm_bot
sogou spider=Sogou
SolomonoBot=SolomonoBot
Sosospider=Sosoimagespider
Sosospider=Sosospider
spbot=spbot
Speedy=Speedy Spider
Speedy=Speedy Spider Beta
SpiderLing=SpiderLing
Spinn3r=Spinn3r
SputnikBot=SputnikBot
SSL-Crawler=SSL-Crawler
SSLBot=SSLBot
StackRambler=StackRambler
StatoolsBot=StatoolsBot
Steeler=Steeler
STINGbot=STINGbot
stq_bot=stq_bot
Strokebot=Strokebot
suggybot=suggybot
SurcentroBot=SurcentroBot
Surphace Scout=Surphace Scout
SurveyBot=SurveyBot
SWEBot=SWEBot
SygolBot=SygolBot
Symfony Spider=Symfony Spider
Szukacz=Szukacz
Tagoobot=Tagoobot
taptubot=taptubot
Technoratibot=Technoratibot
Thumbnail.CZ robot=Thumbnail.CZ robot
ThumbShots-Bot=ThumbShots-Bot
thumbshots-de-Bot=thumbshots-de-bot
Thumbshots.ru=Thumbshots.ru
ThumbSniper=ThumbSniper
TinEye=TinEye
TomTom places company search=TomTom places company search
Topicbot=Topicbot
Toread-Crawler=Toread-Crawler
Touche=Touche
trendictionbot=trendictionbot
TurnitinBot=TurnitinBot
TwengaBot=TwengaBot
TwengaBot=TwengaBot-Discover
Twiceler=Twiceler
Twikle=Twikle
Twingly Recon=Twingly Recon
UASlinkChecker=UASlinkChecker
uMBot FC=uMBot-FC
uMBot LN=uMBot-LN
UnisterBot=UnisterBot
UnwindFetchor=UnwindFetchor
Updownerbot=Updownerbot
UptimeDog=UptimeDog
UptimeRobot=UptimeRobot
URLAppendBot=URLAppendBot
urlfan-bot=urlfan-bot
Urlfilebot (Urlbot)=Urlfilebot
Vagabondo=Vagabondo
Vedma=Vedma
VideoSurf_bot=VideoSurf_bot
Visbot=Visbot
VoilaBot=VoilaBot
voltron=voltron
voyager=voyager
WASALive-Bot= WASALive-Bot
WatchMouse=WatchMouse
WBSearchBot=WBSearchBot
Web-Monitoring=Web-Monitoring
Web-sniffer=Web-sniffer
WebCookies=WebCookies
WebCorp=WebCorp
WebImages=WebImages
webinatorbot=webinatorbot
webmastercoffee=webmastercoffee
WebNL=WebNL
WebRankSpider=WebRankSpider
WebTarantula.com Crawler=WebTarantula.com Crawler
WebThumbnail=WebThumbnail
WebWatch Robot_txtChecker=WebWatch
WeSEE:Ads=WeSEE:Ads/PageBot
WeSEE:Ads=WeSEE:Ads/PictureBot
WeSEE:Search=WeSEE
WeSEE:Search=WeSEE:Search
WeViKaBot=WeViKaBot
Whoismindbot=Whoismindbot
WikioFeedBot=WikioFeedBot
wikiwix-bot=wikiwix-bot
Willow Internet Crawler=Willow Internet Crawler
WillyBot=WillyBot
WinWebBot=WinWebBot
WMCAI_robot=WMCAI_robot
Woko=Woko
WordPress.com mShots=WordPress.com mShots
woriobot=woriobot
Wotbox=Wotbox
wsAnalyzer=wsAnalyzer
wscheck.com=wscheck.com
x28-job-bot=x28-job-bot
XmarksFetch=XmarksFetch
XML Sitemaps Generator=XML Sitemaps Generator
XoviBot=XoviBot
XRL=XRL
Yaanb=Yaanb
yacybot=yacybot
Yahoo!=Y!J-BRI
Yahoo!=Y!J-BRJ
Yahoo!=Y!J-BRO
Yahoo!=Y!J-BRW
Yahoo!=Y!J-BSC
Yahoo!=Yahoo! Slurp
Yahoo!=Yahoo-MMCrawler
Yahoo!=YahooCacheSystem
YamanaLab-bot=Sonic
YandexBot AntiVirus=YandexAntivirus
YandexBot Blogs=YandexBlogs
YandexBot Catalog=YandexCatalog
YandexBot Direct=YandexDirect
YandexBot Favicons=YandexFavicons
YandexBot ImageResizer=YandexImageResizer
YandexBot Images=YandexImages
YandexBot Media=YandexMedia
YandexBot Metrix=YandexMetrika
YandexBot News=YandexNews
YandexBot Server=Yandex.Server
YandexBot Something=YandexSomething
YandexBot Video=YandexVideo
YandexBot Webmaster=YandexWebmaster
YandexBot Zakladki=YandexZakladki
YandexBot=Yandex
Yanga=Yanga
YioopBot=gofind
YioopBot=YioopBot
YodaoBot Image=YodaoBot-Image
YodaoBot=YodaoBot
YoudaoBot=YoudaoBot
YowedoBot=YowedoBot
YRSpider=YRSpider
YYSpider=YYSpider
ZeerchBot LA1=ZeerchBot LA1
ZeerchBot LA2=ZeerchBot LA2
ZeerchBot=ZeerchBot
Zookabot=Zookabot
ZumBot=ZumBot
Shareaholic=Shareaholic-bot
PrivacyAwareBot=PrivacyAwareBot
Twitterbot/1.0=Twitterbot/1.0
NING/1.0=NING/1.0
Bingbot/2.0=Bingbot/2.0
SeznamBot/3.2=SeznamBot/3.2
WikiDo/1.1=WikiDo/1.1
TweetmemeBot/4.0=TweetmemeBot/4.0
Ruby Website Monitor=Ruby
YottaaMonitor=YottaaMonitor
LinkedIN=um-LN/1.0
Facebook=facebookexternalhit/1.1

#258866 - 01/04/17 01:49 PM Re: Spider List [Re: dbremer]  
Joined: Jan 2012
Posts: 94
dbremer Offline
journeyman
dbremer  Offline
journeyman

Joined: Jan 2012
Posts: 94
Minnesota
Nice work! Can I borrow that? Oh, I guess you wouldn't have posted it if you wanted to keep it secret. smile

Thanks.


Life is Good on Bremer Pond

Bremer Pond Weather
#258867 - 01/04/17 02:05 PM Re: Spider List [Re: dbremer]  
Joined: Feb 2007
Posts: 1,309
JAISP Offline
JAISP  Offline


Joined: Feb 2007
Posts: 1,309
Pennsylvania
Anyone is free to use that. It's a combination of my findings and many others in the UBB community.

#258868 - 01/04/17 05:42 PM Re: Spider List [Re: dbremer]  
Joined: Apr 2004
Posts: 850
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer

Joined: Apr 2004
Posts: 850
California
Originally Posted by dbremer
Is there a newer spider list than the one from 2013? Or, is this passe' and I am an idiot?


The most recent version is from October 2014, at
http://www.ubbdev.com/forums/ubbthr...earch-engine-spiders-for-ubbthreads.html

I believe that is the source for where JAISP's edited list came from, since JAISP's list is identical, except for his 12 trailing entries.

http://user-agent-string.info is now a subscription service. The list provided on UBBDev.com is the very most recent version of the data prior to user-agent-string.info becoming a subscription service.

#258869 - 01/04/17 06:03 PM Re: Spider List [Re: dbremer]  
Joined: Jun 2006
Posts: 16,627
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer

Joined: Jun 2006
Posts: 16,627
Portland, OR; USA
I wish that the donation method kept them afloat; they where a really useful service, but I'm not going to pay for a subscription at $42/yr for UA strings, lol.


Forums: A Gardeners Forum
UBB.Threads: UBBWiki, UBBSkins, UBB.Sitemaps
Longtime Supporter & Beta Tester & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
#258870 - 01/04/17 07:10 PM Re: Spider List [Re: dbremer]  
Joined: Feb 2007
Posts: 1,309
JAISP Offline
JAISP  Offline


Joined: Feb 2007
Posts: 1,309
Pennsylvania
Yep the last 12 entries I added in the last 6 months from watching hits on my site with the who's online page. I wish the new who's online page had all the info like the old one had, such a shame that was lost in the new version of the UBB.

#258871 - 01/04/17 07:36 PM Re: Spider List [Re: dbremer]  
Joined: Jun 2006
Posts: 16,627
Gizmo Online tapedshut
UBB.threads Developer
Gizmo  Online Tapedshut
UBB.threads Developer

Joined: Jun 2006
Posts: 16,627
Portland, OR; USA
I spent some time on the search engines earlier and stumbled on this, but I haven't compared it to the latest listing that we where managing in 2009 (way too much going on lol).

Originally Posted by JAISP
I wish the new who's online page had all the info like the old one had, such a shame that was lost in the new version of the UBB.
What's different about it? Are you talking from v6 UBB.classic -> v7 UBB.threads, or v7.6.0 vs 7.5.x?


Forums: A Gardeners Forum
UBB.Threads: UBBWiki, UBBSkins, UBB.Sitemaps
Longtime Supporter & Beta Tester & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
#258872 - 01/05/17 02:26 AM Re: Spider List [Re: JAISP]  
Joined: Apr 2004
Posts: 850
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer

Joined: Apr 2004
Posts: 850
California
Originally Posted by JAISP
I wish the new who's online page had all the info like the old one had, such a shame that was lost in the new version of the UBB.


Your comment is spurious and lacks any sort of merit:

The Who's Online page has been completely rewritten for UBB.threads 7.6.0
• It is now mobile-friendly.
• It now supports IPv6.
• The list of Search Spiders is only displayed to users who have "View IP Addresses" privileges.
• Users must be logged in to a Registered User account to see the "Last Activity" time/date columns.
• Added a jump-list to the top for each of the WOL sections.
• Added IPv4 and IPv6 address lookup links.
• Added an INFO column within the Anonymous Guests section. If the user browsing your forums was referred from another website, within this column, you can hover your mouse over the [Linked Image] icon and view information regarding their browser's User Agent Strings (UAS) and the referring website/page. Click on the [Linked Image] icon to view that web page.
• Also within the Anonymous Guests INFO section, the referrals from the top 14 search engines (as of 2016Q4) will now be displayed. If a user was referred from an image search, the referral name will be listed in bold text.
• Added an INFO column within the Search Spiders section. If you've imported a list of spiders to your your forums (Control Panel > Display Settings > Primary | Who's Online Settings, Search Engine Agents), the spider will be listed here. You can Hover your mouse over the [Linked Image] icon and view information regarding that spider's User Agent Strings (UAS). Click on the [Linked Image] icon to search google for their full UAS. The name of the spider is also shown next to the [Linked Image] icon. Click the name to lookup that BOT on Google search.
• The Location column has been condensed and is now less-redundant.

The only thing removed from the WOL page was the (bolded item above) duplication of the "Forum Title" when a user is reading a post. It now only displays the "Thread Title," instead of "Forum Title > Thread Title" when a user is reading a post. But if the user is just viewing the list of threads within the forum, the "Forum Title" is displayed as it always has been.

Other than the single redundant item mentioned above, every single piece of data that was displayed in UBB.threads 7.5.x series Whos Online page, is still within the 7.6.x series. Nothing has been "lost."

#258956 - 01/20/17 07:31 PM Re: Spider List [Re: JAISP]  
Joined: Mar 2007
Posts: 195
ECNet Offline
member
ECNet  Offline
member
Joined: Mar 2007
Posts: 195
Originally Posted by JAISP
this is the list I have been using. I have added to it and updated it some.


I put this list in (above) and it seems to put all non-logged in Members under the Spiders category. I get no Anonymous Guests

Is there something on that list that might cause that?

Bill
v7.5.9

#258957 - 01/20/17 07:36 PM Re: Spider List [Re: ECNet]  
Joined: Apr 2004
Posts: 850
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer

Joined: Apr 2004
Posts: 850
California
Originally Posted by id242
Originally Posted by dbremer
Is there a newer spider list than the one from 2013? Or, is this passe' and I am an idiot?


The most recent version is from October 2014, at
http://www.ubbdev.com/forums/ubbthr...earch-engine-spiders-for-ubbthreads.html

I believe that is the source for where JAISP's edited list came from, since JAISP's list is identical, except for his 12 trailing entries.

http://user-agent-string.info is now a subscription service. The list provided on UBBDev.com is the very most recent version of the data prior to user-agent-string.info becoming a subscription service.



Originally Posted by ECNet
Originally Posted by JAISP
this is the list I have been using. I have added to it and updated it some.


I put this list in (above) and it seems to put all non-logged in Members under the Spiders category. I get no Anonymous Guests

Is there something on that list that might cause that?


Follow the directions on the link I posted (the source for where Jaisp had copied from).
Specifically, "Having problems? REMOVE any blank lines from the top/bottom of your copied list."

If all the lines in your WOL are still showing up as spiders, then they really are spiders, and not anon humans. The spider name is displayed in the left column of your WOL page.

#258959 - 01/20/17 08:13 PM Re: Spider List [Re: dbremer]  
Joined: Mar 2007
Posts: 195
ECNet Offline
member
ECNet  Offline
member
Joined: Mar 2007
Posts: 195
Nevermind, I figured it out. Somehow I had a blank line following the last item

Bill
blush

#259750 - 03/06/17 07:31 PM Re: Spider List [Re: dbremer]  
Joined: May 2008
Posts: 740
Bad Frog Offline
old hand
Bad Frog  Offline
old hand

Joined: May 2008
Posts: 740
Coast of Maine
I added bingbot/2.0=bingbot/2.0 to mine


"No matter where you go, there you are."
"If you can't do something smart, Do something right"
"There are three kinds of people in the world, those who can count, and those who can't"

Shout Box
Today's Birthdays
No Birthdays
Recent Topics
7.6 take significantly longer to load
by Morgan. 03/25/17 06:24 PM
collapsible catetgories
by Bad Frog. 03/23/17 10:26 AM
Forum Time Issue?
by ECNet. 03/19/17 08:37 PM
UBB.threads 7.6.1 - Not Yet Released
by id242. 03/19/17 01:46 PM
Dashboard Style?
by ECNet. 03/19/17 10:58 AM
Forum Statistics
Forums36
Topics34,911
Posts189,860
Members11,975
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.1
(Snapshot build 20170327)