|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
This is bizarre... My site is currently being visited by nearly 10 times the normal number of anonymous browsers, and this has been going on for several hours. I am seeing 60 to 80 anonymous users when the norm is 2 to 5. What is going on???
By cataloging the IP addresses of the anonymous browsers, I see that each one is on for only a short time -- several cycles of the Who's Online page. Each one seems to be visiting random posts.
I posted a message asking anyone to please email me to let me know where they got a link to my site, but not one of them seems to be visiting that post. So I am thinking that somehow, we have a web robot hitting my site, using an older index of all the links. But the odd thing is that it is using continually changing IP addresses.
Can a search spider change IP addresses like that? Or could it be infected computers from all over hitting the site, and then reporting keywords back to a "mother ship" site?
Very curious....
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
Probably is spiders. If you have not ever edited the agent table. The default install only lists a few. So if you don't add a spider to the list they are a unknown visitor There are a couple of posts here with personal settings. Goto: Control Panel » Display Options[+] Who's Online Search Engine Agents My list although not complete and probably never will is: Accelovation=heritrix
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
AllTheWeb=crawler@fast
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BecomeBot=BecomeBot
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind You can just cut and paste the list in your list Probably will cut down on some of the unknowns anyway. As new spiders come out some good some bad the list needs to be updated on a regular basis but it becomes a big task. If you identify a bad spider then you can ban the ip.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
Thanks! I'll paste your list into my site.
More info... I found on the Who's Online page, if I hover my mouse over the little "i" icon, it pops up the spider's identity. But why, when I hover over my own connection it reads
...Gecko/20100401 Firefox/3.6.3...
as part of the information. I am using Firefox 3/6/3, but what is the "Gecko" part??? (and you don't have Gecko in your list). But a large number of my anonymous browsers show:
Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Referer:
How do I add that one to my spiders list?
And THANKS for the help!
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
Gecko is a layout engine so don't worry about it. Click here for Gecko But if you want to see the ip then goto: Control Panel » Group Management » Administrators>Site Permissions Can See Extra User Info Set this to a 1 Then you can see the ip of all who is online.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
BTW 80legs.com is a spider. Quick down and dirty You can look up www.80legs.com.At first glance the format to enter in the list would be: 80legs=008
Last edited by Ruben; 05/26/2010 4:27 PM. Reason: Added spider
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
Yes, 80legs.com is a spider. Here's their web site For Webmasters page. How do I add it to my list of Search Engine Agents? Will this entry do it? 80legs=80legs
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
I edited my previous post at the same time as your post. Give it a try.
They don't visit my site.
actually the name before the = can be anything you want.
Last edited by Ruben; 05/26/2010 4:29 PM.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
...and at this time, they have stopped crawling the site.
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
Ok, I see you added the 80legs=008. I'll add it to my list.
But I don't mind them hitting the site. Just like to know if it is a person or a robot.
...and I already have my admin access set to see the IP addresses. Thanks for that.
Thanks for all your help. I appreciate it.
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
How does the saying go or at least your governor:
I'll be back!
BTW the spider list on your site is blank. Maybe you need to clear the cache.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
> BTW the spider list on your site is blank.Do you mean on the Who is online display? Ok, I cleared the cache. Now I see three spiders online: Cuilx2 and Google. As for the governor: He's termed-out this year. Got new ones coming along. But with this state, I can't imagine why they'd want to be governor. We've got "democracy gone to seed".... a simple 50% majority of voters passes every proposition (anyone can create one, and get a ~million signatures, spend big $ on a deceptive campaign, and it becomes part of the state's constitution.) Therefore we have property taxes voted so low the state is bankrupt, and the legislature is required to pass its budget with a 2/3 majority. "No" votes carry twice the weight as "Yes" votes (Is that "equal representation"???) SO budgets are chronically late, every pizza thief and drug user is in jail for 25 years, the prison guards get $100K per year and retire on full salary at 50. After 30 years, the best education system in the nation is now the poorest. CA is racing towards Third World economy status.  ...Oh sorry, I was ranting wasn't I. 
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
HA HA.
Oh here is another to add the the list.
They just hit me during our talk.
BaiDuSpider=BaiDuSpider
Let me know if the other one works from 80legs.
They brag that they use multiple servers so it may end up being several entries.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
Just peeking at 80 legs it appears that anyone can submit any site to be crawled by them.
I just tried the free plan and they started crawling.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
Wow! And I see your Who is Online page has correctly identified them as a spider. 
|
|
|
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88
|
|
Joined: Dec 2003
Posts: 6,649 Likes: 88 |
Yea it looks to be a pay per use web crawl service.
It does not have a repeat service for free.
So someone either submitted your page to mess with you or in error or they allowed follow links from their own site to redirect to your site.
But anyway the agent listing works for them.
And it looks like if they are using the free service they would have to manually re-enter a url.
Blue Man Group
There is no such thing as stupid questions. Just stupid answers
|
|
|
|
|
Joined: May 2006
Posts: 243 Likes: 1
Enthusiast
|
|
Enthusiast
Joined: May 2006
Posts: 243 Likes: 1 |
> So someone either submitted your page to mess with you or in error or they allowed follow links from their own site to redirect to your site.No matter. I am trying everything to get more users to come to the site. In fact, I'd really appreciate it if people here would add a link in some insignificant spot so the spiders will find it (but out of the way so the off-topic police won't be offended). We carry useful and important info that was banned from an older more popular site. But new hikers don't even know about us (and automatically go to the other site). So they don't find the info, and therefore go hiking without knowing all that they should. 
|
|
|
|
|
Joined: Jun 2006
Posts: 16,533 Likes: 150
|
|
Joined: Jun 2006
Posts: 16,533 Likes: 150 |
I had a sticky thread somewhere that had a bunch of UserAgents for spiders, not sure where it is offhand and i'm not at my computer
|
|
|
|
|
Joined: Jun 2006
Posts: 684
Addict
|
|
Addict
Joined: Jun 2006
Posts: 684 |
|
|
|
|
|
Joined: Apr 2007
Posts: 3,940 Likes: 1
Former Developer
|
|
Former Developer
Joined: Apr 2007
Posts: 3,940 Likes: 1 |
How does the saying go or at least your governor:
I'll be back! no, it's the age of texting, android, blackberry now days.. he says 'BRB' now  http://rickypoo.com/2010/05/the-textinator/
|
|
|
|
0 members (),
298
guests, and
124
robots. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
|
