Previous Thread
Next Thread
Print Thread
Hop To
#266017 04/13/2024 5:08 PM
Joined: Jun 2006
Posts: 319
Enthusiast
Enthusiast
Joined: Jun 2006
Posts: 319
Hi all,

I have noticed this in the Whos online Bytespider (bad_bot) has over 100 entries.

Another one is MJ12bot which is trying to login.

Should I do something about these two and if so what code do I need to run and where please.

I have noticed that if you have a lot it can cause issues slowing the forum down.

Any assistance would be appreciated.

Outdoorking #266018 04/13/2024 5:43 PM
Joined: Dec 2003
Posts: 6,566
Likes: 78
Joined: Dec 2003
Posts: 6,566
Likes: 78
If the bot honers it you can create a robots.txt file in your root folder and exclude by agent, ip etc
https://en.wikipedia.org/wiki/Robots.txt
https://ubbdev.com/wiki/view/7/ubb-sitemaps.html

If they do not honor the robots file you can ban them via the htaccess file or the UBB control panel using ip.

If it becomes a ddos attack you need to use something like cloudflare or ask your host about ddos protection.


Blue Man Group
There is no such thing as stupid questions. Just stupid answers
Outdoorking #266019 04/13/2024 6:48 PM
Joined: Jun 2006
Posts: 16,301
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,301
Likes: 116
FYI, if the bot visits the login page the Who's Online page will say that it's trying to login, since that's the wording for that system.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Ruben #266020 04/13/2024 11:13 PM
Joined: Jun 2006
Posts: 319
Enthusiast
Enthusiast
Joined: Jun 2006
Posts: 319
Originally Posted by Ruben
If the bot honers it you can create a robots.txt file in your root folder and exclude by agent, ip etc
https://en.wikipedia.org/wiki/Robots.txt
https://ubbdev.com/wiki/view/7/ubb-sitemaps.html

If they do not honor the robots file you can ban them via the htaccess file or the UBB control panel using ip.

If it becomes a ddos attack you need to use something like cloudflare or ask your host about ddos protection.

Thanks for the reply. This is what I placed in the robot text file:

User-agent: Baiduspider
User-agent: 360Spider
User-agent: Yisouspider
User-agent: PetalBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sogou inst spider
Disallow: /

Does this look correct and how long would it take before it starts to work?

Last edited by Outdoorking; 04/13/2024 11:14 PM.
Outdoorking #266021 04/14/2024 2:47 AM
Joined: Jun 2006
Posts: 16,301
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,301
Likes: 116
If they respect the robots.txt standard they will check the robots.txt prior to any crawl.

There is also another method of blocking bots via your .htaccess, some reading on multiple methods:
https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/ (block by user agent)
https://www.ip2location.com/free/visitor-blocker (IP2Location Firewall List by Country)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #266022 04/14/2024 4:59 AM
Joined: Jun 2006
Posts: 319
Enthusiast
Enthusiast
Joined: Jun 2006
Posts: 319
Thanks Gizmo,

I would really like to know what code that I need to put into the htaccess file because it appears that what I have placed in the robot.txt file is not working unless I have to wait longer.

Outdoorking #266023 04/14/2024 7:12 AM
Joined: Jun 2006
Posts: 16,301
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,301
Likes: 116
Well, both methods display their .htaccess input code, the first would be (Blocking by user agent):
Code
# Block aggressive Chinese crawlers/scrapers/bots
# https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/
Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ahrefsbot|Baiduspider|BLEXBot|Bytespider|BuckyOHare|dotbot|exabot|gigabot|Goodzer|gsa-crawler|Kinza|LieBaoFast|LinkFeatureBot|MauiBot|Mb2345Browser|MicroMessenger|mj12bot|musobot|rogerbot|rushBot|semrushbot|serpstatbot|sitebot|Sogou|SputnikBot|VelenPublicWebCrawler|WBSearchBot|WPSpider|zh-CN|zh_CN [NC]
RewriteRule ^ - [F,L]


The second method (IP2Location Firewall List by Country) you'd take the .htaccess coding from their list for an Apache2 server and insert it as (the list is huge, I'm only including a couple of lines):
Code
<Limit GET HEAD POST>
order allow,deny
allow from all
deny from 1.0.1.0/24
deny from 1.0.2.0/23
deny from 1.0.8.0/21
deny from 1.0.32.0/19
deny from 1.1.0.0/24
deny from 1.1.2.0/23
deny from 1.1.4.0/22
deny from 1.1.8.0/21
deny from 1.1.16.0/20
deny from 1.1.32.0/19
...
deny from 223.252.168.0/21
deny from 223.252.177.0/24
deny from 223.252.178.0/23
deny from 223.252.180.0/22
deny from 223.252.184.0/21
deny from 223.252.192.0/18
deny from 223.254.0.0/16
deny from 223.255.0.0/17
deny from 223.255.236.0/22
deny from 223.255.252.0/23
</Limit>


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
Version 7.7.5 Images suddenly not displaying
by Stovebolt - 05/04/2024 11:19 AM
Do I need to rebuild my database?
by Baldeagle - 04/07/2024 2:58 AM
Who's Online Now
1 members (Ruben), 328 guests, and 276 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20240506)