UBB.threads PHP Forum Software Community

Active Threads | Active Posts | Unanswered Today | Since Yesterday | This Week

Re: Bots Gizmo 04/14/2024 11:12 AM

Well, both methods display their .htaccess input code, the first would be (Blocking by user agent):

Code

# Block aggressive Chinese crawlers/scrapers/bots
# https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/
Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ahrefsbot|Baiduspider|BLEXBot|Bytespider|BuckyOHare|dotbot|exabot|gigabot|Goodzer|gsa-crawler|Kinza|LieBaoFast|LinkFeatureBot|MauiBot|Mb2345Browser|MicroMessenger|mj12bot|musobot|rogerbot|rushBot|semrushbot|serpstatbot|sitebot|Sogou|SputnikBot|VelenPublicWebCrawler|WBSearchBot|WPSpider|zh-CN|zh_CN [NC]
RewriteRule ^ - [F,L]

The second method (IP2Location Firewall List by Country) you'd take the .htaccess coding from their list for an Apache2 server and insert it as (the list is huge, I'm only including a couple of lines):

Code

<Limit GET HEAD POST>
order allow,deny
allow from all
deny from 1.0.1.0/24
deny from 1.0.2.0/23
deny from 1.0.8.0/21
deny from 1.0.32.0/19
deny from 1.1.0.0/24
deny from 1.1.2.0/23
deny from 1.1.4.0/22
deny from 1.1.8.0/21
deny from 1.1.16.0/20
deny from 1.1.32.0/19
...
deny from 223.252.168.0/21
deny from 223.252.177.0/24
deny from 223.252.178.0/23
deny from 223.252.180.0/22
deny from 223.252.184.0/21
deny from 223.252.192.0/18
deny from 223.254.0.0/16
deny from 223.255.0.0/17
deny from 223.255.236.0/22
deny from 223.255.252.0/23
</Limit>

6 138 Read More

How Do I?

Re: Bots Outdoorking 04/14/2024 8:59 AM

Thanks Gizmo,

I would really like to know what code that I need to put into the htaccess file because it appears that what I have placed in the robot.txt file is not working unless I have to wait longer.

6 138 Read More

How Do I?

Re: Bots Gizmo 04/14/2024 6:47 AM

If they respect the robots.txt standard they will check the robots.txt prior to any crawl.

There is also another method of blocking bots via your .htaccess, some reading on multiple methods:
https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/ (block by user agent)
https://www.ip2location.com/free/visitor-blocker (IP2Location Firewall List by Country)

6 138 Read More

How Do I?

Re: Bots Outdoorking 04/14/2024 3:13 AM

Originally Posted by Ruben

If the bot honers it you can create a robots.txt file in your root folder and exclude by agent, ip etc
https://en.wikipedia.org/wiki/Robots.txt
https://ubbdev.com/wiki/view/7/ubb-sitemaps.html

If they do not honor the robots file you can ban them via the htaccess file or the UBB control panel using ip.

If it becomes a ddos attack you need to use something like cloudflare or ask your host about ddos protection.

Thanks for the reply. This is what I placed in the robot text file:

User-agent: Baiduspider
User-agent: 360Spider
User-agent: Yisouspider
User-agent: PetalBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sogou inst spider
Disallow: /

Does this look correct and how long would it take before it starts to work?

6 138 Read More

How Do I?

Re: Bots Gizmo 04/13/2024 10:48 PM

FYI, if the bot visits the login page the Who's Online page will say that it's trying to login, since that's the wording for that system.

6 138 Read More

How Do I?

Re: Bots Ruben 04/13/2024 9:43 PM

6 138 Read More

How Do I?

Bots Outdoorking 04/13/2024 9:08 PM

Hi all,

I have noticed this in the Whos online Bytespider (bad_bot) has over 100 entries.

Another one is MJ12bot which is trying to login.

Should I do something about these two and if so what code do I need to run and where please.

I have noticed that if you have a lot it can cause issues slowing the forum down.

Any assistance would be appreciated.

6 138 Read More

UBB.threads PHP Forum Software Community Forums Active Threads This Week