|
Joined: Apr 2006
Posts: 148
Member
|
Member
Joined: Apr 2006
Posts: 148 |
Hey Y'all, Happy New Year.
Here's an interesting one. Still running V7.7.5 (old PHP still) with plans to upgrade the server and board software in the future.
About a week ago the board locked up with the "SQL Error: The table 'ubbt_ONLINE' is full". Clearing the cache brought the board back up, but it soon crashed again with the same error. A quick search here in this forum lead me to the thread about this error and the solution to change this data table entry from Memory to MyISAM (disk). This is the default setting in V8, not so in V7. That change solved the issue. I find it interesting that another board admin here experienced the same error on their forum about the same time and reported it here in the Version 7/8 Support forum.
While dealing with this issue I took a closer look at my Who's Online list. I suspected there were far more bots crawling my forum than were listed (tons of guests, few robots listed). I decided to look for an updated robot / crawler list (which I hadn't updated in many many years). Over on the UBBDev message board, I found the newest version of the robots list was from 2022. I installed that, but it did not drastically reduce the number of 'Guests'. So I decided to create an updated list, using the resources listed on the UBBDev thread to identify all known robots / crawlers as of today. I uploaded my updated version of the robots / crawlers text file to that UBBDev thread, and installed it on my forum. Again, it helped expand the robots listed on Who's Online, but I I'm still getting inundated with 'guest users'. As an example, with a 10 minute view period set for Who's Online, I'm seeing almost 500 guests, and over 60 robots.
Here's the interesting part.... Looking at the Guest Users, I see multiple instances from different IP address (all with a standard user type user agent), simultaneously reading a thread from 14 years ago. Sometimes 5 or more 'Guests' are reading the same thread from different IPs. These are obviously not normal human users / guests. Each instance carries a different browser type user-agent, and none look like robots.
Does anyone have any insight into what's going on here?
My forum just past 25 years old... and like most, the online activity has dwindled over the years in favor of other social media outlets. But we have a small loyal following and 25 years of archived technical information that can't be duplicated anywhere else. Even so, it's interesting to see almost 500 'visits' in a 10 minute period, most not identified as a web crawler / robot.
I'm curious if anyone else is experiencing inundation of 'Guests'.
Thanks all.
PaulC MonteCarloSS.com
Stress the system until it breaks. Hey.. it works for Spacecraft.. why not here? UBB since 1999: MonteCarloSS.com
|
|
|
|
Joined: Jun 2006
Posts: 16,375 Likes: 129
|
Joined: Jun 2006
Posts: 16,375 Likes: 129 |
Pretty common everywhere I access... UBBCentral: 2 members, 167 guests, and 162 robots. UBBDev: 2 Members, 380 Guests, and 587 Robots AGF: 3 Members, 2152 Guests, 249 Robots
UBBDev and AGF has the guest timeframe set to 720 minutes (12 hours)
|
1 member likes this:
Z65Paul |
|
|
|
Joined: Apr 2004
Posts: 1,989 Likes: 161
|
Joined: Apr 2004
Posts: 1,989 Likes: 161 |
I see a lot of bots as well. They mostly look like deep crawlers, looking to cache data. Possibly for additional AI training (?) since I have a niche topic site, also with information progressing back about 25 years. About 25% of my crawler/bot activity lately has been just from Facebook's "Meta Web Crawler." The rest with similar IP addresses, i assume are also either content bots or users on VPNs, which are attempting to pre-fetch data by downloading everything attached to the page they are visiting.
|
2 members like this:
Gizmo, Z65Paul |
|
|
|
Joined: Apr 2006
Posts: 148
Member
|
Member
Joined: Apr 2006
Posts: 148 |
Thanks for the input guys. It's just getting worse. I ending up backing the Who's Online window from 10 down to 5 minutes just to keep the viewable list manageable... But even that is no longer tenable. A little bit ago, for a 5 minute period, I had just shy of 1000 guests. This on a near dormant forum. CPU usage is creeping up and I'm starting to get resource warnings.
I just took the drastic step of changing Guest permissions to not be able to read threads. This immediately eliminated the inundation.. but it's obviously not ideal as search engine visibility is now nill. This probably won't be a permanent change.
The Bot-scape has obviously ramped up dramatically in recent months. I wish there was a way to curtail it. Yes, I know about the robots.txt protocol, but you can't place blocks / rules in there if you don't know who the bots are. These all show up as normal browser user-agents. I suspect many disregard the robots.txt rules anyway.
Exasperated.
Stress the system until it breaks. Hey.. it works for Spacecraft.. why not here? UBB since 1999: MonteCarloSS.com
|
|
|
|
Joined: Jun 2006
Posts: 16,375 Likes: 129
|
Joined: Jun 2006
Posts: 16,375 Likes: 129 |
Are you being bombed by a foreign entity? I was being spammed by users in Singapore so I blocked the country and the malicious traffic stopped.
This was monitored on my CDN Cloudflare by finding problem traffic on Analytics -> Security -> Threats by Country then setting a custom rule to block them on Securty -> WAF -> Custom Rules
|
|
|
|
Joined: Apr 2006
Posts: 148
Member
|
Member
Joined: Apr 2006
Posts: 148 |
That's a good question. I hadn't noticed any foreign IPs in the few that I checked from the Who's Online list. I just dove into Welalizer and found these stats so far for January. The vast majority of hits are listed as Unresolved/Unknown as far as Country of origin. In the second list showing prevalence of sites hitting the board... by way of comparison, looking back at previous months, the average site hits for a total month is between 60,000 and 80,000. So far as of January 19th, we're seeing just shy of 400,000. There was an obvious jump in activity starting on January 6th.. Normal daily sites averaged about 5000, then after the 6th, the average jumped up to 55,000, with peak dailies of 13K and 14K on the 7th and 8th of January.
Breaking down the top 5 sites in that list: 1) 216.244.66.233 is Wowrack.com out of Washington State 2) 66.249.65.45 is Google of of Dallas 3 - 9) 38.180.91.112 and all of the 45.137.213.* are 3NT Solutions out of Dallas (this may be one of the culprits) 10 - 11) 47.76.* are Alibaba Cloud out of Hong Kong
I just IP blocked the Wowrack, 3NT Solutions and Alibaba Cloud IPs and some other foreign IPs further down the list. I'll open Guests back up to reading the board and see if this helps.
Top 16 of 16 Total Countries # Hits Files KBytes Country 1 2982462 92.92% 2772083 92.76% 45724596 94.71% Unresolved/Unknown 2 211109 6.58% 207762 6.95% 2340566 4.85% Commercial (com) 3 13418 0.42% 13172 0.44% 192604 0.40% Network (net) 4 1676 0.05% 1464 0.05% 14180 0.03% European Union 5 548 0.02% 68 0.00% 1339 0.00% Czech Republic 6 173 0.01% 142 0.00% 970 0.00% Non-Profit (org) 7 47 0.00% 39 0.00% 3475 0.01% Germany 8 34 0.00% 33 0.00% 301 0.00% Russian Federation 9 31 0.00% 31 0.00% 902 0.00% US Government (gov) 10 26 0.00% 12 0.00% 198 0.00% British Indian Ocean Territory 11 23 0.00% 23 0.00% 802 0.00% Canada 12 21 0.00% 13 0.00% 387 0.00% Switzerland 13 4 0.00% 2 0.00% 1 0.00% Denmark 14 4 0.00% 0 0.00% 5 0.00% United Kingdom 15 2 0.00% 0 0.00% 0 0.00% US Military (mil) 16 1 0.00% 0 0.00% 1 0.00% Generic Business (biz)
Top 30 of 398956 Total Sites # Hits Files KBytes Visits Hostname 1 95444 2.97% 94978 3.18% 11695140 24.22% 18 0.05% 216.244.66.233 2 78648 2.45% 73067 2.44% 983736 2.04% 65 0.20% 66.249.65.45 3 48450 1.51% 46735 1.56% 394663 0.82% 131 0.40% 38.180.91.112 4 43082 1.34% 41404 1.39% 352299 0.73% 115 0.35% 45.137.213.83 5 42877 1.34% 41285 1.38% 350827 0.73% 125 0.38% 45.137.213.88 6 42664 1.33% 41112 1.38% 349684 0.72% 118 0.36% 45.137.213.92 7 42617 1.33% 41049 1.37% 346164 0.72% 116 0.35% 45.137.213.93 8 42512 1.32% 40886 1.37% 347470 0.72% 129 0.39% 45.137.213.86 9 41726 1.30% 40129 1.34% 337186 0.70% 122 0.37% 45.137.213.84 10 39427 1.23% 19364 0.65% 116637 0.24% 84 0.26% 47.76.209.138 11 39291 1.22% 19479 0.65% 116111 0.24% 75 0.23% 47.76.99.127 12 32364 1.01% 30772 1.03% 428749 0.89% 24 0.07% 66.249.65.46 13 32033 1.00% 32017 1.07% 4937 0.01% 5 0.02% 47.233.39.116 14 30232 0.94% 27142 0.91% 234925 0.49% 82 0.25% 38.34.183.94 15 28489 0.89% 25716 0.86% 217654 0.45% 79 0.24% 38.145.218.218 16 17768 0.55% 17669 0.59% 28285 0.06% 12 0.04% 108.21.78.84 17 11850 0.37% 11234 0.38% 158145 0.33% 22 0.07% crawl-66-249-65-32.googlebot.com 18 10566 0.33% 10522 0.35% 36493 0.08% 13 0.04% 68.83.255.52 19 7608 0.24% 7607 0.25% 74797 0.15% 0 0.00% 217.113.194.73 20 7560 0.24% 7558 0.25% 74896 0.16% 0 0.00% 217.113.194.80 21 7467 0.23% 7466 0.25% 73021 0.15% 1 0.00% 154.54.249.200 22 7418 0.23% 7281 0.24% 77372 0.16% 0 0.00% 51.222.253.18 23 7401 0.23% 7278 0.24% 76895 0.16% 7 0.02% 51.222.253.16 24 7365 0.23% 7222 0.24% 76593 0.16% 8 0.02% 51.222.253.5 25 7335 0.23% 7333 0.25% 72297 0.15% 0 0.00% 217.113.194.76 26 6907 0.22% 6907 0.23% 67419 0.14% 0 0.00% 217.113.194.81 27 6901 0.22% 6776 0.23% 73221 0.15% 3 0.01% proxy-ca013-ext2.a.ahrefs.com 28 6843 0.21% 6842 0.23% 67669 0.14% 0 0.00% 217.113.194.72 29 6842 0.21% 6841 0.23% 67174 0.14% 0 0.00% 217.113.194.79 30 6799 0.21% 6799 0.23% 65900 0.14% 0 0.00% 217.113.194.75
Stress the system until it breaks. Hey.. it works for Spacecraft.. why not here? UBB since 1999: MonteCarloSS.com
|
|
|
|
Joined: Apr 2006
Posts: 148
Member
|
Member
Joined: Apr 2006
Posts: 148 |
Well that didn't work LOL. Earlier when I limited the Guests viewing permissions, I kept one forum open for them to view.... my MonteCarloSS.com News forum. I looked at my Who's Online list and this is what I see: https://montecarloss.com/Guests.jpgAll those different IPs are now looking at one thread from 2006 in that News forum! It's like a swarm of bees. They all have differing User-agents.. some Windows based, some Linux, some Mac OS. I then opened Guests back up to read threads and the flood gates opened once more, amassing 1200+ 'Guests' in the first 10 minutes. For now.. I'm going to cut Guests off again from reading. Don't know what else to do.
Stress the system until it breaks. Hey.. it works for Spacecraft.. why not here? UBB since 1999: MonteCarloSS.com
|
|
|
2 members (Gizmo, 1 invisible),
198
guests, and
94
robots. |
Key:
Admin,
Global Mod,
Mod
|
|
|
|