If they respect the robots.txt standard they will check the robots.txt prior to any crawl.

There is also another method of blocking bots via your .htaccess, some reading on multiple methods:
https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/ (block by user agent)
https://www.ip2location.com/free/visitor-blocker (IP2Location Firewall List by Country)

I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!