According to the web they both honor the robots.txt file.
But they do state it may take a few days to honor.
If you google each bot there is reference on how to do so.
Also as I recall you are using cloudflare. which can be done in the firewall rules.
I love cloudflare.
