NeuralTextBot obeys robots.txt file on your website. So you can fully control it on your website if you need.
If for some reason you want to prevent NeuralTextBot from visiting your site, put the two following lines into the robots.txt file on your server:
|User-agent: NeuralTextBot |
Please also note that if your robots.txt returns a 404 status code, the crawler will assume that no robots.txt rule should be respected.
You can test any URL by visiting our Robot Page .
Read more about robots.txt at www.robotstxt.org.
If you think that NeuralTextBot is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team email@example.com .