I have a thing installed that detects and new web crawler bots indexing the content here:
Potential New Bots Detected
which is this lad:
Should I block it?
Potential New Bots Detected
- mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; gptbot/1.0; +OpenAI Platform)
which is this lad:
OpenAI Platform
Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.
platform.openai.com
Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety. Below, we also share how to disallow GPTBot from accessing your site.
Should I block it?