OpenAI clarified in its crawler help documentation that the version number of the user-agent string may be updated and that it may use a user-agent string with an additional robots.txt marker. The example given in the doc added "the version number may change." OpenAI also added a section stating: "When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker. The marker helps site owners more easily distinguish requests for the robots.txt file from requests for other resources, especially when their logs do not include paths: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot."