Skip to content

Releases: ai-robots-txt/ai.robots.txt

v1.8: adds VelenPublicWebCrawler

29 Jul 19:14
d49e860
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.7...v1.8

v1.7: adds Meta-ExternalAgent

29 Jul 15:27
6e32355
Compare
Choose a tag to compare

v1.6: adds OAI-SearchBot to robots.txt

26 Jul 16:07
2972926
Compare
Choose a tag to compare
chore: add OAI-SearchBot

v1.5: drops AdsBot-Google; adds GoogleOther-Image/Video

16 Jul 19:08
3692d66
Compare
Choose a tag to compare

v1.4: add ImagesiftBot, remove peer39

10 Jul 00:45
0ca6bce
Compare
Choose a tag to compare

What's Changed

  • Adds ImagesiftBot (thanks @corbindavenport!)
  • Removes peer39 bots as they are not used to train/related to AI
  • Add reference by @glyn in #18

Full Changelog: v1.3...v1.4

chore: updates to bots table, readme

22 Jun 19:44
70e8985
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.2...v1.3

chore: tidy up bots

20 Jun 15:13
89de2d2
Compare
Choose a tag to compare

What's Changed

  • Narrows the scope of bots blocked to strictly include those relating to AI
    • This sees some bots removed that should likely still be blocked, but for different reasons.
  • Remove potential false positives by @nisbet-hubbard in #16

v1.1: updates bots list to include Applebot-Extended in favor of Applebot

15 Jun 16:31
429336d
Compare
Choose a tag to compare
  • Updates bot listings to include Applebot-Extended in favor of Applebot at the former is for AI training, the latter is used for search.

v1: First proper release

13 Jun 02:04
b5d4d5b
Compare
Choose a tag to compare

Let's consider this the first proper release, with additional updates to be formally marked with a release.

Full Changelog: https://github.com/ai-robots-txt/ai.robots.txt/commits/v1