Skip to content

Releases: ai-robots-txt/ai.robots.txt

v1.18: adds ISSCyberRiskCrawler

01 Oct 00:38
Compare
Choose a tag to compare

Adds ISSCyberRiskCrawler. Thanks to Brian at Skewray Research for calling this one out!

v1.17: add Sidetrade indexer bot + other updates

28 Sep 21:03
Compare
Choose a tag to compare

What's Changed

  • Add Sidetrade indexer bot (Thank you to Brian at Skewray Research for calling this one out.)
  • Add instructions for AI bot blocking on Vercel by @cramforce in #36
  • Update README.md by @cityrolr in #41
  • feat: make CCBot entry more accurate by @wumpus in #42
  • Add an additional resource - README.md by @urvish-p80 in #38

New Contributors

Full Changelog: v1.16...v1.17

v1.16: adds iaskspider/2.0

07 Sep 02:06
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

What's Changed

Full Changelog: v1.15...v1.16

v1.15: adds AI2Bot

29 Aug 03:09
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.14...v1.15

v1.14: adds Webzio-Extended

27 Aug 22:42
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.13...v1.14

v1.13: corrects agent name

04 Aug 21:54
Compare
Choose a tag to compare

What's Changed

  • Fix Imagesift user agent by @jsheard in #27
  • Add Cloudflares first-party scraper blocking to FAQ by @jsheard in #28

New Contributors

Full Changelog: v1.12...v1.13

v1.12: adds ICC-Crawler

04 Aug 19:30
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.11...v1.12

v1.11: adds PetalBot and facebookexternalhit

01 Aug 14:55
6c596a5
Compare
Choose a tag to compare

What's Changed

Full Changelog: v1.10...v1.11

v1.10: adds Scrapy

30 Jul 17:29
fa7b64a
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.9...v1.10

v1.9: adds Timpibot

29 Jul 19:38
55b4505
Compare
Choose a tag to compare