1.0.9

• add support for PHP 8.0 + move to GitHub actions (#27)

1.0.7

• fix find robots meta tag line if minified code (#23)

1.0.6

• fix headers checking (nofollow, noindex) for custom userAgent (#21)

1.0.5

• improvements around handling of wildcards, end-of-string, query string

1.0.4

• improve readability

1.0.3

• fix parsing robotstxt urls with keywords (#14)

1.0.2

• make robots.txt check case insensitive

1.0.1

• prevent exception if the domain has no robots.txt

1.0.0

• initial release

0.1.3

Even more header parsing improvements.