Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why does robots.txt not reflect the order of robots.json after daily dark visitors update #48

Open
glyn opened this issue Oct 19, 2024 · 5 comments

Comments

@glyn
Copy link
Contributor

glyn commented Oct 19, 2024

          Me too. Another small mystery is that `robots.txt` is still in the old order and the latest run didn't update it. I think that's a separate issue though!

Originally posted by @glyn in #44 (comment)

@glyn
Copy link
Contributor Author

glyn commented Oct 19, 2024

There was a failure to update robots.txt: https://github.com/ai-robots-txt/ai.robots.txt/actions/runs/11413277042/job/31760570236

@glyn
Copy link
Contributor Author

glyn commented Oct 19, 2024

We are now at the mercy of PHP (rather than Python in the dark visitors processing).

I tried reading the PHP docs for json_decode and dictionaries in PHP, but didn't find anything about ordering.

I used an online runner and the behaviour of json_decode and foreach seemed to respect the ordering of the input JSON, but I guess that might depend on the version of PHP that our runner is using.

@glyn
Copy link
Contributor Author

glyn commented Oct 19, 2024

@newbold Any thoughts on this issue, please?

@glyn
Copy link
Contributor Author

glyn commented Oct 19, 2024

I reran the failed job just to be on the safe side, but it made no difference: robots.txt was not updated.

@glyn
Copy link
Contributor Author

glyn commented Oct 19, 2024

I added some diagnostics and it seems that the output robots.txt is in the correct order, but the result of a rerun does not get picked up as a git change. So the issue may be more to do with the github actions than with the PHP code itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant