[BUG] Vale workflow randomly fail after check starts

[peterzhuamazon]

This is the vale github action workflow file we use: https://github.com/opensearch-project/documentation-website/blob/077f613230f332b86d880e929c7d8a52786e90c6/.github/workflows/vale.yml

Sometimes the checks are randomly failing like this:
https://github.com/opensearch-project/documentation-website/actions/runs/8350082188/job/22855709741?pr=6720

Run errata-ai/vale-action@reviewdog
/usr/bin/pip install docutils
Defaulting to user installation because normal site-packages is not writeable
Collecting docutils
  Downloading docutils-0.20.1-py3-none-any.whl (572 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.7/572.7 KB 14.0 MB/s eta 0:00:00
Installing collected packages: docutils
Successfully installed docutils-0.20.1
/usr/bin/gem install asciidoctor --user-install
WARNING:  You don't have /home/runner/.local/share/gem/ruby/3.0.0/bin in your PATH,
	  gem executables will not run.
Successfully installed asciidoctor-2.0.22
Parsing documentation for asciidoctor-2.0.22
Installing ri documentation for asciidoctor-2.0.22
Done installing documentation for asciidoctor after 2 seconds
1 gem installed
Installing Vale version '2.28.0' ...
/usr/bin/tar xz --overwrite --warning=no-unknown-keyword --overwrite -C /home/runner -f /home/runner/work/_temp/f6efbd83-2d0f-490a-8275-2360e3acf6c9
Installed version '2.28.0' into '/home/runner/vale'.
Installing ReviewDog version '0.17.0' ...
/usr/bin/tar xz --overwrite --warning=no-unknown-keyword --overwrite -C /home/runner -f /home/runner/work/_temp/6128c418-bd55-47dd-8fb0-0b64114771ef
Installed reviewdog from 'https://github.com/reviewdog/reviewdog/releases/download/v0.17.0/reviewdog_0.17.0_Linux_x86_64.tar.gz' into '/home/runner/reviewdog'.
/home/runner/vale --no-exit sync
Running vale with reviewdog 🐶 ...
Error: reviewdog exited with status code: 1

It was supposed to be running and ignore errors, but seems like the error is not caused by the action itself, need investigation.

Thanks.

[peterzhuamazon]

More info:
Initially reported by @kolchfa-aws there are more of these examples here:

https://github.com/opensearch-project/documentation-website/blob/main/.github/workflows/vale.yml#L19
(supposed to return 0 even if errors are found https://github.com/errata-ai/vale-action#fail_on_error-default-false)

https://github.com/opensearch-project/documentation-website/blob/main/.github/workflows/vale.yml#L22
(supposed to return 0 even if errors are found https://vale.sh/manual/no-exit/)

[peterzhuamazon]

More info:

2024-03-19T21:05:44.0512310Z [command]/home/runner/reviewdog -f=rdjsonl -name=vale -reporter=github-pr-check -fail-on-error=false -filter-mode=added -level=info
2024-03-19T21:05:44.0512673Z reviewdog: This GitHub token doesn't have write permission of Review API [1], 
2024-03-19T21:05:44.0512994Z so reviewdog will report results via logging command [2] and create annotations similar to
2024-03-19T21:05:44.0513149Z github-pr-check reporter as a fallback.
2024-03-19T21:05:44.0513638Z [1]: https://docs.github.com/en/actions/reference/events-that-trigger-workflows#pull_request_target, 
2024-03-19T21:05:44.0514354Z [2]: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/development-tools-for-github-actions#logging-commands
2024-03-19T21:05:44.0514502Z reviewdog: found at least one result in diff
2024-03-19T21:05:44.0514814Z reviewdog: Reporting results for "vale"
2024-03-19T21:05:44.0543019Z ##[error][vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: systemd. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: systemd. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_install-and-configure/install-opensearch/debian.md", "range": {"start": {"line": 555, "column": 102}}}, "severity": "ERROR"}
2024-03-19T21:05:44.0554383Z ##[error][vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: systemd. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: systemd. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_install-and-configure/install-dashboards/debian.md", "range": {"start": {"line": 166, "column": 124}}}, "severity": "ERROR"}
2024-03-19T21:05:44.0555203Z ##[endgroup]
2024-03-19T21:05:44.0555808Z ##[error]reviewdog exited with status code: 1

[peterzhuamazon]

When green:

2024-03-18T22:53:16.4176163Z [command]/home/runner/reviewdog -f=rdjsonl -name=vale -reporter=github-pr-check -fail-on-error=false -filter-mode=added -level=info
2024-03-18T22:53:16.4177386Z reviewdog: This GitHub token doesn't have write permission of Review API [1], 
2024-03-18T22:53:16.4178211Z so reviewdog will report results via logging command [2] and create annotations similar to
2024-03-18T22:53:16.4178892Z github-pr-check reporter as a fallback.
2024-03-18T22:53:16.4179800Z [1]: https://docs.github.com/en/actions/reference/events-that-trigger-workflows#pull_request_target, 
2024-03-18T22:53:16.4181174Z [2]: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/development-tools-for-github-actions#logging-commands
2024-03-18T22:53:16.4182167Z reviewdog: Reporting results for "vale"
2024-03-18T22:53:16.4182761Z reviewdog: No results found for "vale". 2548 results found outside diff.

[peterzhuamazon]

Seems like legit failure as vale indeed find errors.
More like spelling.

[kolchfa-aws]

The Vale GitHub action is unstable. It always comments on the PR with the errors, but it randomly fails some PRs and passes others, even though we have specified for it not to fail in 2 places (--no-exit flag and fail_on_error). We're OK with setting it to pass by default because it still leaves comments and we can direct contributors to those comments when reviewing the PR. Thanks!

[peterzhuamazon]

This bug should be fixed by Jan 16, 2023 or around, but for version 2.28.0 in Jul 4, 2023 it seems still not fixed, thus it seems better to mute the return status:

• Action fails even if fail_on_error is false errata-ai/vale-action#89

At the moment we will just let reviewers inform the contributors on style changes based on the code comments.