Retype build error handling and exit code

[DannyBen]

I am posting this as a discussion and not an issue, to let the team decide if its an issue-worthy topic.

I have encountered two issues that I wish handled differently by retype build:

Exit code for warnings

When running retype build on a site that generates warnings (like a link to a broken URL in one of the files), I believe the exit code should be non zero. This is particularly important in the GitHub Action - it should also fail if there are warnings.

Of course, in case there are some other types of warnings that are truly warnings, then perhaps this should be configurable (build.strict: true or build.fail_on_warning: true).

My rule of thumb is this: If retype build will end up building a site with problems, I want the build to tell me that it failed in the exit code.

Human friendly error

I wanted to test how errors look, so I just changed the config input directory to one that does not exist. The error displayed to the user seems like a backtrace that the developer should care about, not the user. As a user, I expect to see a friendly error (only the message, and perhaps the class of the exception): Input directory does not exist: '/vagrant/sites/jobly/i-broke-this'

[geoffreymcgill]

Thanks for the comments. We will investigate and get back to you if we have any questions.

I do agree that some of the errors need to be improved. There are still some situations where a full stack trace is printed and the retype watch server needs to be restarted. Retype should do a better job of recovering from those errors and printing human friendly errors.

I'm not sure about the exit codes. We'll have to discuss. I do like the strict mode. That could be helpful.

[geoffreymcgill]

A warning would not constitute a failure, from my perspective. Unless, you're in a strict mode, then treating a warning as an error makes sense, plus then failing with an exit code.

[DannyBen]

While I might agree with this statement when running in my own dev environment, consider CI:

The generation of the documentation site, if done automatically in CI/CD, does not have any "tests". So, today, unless there is a totally fatal error - like an invalid retype.json - the site is just built and goes live, with zero quality control.

These warnings - at least some of them, as "unresolved link" - are in fact errors, when you think about the production-quality output. It will generate a site with broken links. In any other CI environment, I would want this to fail - why not here?

This brings an idea to mind: retype verify - test the validity of the site, report each warning nicely, in one line (problem, and in which page), and exit with non zero unless all green.

This command can be executed in the GitHub Actions package prior to building.
Then, the retype build can be left alone, or perhaps even add retype build --verify to verify before build.

See the tests folder in my mdbook site - it explicitly contains a broken links test, spell checker, etc. You already have that built in the CLI - it is a shame not to use it.

[geoffreymcgill]

Thanks for the feedback. We are going to review the warning and error processing and come up with a plan on how we can better handle the reporting.

The quality of local link resolution and reporting will be improving in v1.9.0 with the introduction of the page dependency graph, so that will at least help us find the potentially broken links.

External link verification is certainly something we've been discussing, although requires external calls and would likely have to be run in a separate process, such as your retype verify suggestion, or async from build.

[DannyBen]

External link verification is certainly something we've been discussing

I don't expect a site building tool to do that for me.

[geoffreymcgill]

Another follow-up comment, an unresolved link does not necessarily mean it is (or will be) broken.

For instance, the Retype generated site could be deployed into to a /docs/ sub-folder, such as examples.com/docs/. Within the .md content, there could be relative links to content within the main site, such as a [Products](../products) link from within /docs/index.md. Currently, I believe that link would be returned as unresolved by Retype and a warning would be thrown.

Now of course, Retype could probably work out that your ../products link was pointing to an external path (outside of the input dir), at least external to the Retype project, and handle the reporting of that unresolved link differently than local links.

Over time I do believe we will be able to nicely refine the resolution accuracy and reporting. It will get better and handle more scenarios.