Shiny prerendered dependencies are sometimes written twice into prerendered HTML

[gadenbuie]

learnr has long-suffered from a hard-to-replicate bug that results in an error like this:

Error: parse error: trailing garbage

          ":{},"value":["0.10.1"]}]}]} {"type":"list","attributes":{},

                     (right here) ------^

Execution halted

I've finally tracked down the core mechanism, but I don't know the root cause. When a shinyrmd document is pre-rendered into HTML, it writes the dependencies used in the app into the prerendered HTML so that it can be used at runtime:

rmarkdown/R/render.R

Lines 857 to 862 in 1d739bd

	# write shiny_prerendered_dependencies if we have them
	if (is_shiny_prerendered(runtime)) {
	shiny_prerendered_append_dependencies(input,
	shiny_prerendered_dependencies,
	files_dir,
	output_dir)

shiny_prerendered_append_dependencies() writes two JSON-serialized objects into <script> tags with data-context attributes "dependencies" and "execution_dependencies".

rmarkdown/R/shiny_prerendered.R

Lines 370 to 380 in 1d739bd

	# write deps to connection
	dependencies_json <- jsonlite::serializeJSON(dependencies, pretty = FALSE)
	shiny_prerendered_append_context(con, "dependencies", dependencies_json)
	# write r major version and execution package dependencies
	execution_json <- jsonlite::serializeJSON(
	# visibly display what is being stored
	shiny_prerendered_dependencies["packages"],
	pretty = FALSE
	)
	shiny_prerendered_append_context(con, "execution_dependencies", execution_json)

In rstudio/learnr#597 (comment) I was sent copies of tutorial files that exhibited this behavior, and they contain duplicated dependency chunks. Importantly, the entire group of dependencies + execution dependencies are duplicated, rather than each one being written twice in place. Here's roughly what the prerendered html looks like:

 <!--html_preserve-->
<script type="application/shiny-prerendered" data-context="dependencies">
{"type":"list","attributes":{},"value":[... "value":["0.10.1"]}]}]}
</script>
<!--/html_preserve-->
<!--html_preserve-->
<script type="application/shiny-prerendered" data-context="execution_dependencies">
{"type":"list","attributes":{"names": ...
</script>
<!--/html_preserve-->
 <!--html_preserve-->
<script type="application/shiny-prerendered" data-context="dependencies">
{"type":"list","attributes":{},"value":[... "value":["0.10.1"]}]}]}
</script>
<!--/html_preserve-->
<!--html_preserve-->
<script type="application/shiny-prerendered" data-context="execution_dependencies">
{"type":"list","attributes":{"names": ...
</script>
<!--/html_preserve-->

When starting up the app, shiny_prerendered_html() extracts the "dependencies" and "execution_dependencies" contexts

rmarkdown/R/shiny_prerendered.R

Lines 157 to 159 in 1d739bd

	html_lines <- read_utf8(rendered_html)
	dependencies_json <- shiny_prerendered_extract_context(html_lines, "dependencies")
	dependencies <- jsonlite::unserializeJSON(dependencies_json)

but strongly assumes that there is exactly a single string returned by shiny_prerendered_extract_context() when it passes the extracted context to jsonlite::unserializedJSON(), which does not like the invalid JSON.

obj_json <- jsonlite::serializeJSON(list(a = 1))
jsonlite::unserializeJSON(c(obj_json, obj_json))
#> Error: parse error: trailing garbage
#>           ttributes":{},"value":[1]}]} {"type":"list","attributes":{"n
#>                      (right here) ------^

I'd like to know why shiny_prerendered_append_dependencies() is called twice during pre-render, but I also think we can reasonably avoid this issue by checking the "dependencies" context contents before handing them to unserializedJSON(). I'll submit a PR with this change shortly.

[gadenbuie]

I have a suspicion that there are somehow two processes running at the same time to prerender the source document and are inadvertently writing into the same file. I've uncovered a reproducible example that produces the expected error (here shown before the fix in #2500).

render_bg <- function(path) {
  callr::r_bg(
    function(path) {
      rmarkdown::render(path)
    },
    args = list(path = path)
  )
}

# Remove prerendered HTML and source if they exist
unlink("issue.html")
unlink("issue.Rmd")

writeLines('---
title: "issue"
output: html_document
runtime: shiny_prerendered
---

```{r}
textInput("text", "Text", "Text")
renderText(input$text)
```', "issue.Rmd"
)

# Render the Rmd twice at basically the same time
r1 <- render_bg("issue.Rmd")
r2 <- render_bg("issue.Rmd")
r2$wait()

When you try to run the document, it will have been corrupted by simultaneous writes into the .html file.

rmarkdown::run("issue.Rmd")
#> Error: parse error: trailing garbage
#>           ":{},"value":["2.23.3"]}]}]} {"type":"list","attributes":{},
#>                      (right here) ------^

After applying the fix in #2500, we still get errors due to failures in other places in the file.

rmarkdown::run("issue.Rmd")
#> Error: lexical error: invalid char in json text.
#>           s":{},"value":["jquerylib"]} <!--html_preserve--> {"type":"l
#>                      (right here) ------^

[cderv]

Good find !

I don't know if we should do something about this in rmarkdown (at least as a bug fix). I don't think we do anything against this not even for render().

This is the responsability to who or what is calling rmarkdown::run() or rmarkdown::render() to do that in a safe context for concurrency.

Related issue on that concurrency topic:

• [FR] *.knit.md should have a unique name to allow parallelization #2454
• parallel render()'s with different output_file's overwrite each other #499
• Parallel creation of parameterized reports fails due to file conflicts quarto-dev/quarto-cli#4730
• Parallel Computing - using one .qmd to render n output files - cannot open the connection quarto-dev/quarto-cli#5861

[gadenbuie]

In this case, my reprex is a hypothetical example that's at least consistent with the real-world scenario. I've heard reports of this kind of problem for learnr users on Connect and shinyapps.io as well as Posit Cloud. Cloud is surprising to me because in theory it should be equivalent to a local installation, where this problem hasn't been reported really.

Related to the concurrency option, I understand that there are many reasons that make writing unique files difficult. That said, if we're using a mechanism like writeLines(text, file_connection), I do think rmarkdown/knitr should write into a temporary uniquely-named file that's moved into place when the contents are finished. This way, even if there are race conditions, then at least the final intermediate file is the atomically the complete and correct winner of the race.

[cderv]

Related to the concurrency option, I understand that there are many reasons that make writing unique files difficult. That said, if we're using a mechanism like writeLines(text, file_connection), I do think rmarkdown/knitr should write into a temporary uniquely-named file that's moved into place when the contents are finished. This way, even if there are race conditions, then at least the final intermediate file is the atomically the complete and correct winner of the race.

Yes makes sense

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.