Skip to content

Commit

Permalink
add reference to pipeline article to vignette
Browse files Browse the repository at this point in the history
  • Loading branch information
thomaszwagerman committed Oct 23, 2024
1 parent 749627f commit bdfb3d6
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 11 deletions.
18 changes: 10 additions & 8 deletions vignettes/articles/butterfly_in_pipeline.Rmd
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 'Using butterfly in a data processing pipeline'
title: 'Using butterfly in an operational data pipeline'
---

```{r, include = FALSE}
Expand All @@ -15,7 +15,7 @@ font-size: 0.6em;
}
</style>

This article is a **simplified** demonstration of a real data processing pipeline we implemented at the British Antarctic Survey called asli-pipeline. You can inspect the full source code of this pipeline in the repository: [asli-pipeline repository](https://github.com/antarctica/asli-pipeline) (Zwagerman & Wilby).
This article is a **simplified** demonstration of a real operational data pipeline we are implementing at the British Antarctic Survey called asli-pipeline. You can inspect the full source code of this pipeline in the repository: [asli-pipeline repository](https://github.com/antarctica/asli-pipeline) (Zwagerman & Wilby).

This package was originally developed to deal with [ERA5](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=documentation)'s initial release data, **ERA5T**. ERA5T data for a month is overwritten with the final ERA5 data two months after the month in question.

Expand All @@ -25,17 +25,17 @@ In a pipeline that generates ERA5-derived data, and continually updates and **pu

## Pipeline overview

Consider a classic input/output (I/O) data processing pipeline where we read in data from an external source, perform some sort of calculation to it, and transfer the output to a different location.
Consider a classic input/output (I/O) data pipeline where we read in data from an external source, perform some sort of calculation to it, and transfer the output to a different location.

```{r simple_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A simple diagram showing the steps in a generic data processing pipeline."}
```{r simple_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A simple diagram showing the steps in a generic data pipeline."}
knitr::include_graphics("img/simple_diagram.png")
```

We use a pipeline to calculate the **'Amundsen Sea Low Index'**, or ASLI. The Amundsen Seas Low (ASL) is a highly dynamic and mobile climatological low pressure system located in the Pacific sector of the Southern Ocean. If you are interested in ASLI, and why these values are significant for environmental forecasting, please refer to [Hosking et al. (2016)](https://doi.org/10.1002/2015GL067143).

In our case, we run this pipeline on a monthly basis:

```{r bas_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A diagram showing the steps in our British Antarctic Survey data processing pipeline to calculate and publish the Amundsen Sea Low Index dataset."}
```{r bas_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A diagram showing the steps in our British Antarctic Survey data pipeline to calculate and publish the Amundsen Sea Low Index dataset."}
knitr::include_graphics("img/bas_example.png")
```

Expand All @@ -56,7 +56,7 @@ Keeping up-to-date with the [Climate Data Store's Forum](https://forum.ecmwf.int

To maintain the integrity of our published dataset, we need to impose robust checks to ensure our new results match our published data, where we expect it to.

```{r full_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A diagram showing the steps in our British Antarctic Survey data processing pipeline to calculate and publish the Amundsen Sea Low Index dataset, while using butterfly to check for unexpected changes in our results."}
```{r full_example, out.width = '100%', fig.align='center', echo = FALSE, fig.cap="A diagram showing the steps in our British Antarctic Survey data pipeline to calculate and publish the Amundsen Sea Low Index dataset, while using butterfly to check for unexpected changes in our results."}
knitr::include_graphics("img/full_pipeline_example.png")
```

Expand All @@ -75,7 +75,8 @@ existing_file <- read.csv(
)
knitr::kable(
existing_file[(nrow(existing_file) - 3):nrow(existing_file) - 1,]
existing_file[(nrow(existing_file) - 3):nrow(existing_file) - 1,],
row.names = FALSE
)
```

Expand All @@ -85,7 +86,8 @@ As you can see, all data for previous months are also included in the data:

```{r subsequent_month, echo=FALSE}
knitr::kable(
existing_file[(nrow(existing_file) - 4):nrow(existing_file),]
existing_file[(nrow(existing_file) - 4):nrow(existing_file),],
row.names = FALSE
)
```

Expand Down
7 changes: 4 additions & 3 deletions vignettes/butterfly.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,14 @@ butterflycount

This dataset is entirely fictional, and merely included to aid demonstrating butterfly's functionality.

## Using `butterfly` in a data processing pipeline

If you would like to know more about using `butterfly` in an operational data processing pipeline, please refer to the article on [using `butterfly` in an operational pipeline](https://thomaszwagerman.github.io/butterfly/articles/butterfly_in_pipeline.html).

## A note on controlling verbosity

Although verbosity is mostly the purpose if this package, **should** you wish to silence messages and warnings, you can do so with `options(rlib_message_verbosity = "quiet")` and options `(rlib_warning_verbosity = "quiet")`.

## Incorporating in data pipeline

Examples of using applying butterfly in a pipeline.

## Rationale

Expand Down

0 comments on commit bdfb3d6

Please sign in to comment.