From 6611c9e0ffb61e89ffa84316d29f6f9f98c980c7 Mon Sep 17 00:00:00 2001 From: johnhummelWallaroo Date: Mon, 8 Aug 2022 11:29:48 -0600 Subject: [PATCH] Added definition and blog post. --- shadow_deploy/shadow_deployment_tutorial.ipynb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/shadow_deploy/shadow_deployment_tutorial.ipynb b/shadow_deploy/shadow_deployment_tutorial.ipynb index 6c73e0ff..0432cc60 100644 --- a/shadow_deploy/shadow_deployment_tutorial.ipynb +++ b/shadow_deploy/shadow_deployment_tutorial.ipynb @@ -8,6 +8,12 @@ "\n", "Wallaroo provides a method of testing the same data against two different models or sets of models at the same time through **shadow deployments** otherwise known as **parallel deployments**. This allows data to be submitted to a pipeline with inferences running on two different sets of models. Typically this is performed on a model that is known to provide accurate results - the **champion** - and a model that is being tested to see if it provides more accurate or faster responses depending on the criteria known as the **challengers**. Multiple challengers can be tested against a single champion.\n", "\n", + "As described in the Wallaroo blog post [The What, Why, and How of Model A/B Testing](https://www.wallaroo.ai/blog/the-what-why-and-how-of-a/b-testing):\n", + "\n", + ">> In data science, A/B tests can also be used to choose between two models in production, by measuring which model performs better in the real world. In this formulation, the control is often an existing model that is currently in production, sometimes called the champion. The treatment is a new model being considered to replace the old one. This new model is sometimes called the challenger....\n", + "\n", + ">> Keep in mind that in machine learning, the terms experiments and trials also often refer to the process of finding a training configuration that works best for the problem at hand (this is sometimes called hyperparameter optimization).\n", + "\n", "When a shadow deployment is created, only the inference from the champion is returned in the [InferenceResult Object](https://docs.wallaroo.ai/staging.documentation/wallaroo-sdk/wallaroo-sdk-essentials-guide/#inferenceresult-object) `data`, while the result data for the shadow deployments is stored in the [InferenceResult Object](https://docs.wallaroo.ai/staging.documentation/wallaroo-sdk/wallaroo-sdk-essentials-guide/#inferenceresult-object) `shadow_data`.\n", "\n", "The following tutorial will demonstrate how:\n",