Skip to content

Commit

Permalink
Added definition and blog post.
Browse files Browse the repository at this point in the history
  • Loading branch information
johnhansarickWallaroo committed Aug 8, 2022
1 parent c023308 commit 6611c9e
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions shadow_deploy/shadow_deployment_tutorial.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@
"\n",
"Wallaroo provides a method of testing the same data against two different models or sets of models at the same time through **shadow deployments** otherwise known as **parallel deployments**. This allows data to be submitted to a pipeline with inferences running on two different sets of models. Typically this is performed on a model that is known to provide accurate results - the **champion** - and a model that is being tested to see if it provides more accurate or faster responses depending on the criteria known as the **challengers**. Multiple challengers can be tested against a single champion.\n",
"\n",
"As described in the Wallaroo blog post [The What, Why, and How of Model A/B Testing](https://www.wallaroo.ai/blog/the-what-why-and-how-of-a/b-testing):\n",
"\n",
">> In data science, A/B tests can also be used to choose between two models in production, by measuring which model performs better in the real world. In this formulation, the control is often an existing model that is currently in production, sometimes called the champion. The treatment is a new model being considered to replace the old one. This new model is sometimes called the challenger....\n",
"\n",
">> Keep in mind that in machine learning, the terms experiments and trials also often refer to the process of finding a training configuration that works best for the problem at hand (this is sometimes called hyperparameter optimization).\n",
"\n",
"When a shadow deployment is created, only the inference from the champion is returned in the [InferenceResult Object](https://docs.wallaroo.ai/staging.documentation/wallaroo-sdk/wallaroo-sdk-essentials-guide/#inferenceresult-object) `data`, while the result data for the shadow deployments is stored in the [InferenceResult Object](https://docs.wallaroo.ai/staging.documentation/wallaroo-sdk/wallaroo-sdk-essentials-guide/#inferenceresult-object) `shadow_data`.\n",
"\n",
"The following tutorial will demonstrate how:\n",
Expand Down

0 comments on commit 6611c9e

Please sign in to comment.