workshop/3-Experimentation/3.5-Amplitude-Performance-Metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measuring Personalization Performance Using Amplitude\n",
    "In this exercise you will explore the conversion performance of your Personalize deployment in Retail Demo Store using [Amplitude's](https://amplitude.com/) real-time analytics platform.\n",
    "\n",
    "*Recommended Time: 30 minutes*\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "This module uses [Amplitude](https://amplitude.com/) to measure the performance of the personalization features in the Retail Demo Store. It is assumed that you have either completed the Personalization workshop or those resources have been pre-provisioned in your AWS environment. If you are unsure and attending an AWS managed event such as a workshop, check with your event lead.\n",
    "\n",
    "This project also assumes that Amplitude has been configured during the deployment of the Retail Demo Store in your AWS account or managed event account.\n",
    "\n",
    "In order to deploy Amplitude, you will need access to an Amplitude workspace, and an Amplitude API key that allows you to send data to Amplitude.  We recommend setting up a separate Amplitude project for the workshop, so that you can delete the demo data that will be passed to Amplitude as part of the workshop when you are finished.\n",
    "\n",
    "## Overview\n",
    "\n",
    "Deploying personalizaiton can be challenging in two ways:\n",
    "\n",
    "1. Deciding where to deploy personalized experiences is often done using best guesses\n",
    "2. Measuring the conversion outcomes of personalized experiences once they are deployed can be difficult without proper tooling\n",
    "\n",
    "Amplitude addresses these issues by providing a comprehensive real-time measurement platform that allows you to track user behavior in the form of real-time events before and after the deployment of Personalize. \n",
    "\n",
    "In this workshop, you will set up tracking for Amplitude events, analyze user behavior prior to peronalization being deployed, and then measure the effects of personalization on user behavior after Personalize is deployed in the Retail Demo Store.\n",
    "\n",
    "In the following steps, you will feed data about user behavior into Amplitude.  The dataset will include 60 days worth of simulated data going back from today.  The first 30 days will represent the time period before personalization is deployed, and the last thirty days will include user behavior data after personalization is deployed in the Retail Demo Store site.\n",
    "\n",
    "## Amplitude Deployment Architecture\n",
    "\n",
    "![Amplitude Deployment Architecture](./images/amplitude/workshop-architecture.png)\n",
    "\n",
    "## Set Up Amplitude\n",
    "\n",
    "Amplitude works by collecting real-time data from user behavior in the Retail Demo Store website.  The Amplitude tracking code is included in the workshop for you, so user interactions with the application will be collected in Amplitude if you included your Amplitude API key during the deployment of the Retail Demo Store environment.\n",
    "\n",
    "Because the Retail Demo Store is developed using the Vue framework, we can load Amplitude when the user loads a page by including the Amplitude tracking script using an import statement in `../src/web-ui/src/analytics/AnalyticsHandler.js` (this file can be found in the Github repo for this project):\n",
    "\n",
    "```javascript\n",
    "/* \n",
    " * Centralized handling of all analytics calls for Pinpoint, Personalize \n",
    " * (event tracker), and partner integrations.\n",
    " */\n",
    "import Vue from 'vue';\n",
    "import { Analytics as AmplifyAnalytics } from '@aws-amplify/analytics';\n",
    "import Amplitude from 'amplitude-js'\n",
    "```\n",
    "\n",
    "Once the tracking script is loaded into a page, it can be used to fire events in response to user actions.  As an example, when a user clicks on a product on one of the Retail Demo Store pages, a View event will be sent to Amplitude, using this code:\n",
    "\n",
    "```javascript\n",
    "if (this.amplitudeEnabled()) {\n",
    "    // Amplitude event\n",
    "    var eventProperties = {\n",
    "        productId: product.id,\n",
    "        name: product.name,\n",
    "        category: product.category,\n",
    "        image: product.image,\n",
    "        feature: feature,\n",
    "        experimentCorrelationId: experimentCorrelationId,\n",
    "        price: +product.price.toFixed(2)\n",
    "    };\n",
    "    Amplitude.getInstance().logEvent('View', eventProperties);\n",
    "}\n",
    "```\n",
    "\n",
    "These events are the same as events that are used to train the Personalize models used to offer users Personalize recommendations.  This is important since you will want the reports in Amplitude's analytics platform to reflect the same information that is being used to create Personalize models.  This will give you the most accurate way to measure user behavior before Personalize is deployed, and a consistent way to measure user behavior after Personalize is deployed in the Retail Demo Store site.\n",
    "\n",
    "Retail Demo Store sends the following events to Personalize and Amplitude:\n",
    "\n",
    "|Event Name|Event Description|Personalize|Amplitude|\n",
    "|:-- |:-- |:-: |:-: |\n",
    "|AddToCart|Every time a user adds a product to their cart; tracks the ID of the product that was added|X|X|\n",
    "|RemoveFromCart|Every time a user removes a product from their cart||X|\n",
    "|UpdateQuantity|Every time a user adds an additional unit of a product|X|X|\n",
    "|View|Every time a user views a product detail page|X|X|\n",
    "|ViewCart|Every time a user views the contents of their shopping cart|X|X|\n",
    "|StartCheckout|Every time a user begins the checkout process|X|X|\n",
    "|Purchase|Every time a user completes a purchase successfully|X|X|\n",
    "\n",
    "## Measuring Conversion Before Personalization is Deployed\n",
    "\n",
    "Because Amplitude reports rely on real user data, you will be using a data simulator to create enough events for you to be able to run a useful report in Amplitude.  The data simulator included in the Retail Demo Store will send the events shown above as though they were being generated by real users of the Retail Demo Store.  This will simulate common browsing behaviors at a sufficent volume to allow you to run a report that looks like one that might be used in your own production applications.\n",
    "\n",
    "The first simulated set of data will represent 30 days of user behavior in the Retail Demo Store prior to the deployment of the three personalization features you deployed in the first workshop exercise.  First, let's get some environment setup out of the way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import os\n",
    "import sys\n",
    "\n",
    "sys.path.insert(0, os.path.abspath('../../generators'))\n",
    "!pip install -r ../../generators/requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Then, let's get your Amplitude API key from SSM.  This will allow you to send simulated data directly to Amplitude.  If you did not set this up during your deployment of Retail Demo Store, skip to the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ssm = boto3.client('ssm')\n",
    "\n",
    "response = ssm.get_parameter(\n",
    "    Name='retaildemostore-amplitude-api-key'\n",
    ")\n",
    "\n",
    "amplitude_config = {\n",
    "    'api_key': response['Parameter']['Value']     # Do Not Change\n",
    "}\n",
    "\n",
    "print(amplitude_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the cell above returned a valid Amplitude API key, you can run the cells after the next cell to populate your Amplitude project with data.\n",
    "\n",
    "If you did not get a valid API key, it is likely you did not configure it in your deployment of Retail Demo Store.  You will need to set your Amplitude API key in the `amplitude_api_key` variable and then run the following cell to set the SSM key for Amplitude.\n",
    "\n",
    "You will then need to re-deploy the web ui for the Retail Demo Store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the Amplitude API key in the string below, and run this cell.  \n",
    "\n",
    "# THIS IS ONLY REQUIRED IF YOU DID NOT SET THE AMPLITUDE API KEY IN YOUR ORIGINAL DEPLOYMENT\n",
    "\n",
    "amplitude_api_key = ''\n",
    "\n",
    "if amplitude_api_key:\n",
    "    response = ssm.put_parameter(\n",
    "        Name='retaildemostore-amplitude-api-key',\n",
    "        Value='{}'.format(amplitude_api_key),\n",
    "        Type='String',\n",
    "        Overwrite=True\n",
    "    )\n",
    "\n",
    "    amplitude_config = {\n",
    "        'api_key': '{}'.format(amplitude_api_key)\n",
    "    }\n",
    "\n",
    "print(amplitude_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo Data Simulator\n",
    "\n",
    "The code in the following cell implements a user simulator that mimics the behavior of real users.  The `usage_funnels_pre_personalize` array specifies a series of funnels that the simulator will use to generate events that are sent to Amplitude.  \n",
    "\n",
    "The simulator uses the user profiles that are provided with the Retail Demo Store workshop, so that the user profiles shown in Amplitude reflect the same profiles you will see in the Retail Demo Store Users service.  Once a user profile is selected (at random), it will attempt to run the user through a selected funnel.  Not all users will complete each funnel, and some users may end up running through multiple funnels, depending on a weighted randomization function in the simulator.  This is done to provide realistic user behavior in the event patterns shown in Amplitude.  \n",
    "\n",
    "The simulator will also create events based on a time of day algorithm, where a majority of users will end up being more active during business hours, and fewer users will be active during after hours.  User profiles are also mapped to particular product affinities in the Retail Demo Store catalog, so that it is possible to create user behaviors that are specific to certain kinds of product categories.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "from datagenerator.rdscatalog import RDSCatalog\n",
    "from datagenerator.rdsuserstate import RDSUserSelectionState\n",
    "from datagenerator.users import UserPool\n",
    "from datagenerator.sessions import Sessions\n",
    "from datagenerator.output import OutputWriter\n",
    "\n",
    "search_props_template = {\n",
    "  'query': RDSUserSelectionState.user_search,\n",
    "  'resultCount': RDSUserSelectionState.num_results,\n",
    "  'reranked': False\n",
    "}\n",
    "\n",
    "search_props_feature_template = {\n",
    "  'query': RDSUserSelectionState.user_search,\n",
    "  'resultCount': RDSUserSelectionState.num_results,\n",
    "  'reranked': True\n",
    "}\n",
    "\n",
    "prod_clicked_template = {\n",
    "  'expand': RDSUserSelectionState.item_viewed_event_props\n",
    "}\n",
    "\n",
    "prod_clicked_feature_template = {\n",
    "  'expand': RDSUserSelectionState.item_viewed_event_props,\n",
    "  'feature': ['home_product_recs', 'product_detail_related']\n",
    "}\n",
    "\n",
    "prod_added_template = {\n",
    "  'expand': RDSUserSelectionState.item_added_event_props,\n",
    "}\n",
    "\n",
    "checkout_start_template = {\n",
    "  'expand': RDSUserSelectionState.cart_viewed_event_props,\n",
    "}\n",
    "\n",
    "order_completed_template = {\n",
    "  'expand': RDSUserSelectionState.cart_viewed_event_props\n",
    "}\n",
    "\n",
    "catalog = RDSCatalog('../../src/products/data/products.yaml')\n",
    "\n",
    "usage_funnels_pre_personalize = [\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('Search', search_props_template),\n",
    "      ('View', prod_clicked_template),\n",
    "      ('AddToCart', prod_added_template),\n",
    "      ('ViewCart', prod_clicked_template),\n",
    "      ('StartCheckout', checkout_start_template),\n",
    "      ('Purchase', order_completed_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('View', prod_clicked_template),\n",
    "      ('AddToCart', prod_added_template),\n",
    "      ('ViewCart', prod_clicked_template),\n",
    "      ('StartCheckout', checkout_start_template),\n",
    "      ('Purchase', order_completed_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('View', prod_clicked_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('Search', prod_clicked_template)\n",
    "    ]\n",
    "  }\n",
    "]\n",
    "\n",
    "date_time_now = datetime.datetime.now()\n",
    "\n",
    "pool = UserPool.from_file('../../src/users/src/users-service/data/users.json.gz')  # Load the known pool of users from the users file\n",
    "print(f'Loaded {pool.size()} users...')\n",
    "# Sends events to simulate application prior to personalization\n",
    "sessions = Sessions(date_time_now - datetime.timedelta(days=60), date_time_now - datetime.timedelta(days=30), usage_funnels_pre_personalize, 100, pool)\n",
    "writer = OutputWriter(sessions)\n",
    "writer.to_amplitude(amplitude_config)\n",
    "print(f'Events Send Completed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Amplitude Conversion Reports\n",
    "\n",
    "Once the event generator has completed, log in to the [Amplitude console](https://analytics.amplitude.com).\n",
    "\n",
    "In the console, you will use Amplitude's Funnel Analysis report to generate a report that enables you to see the conversion rate of the Retail Demo Store main purchase funnel.  Amplitude allows you to build a funnel for any user flow in your applications, but here we will focus on events leading to a purchase through the Retail Demo Store main purchase funnel. You will build a report that tracks all user sessions that perform the following actions in this order:\n",
    "\n",
    "View >\n",
    "AddToCart >\n",
    "StartCheckout >\n",
    "Purchase\n",
    "\n",
    "Note that users might perform other steps in the middle of this flow, such as viewing other products, adding and removing items from their cart, etc. but the Funnel report will ignore those events for the purpose of this report, since we are looking to see only conversions for users that have viewed a particular product and then purchased it through the minimum checkout flow steps for the Retail Demo Store.\n",
    "\n",
    "![](images/amplitude/conversion-funnel-pre-setup.png)\n",
    "\n",
    "To configure a Funnel report, select the New button in the left hand side of the screen, and then Report. Select Funnel Analysis for your report type at the top of the screen. In the events section below that, you will see a screen that looks like this:\n",
    "\n",
    "![](images/amplitude/conversion-funnel-pre-settings.png)\n",
    "\n",
    "Select the View, AddToCart, StartCheckout, and Purchase events in that order.  Once that is complete, select the data dropdown on the lower right hand side of the screen, and select the Between tab. Then input a start date 60 days before today as a start and 30 days before today as an end date.  This report will show conversion data for the 30 days *prior* to personalization being deployed in the Retail Demo Store.  \n",
    "\n",
    "![](images/amplitude/conversion-funnel-pre-dates.png)\n",
    "\n",
    "Your funnel report will look something like this, though the exact conversion numbers and number of events will vary slightly from this screen shot:\n",
    "\n",
    "![](images/amplitude/conversion-funnel-pre-pers.png)\n",
    "\n",
    "Click on the lower part of the Purchase bar graph. Note that the purchase conversion rate for the main funnel in the Retail Demo Store is 23.8%.  Not bad, but would we see an improvement in conversion if users were to receive personalized recommendations while they are interacting with the Retail Demo Store?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measuring the Conversion Effects of Personalized Product Recommendations\n",
    "\n",
    "In the first workshop, you deployed the following Personalization features in the Retail Demo Store:\n",
    "\n",
    "1. Personalized product recommendations\n",
    "2. Similar product recommendations\n",
    "3. Personalized ranking of products on category detail pages and personalized ranking of search results\n",
    "\n",
    "In order to measure how users convert after interacting with these features, we need to add a parameter to the event tracking sent to Amplitude, so that our funnel report can differentiate between product views and purchases that are *not* influenced via personalization features and product views and purchases that are a result of user interaction with the newly-added personalization features.\n",
    "\n",
    "In the `AnalyticsHandler.js` file, you will see code that adds a parameter to the View event:\n",
    "\n",
    "```javascript\n",
    "if (this.amplitudeEnabled()) {\n",
    "    // Amplitude event\n",
    "    var eventProperties = {\n",
    "        productId: product.id,\n",
    "        name: product.name,\n",
    "        category: product.category,\n",
    "        image: product.image,\n",
    "        feature: feature,\n",
    "        experimentCorrelationId: experimentCorrelationId,\n",
    "        price: +product.price.toFixed(2)\n",
    "    };\n",
    "    Amplitude.getInstance().logEvent('View', eventProperties);\n",
    "}\n",
    "```\n",
    "\n",
    "The `feature` property will contain the name of the personalization feature that the user clicked on.  This will be used for personalized product recommendations and for similar product recommendations.  \n",
    "\n",
    "This property will allow us to create a funnel report in Amplitude that shows the conversion rate of products clicked by users as a result of viewing a Personalization feature.\n",
    "\n",
    "Run the following cell to generate data that will enable you to build a report that uses the `feature` property to analyze conversions after the personalization features have been deployed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "usage_funnels_post_personalize_features = [\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('Search', search_props_feature_template),\n",
    "      ('View', prod_clicked_template),\n",
    "      ('AddToCart', prod_added_template),\n",
    "      ('ViewCart', prod_clicked_template),\n",
    "      ('StartCheckout', checkout_start_template),\n",
    "      ('Purchase', order_completed_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('View', prod_clicked_feature_template),\n",
    "      ('AddToCart', prod_added_template),\n",
    "      ('ViewCart', prod_clicked_template),\n",
    "      ('StartCheckout', checkout_start_template),\n",
    "      ('Purchase', order_completed_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('View', prod_clicked_template)\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    'platform': 'web',\n",
    "    'state': lambda user: RDSUserSelectionState(catalog, user),\n",
    "    'templates': [\n",
    "      ('View', prod_clicked_template),\n",
    "      ('AddToCart', prod_added_template),\n",
    "      ('ViewCart', prod_clicked_template),\n",
    "      ('StartCheckout', checkout_start_template),\n",
    "      ('Purchase', order_completed_template)\n",
    "    ]\n",
    "  }\n",
    "]\n",
    "\n",
    "# Sends events to simulate application after personalization features are added\n",
    "sessions = Sessions(date_time_now - datetime.timedelta(days=30), date_time_now, usage_funnels_post_personalize_features, 100, pool)\n",
    "writer = OutputWriter(sessions)\n",
    "writer.to_amplitude(amplitude_config)\n",
    "print(f'Events Send Completed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Amplitude Personalization Conversion Reports \n",
    "\n",
    "Once the event generator has completed, go back to the Amplitude console.\n",
    "\n",
    "In the console, you will once again use Amplitude's Funnel Analysis report to generate a report that enables you to see the conversion rate of the Retail Demo Store main purchase funnel.  This time after the Personalize features have been deployed.  We will still track the normal conversion funnel as before, but now we are going to drill down into View events that were a result of the first two personalizaiton features we deployed earlier.\n",
    "\n",
    "View >\n",
    "AddToCart >\n",
    "StartCheckout >\n",
    "Purchase\n",
    "\n",
    "Instead of looking at all user paths through this funnel, we will look individually at the number of conversions that start with an interaction with a personalization feature.  Under the View event, select `where` and then the `feature` property.  This will show a dropdown window that allows you to specify which values for the feature will be selected for the report.  In this case, we will look for events that are tied to 'product_detail_related' and 'home_product_recs'.  These values correspond to the Related Products feature, and the Product Recommendations feature for users that land on the home page.\n",
    "\n",
    "![](./images/amplitude/product-viewed-feature-select.png)\n",
    "\n",
    "This flow will now show conversions only for events that resulted in a purchase as a result of the user having initially looked at a product that was recommended on the home page (user personalization) or as a result of a related product recommendation (similar products personalization).\n",
    "\n",
    "![](images/amplitude/conversion-funnel-post-pers.png)\n",
    "\n",
    "This funnel shows a combined conversion rate of 30.6%.  This means that for users that have clicked on products that are recommended by these two features, there is a 6.8% increase in conversion (over the 23.8% conversion rate for for all users that view recommended or similar products over users that find products organically.\n",
    "\n",
    "Speaking of organic searches, let's look at how personalized search results ranking performed against regular searches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Amplitude Search Personalization Conversion Reports\n",
    "\n",
    "The Retail Demo Store allows users to perform a text search for products.  In the earlier part of the workshop, you deployed Personalize to re-sort product searches to show users products that might be more relevant based on their browsing history.\n",
    "\n",
    "When the user performs a search in the Retail Demo Store, a Search event is sent:\n",
    "\n",
    "```javascript\n",
    "if (this.amplitudeEnabled()) {\n",
    "    // Amplitude event\n",
    "    var eventProperties = {\n",
    "        query: query,\n",
    "        reranked: (user ? 'true' : 'false'),\n",
    "        resultCount: numResults\n",
    "    };\n",
    "    Amplitude.getInstance().logEvent('Search', eventProperties);\n",
    "}\n",
    "```\n",
    "\n",
    "If the search results are re-ranked by Personalize, the `reranked` property will be set to True in the event.  This means that we can compare user funnels from before and after Personalize is deployed.\n",
    "\n",
    "For this funnel report, you will create an Amplitude funnel report that looks at:\n",
    "\n",
    "Search >\n",
    "AddToCart >\n",
    "StartCheckout >\n",
    "Purchase\n",
    "\n",
    "First, let's look at search conversion before Personalize is deployed.  Set up a funnel report, with the events shown above.\n",
    "\n",
    "![](images/amplitude/search-funnel-pre-setup.png)\n",
    "\n",
    "Then select a time range that between 60 and 30 days before today (this is the time period prior to personalization being deployed in your dataset).\n",
    "\n",
    "![](images/amplitude/search-funnel-pre-dates.png)\n",
    "\n",
    "You should see a report that looks like this.\n",
    "\n",
    "![](images/amplitude/search-funnel-pre-pers.png)\n",
    "\n",
    "In this case, there is a 43% conversion rate for organic searches without personalized ranking.\n",
    "\n",
    "Now, let's compare these results to search conversion after Personalize is deployed to re-rank search results.  In your funnel report, select the `where` option for the `Search` event, and then select the `reranked` property.  Select the `True` option.\n",
    "\n",
    "![](images/amplitude/search-funnel-post-setup.png)\n",
    "\n",
    "Then, select a time frame for the last 30 days, since this how long your dataset has personalization event data.\n",
    "\n",
    "![](images/amplitude/search-funnel-post-dates.png)\n",
    "\n",
    "Your report should now show conversions only for personalized events.\n",
    "\n",
    "![](images/amplitude/search-funnel-post.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workshop Complete\n",
    "\n",
    "Hopefully this introduction to user behavior monitoring using Amplitude showed you how you can deploy a similar solution along with Amazon Personalize. \n",
    "\n",
    "### Cleanup\n",
    "\n",
    "Please remember to delete the Amplitude data that was sent to your Amplitude project when you are finished with it.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}