06-big-data-visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "75c08ab2-1c66-46ad-82a3-c02d1166064d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-17T22:22:44.978029Z",
     "iopub.status.busy": "2023-04-17T22:22:44.977273Z",
     "iopub.status.idle": "2023-04-17T22:22:44.987478Z",
     "shell.execute_reply": "2023-04-17T22:22:44.986174Z",
     "shell.execute_reply.started": "2023-04-17T22:22:44.977988Z"
    },
    "tags": []
   },
   "source": [
    "<img src=\"https://hvplot.holoviz.org/_static/logo_horizontal.svg\" width=\"20%\" align=\"right\"/>\n",
    "\n",
    "# Big data visualization with Dask and hvPlot\n",
    "\n",
    "In this notebook, we'll learn to use the hvPlot APIs with Dask DataFrames.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e329369-2c9f-4ab2-afb1-ee3677bb4dbc",
   "metadata": {},
   "source": [
    "## Reconnect to our Dask Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f3eae75-0829-4bca-bebc-6d49c8ba0050",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dask_gateway\n",
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9265dde-8387-4c27-8702-1cb9015045c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gateway = dask_gateway.Gateway()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "168b1ded-076a-4894-a3ad-64dceefbe3c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(running_clusters := gateway.list_clusters())>0:\n",
    "    cluster = gateway.connect(running_clusters[0].name)\n",
    "else:\n",
    "    cluster = gateway.new_cluster(conda_environment=\"scipy/scipy-data-of-unusual-size\", profile=\"Medium Worker\")\n",
    "    cluster.adapt(1,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f895739f-ff0f-41e8-a933-6057b138b212",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993ca5bf-59d8-472d-aac3-85ea367e175c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client = cluster.get_client()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4d8612-d103-406f-bb21-5774e9af4129",
   "metadata": {},
   "source": [
    "## Load a subset of columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f27e8e6-0066-46c3-819c-209d22cb065f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "columns = [\n",
    "    'YEAR', 'MONTH', 'DAY_OF_MONTH', 'DAY_OF_WEEK', 'FL_DATE', 'OP_CARRIER', \n",
    "    'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'CRS_DEP_TIME', \n",
    "    'DEP_TIME', 'DEP_DELAY', 'ARR_TIME', 'ARR_DELAY', 'CANCELLED', \n",
    "    'CANCELLATION_CODE', 'DIVERTED', 'AIR_TIME', 'FLIGHTS', 'DISTANCE',\n",
    "    'CARRIER_DELAY', 'WEATHER_DELAY', 'NAS_DELAY', 'SECURITY_DELAY', \n",
    "    'LATE_AIRCRAFT_DELAY', 'DIV_ARR_DELAY'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c7ef3af-90e2-4205-ae7e-95a5cb76752a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "    f\"gcs://quansight-datasets/airline-ontime-performance/sorted/full_dataset.parquet\", \n",
    "    columns=columns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd8633a4-8781-4d8a-8fcc-b0f2e6f10de3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de8c1c82-53e5-4163-87ba-db873a8cbc70",
   "metadata": {},
   "source": [
    "## hvPlot + Dask\n",
    "\n",
    "To use hvPlot's build in Dask integration, we need to switch out:\n",
    "\n",
    "`import hvplot.pandas` for `import hvplot.dask` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657583d5-91f2-4926-9bea-6d40dda3c45a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import hvplot.dask\n",
    "hvplot.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5a95e4-5af7-4dc5-918f-ec74e63f4b38",
   "metadata": {},
   "source": [
    "### Plot the departure delay per day for the entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37290761-6df5-485b-81a2-8012d3a91a0d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.groupby('FL_DATE')['DEP_DELAY'].count().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64f15fc-d30c-45b3-a1ba-e1caa8b3ab17",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Visualize the weekly distribution of the mean of any variable in the datasets\n",
    "\n",
    "You can any plot type from the [hvPlot Gallery](https://hvplot.holoviz.org/reference/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a0542e3-5571-4221-b183-b21c79f186ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots below for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "232e750f-7270-47d4-9453-e1cddd5763d3",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.groupby('DAY_OF_WEEK')['ARR_DELAY'].mean().hvplot.scatter(x=\"DAY_OF_WEEK\", y='ARR_DELAY')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f7d953b-8bfa-4b04-9f3e-e25a07c60250",
   "metadata": {},
   "source": [
    "## Plotting large datasets\n",
    "\n",
    "In the above visualization of daily counts we saw a bunch of compute happening before we saw the plot appear. But after it was generated, panning and zooming did not cause any new Dask computes.\n",
    "\n",
    "This is because the final dataset after the groupby is only about `20 years * 365 days` long, so it fits completely in memory.\n",
    "\n",
    "Now let's look at the entire dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c312a4-34f7-4591-9cfd-955d4c754732",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"The full dataset has {len(flights)/1e6:2} million rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98673fe0-af60-4d1e-92fa-4d0a47017dfe",
   "metadata": {},
   "source": [
    "If we try and send these many data points to the browser for visualization in a plot, the browser would run out of memory and crash.\n",
    "\n",
    "<img src=\"images/datashader.svg\" width=\"30%\" align=\"right\">\n",
    "\n",
    "The solution for this is to take advantage of the fact that the output plot has a fixed resolution in terms of number of pixels. A 600x400 image has 240,000 pixels. This means that if we plotted 125 million points on the these pixels, most would overlay each other and not be visible. Instead, we pre-render or rasterize the data and shade in a manner that maintains an accurate the distribution of your data. \n",
    "\n",
    "We do this via the hvPlot integration with **Datashader**.\n",
    "\n",
    "We will use a smaller dataset for the next few examples for quick outputs. These examples will work with the full dataset, but will take a bit longer to run with the 10 computer nodes we are currently using for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06227e51-764e-4071-9348-817e86b80466",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "        f\"gcs://quansight-datasets/airline-ontime-performance/sorted/parquet_by_year\", \n",
    "        filters=[('YEAR', '>', 2017)],\n",
    "        columns=columns,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ffdc289-c85f-48f7-9e3b-1979b08c6b6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"The smaller dataset has {len(flights)/1e6} million rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7474b7ff-eda1-4305-a368-13c76e133539",
   "metadata": {
    "tags": []
   },
   "source": [
    "In these next two visualizations, Datashader data is displayed on the plots. \n",
    "As we pan and zoom, Datashader recomputes the appropriate pixel shades using Dask.\n",
    "\n",
    "This allows us to easily look at the entire 30 million row dataset, but still\n",
    "zoom into a single point, without requiring downsampling or decimation of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb333fe-8ffb-439e-ba77-52d1a8312a6d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.hvplot.line(x='FL_DATE', y='DEP_DELAY', datashade=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c2d360-a548-44f0-87e5-488b59d7996e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights[['ARR_DELAY', 'DISTANCE']].hvplot.scatter(x='ARR_DELAY', y='DISTANCE', datashade=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abd28a3-003e-4512-9b81-b577d824f945",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# To shutdown the cluster, uncomment and run the next line\n",
    "# cluster.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12a5a268-68ca-4714-81f9-f4c274d1f6be",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Next →\n",
    "\n",
    "[Big data dashboards](./07-big-data-dashboards.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb459bb3-3bac-406b-a175-a6742ba3b752",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-data-of-unusual-size",
   "language": "python",
   "name": "conda-env-scipy-scipy-data-of-unusual-size-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}