3-Foldby_approach.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  FoldBy Approach\n",
    "\n",
    "In this second approach we take a different path, exploiting Dask bags in order to use a foldby operation to perform the needed aggregation.\n",
    "\n",
    "Apart from the data cleansing, which is exactly equal, this branch of our work is developed hand in hand with the previous one, having in mind the goal of obtaining the same results.\n",
    "\n",
    "The goal of this notebook is indeed to compare, on the same task, the foldby method against a groupby on a distributed Dask data frame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "distributed.deploy.ssh - INFO - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy\n",
      "distributed.deploy.ssh - INFO - distributed.scheduler - INFO - Clear task state\n",
      "distributed.deploy.ssh - INFO - distributed.scheduler - INFO -   Scheduler at:   tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.145:33165'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.145:35837'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.145:36981'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.145:43525'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.253:44319'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.253:46563'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.160:44677'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.253:46309'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.160:36245'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.253:45579'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.160:41191'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.160:45441'\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.145:37847\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.145:37847\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.145:33843\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-ruv9yxu5\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.41:45607'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.41:40151'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.41:38381'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.41:36747'\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.145:45183\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.145:45183\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.145:35327\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-_t4mlrni\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.145:42289\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.145:42289\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.145:45839\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-wwcrkbb4\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.145:36567\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.146:46689'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.146:40835'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.146:42939'\n",
      "distributed.deploy.ssh - INFO - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.67.22.146:43183'\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.253:42649\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.253:42649\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.253:39719\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-mdntj9cj\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.146:38237\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.146:38237\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.146:38955\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-x0fku75j\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.146:45481\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.146:45481\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.146:40469\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-gauydi1o\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.146:33977\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.146:33977\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.146:32885\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-vjuu2qd0\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.146:44605\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.160:36929\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.160:36929\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.160:35975\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-5vbr96x8\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.160:38703\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.160:38703\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.160:39641\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-69nj1maq\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.160:34069\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.160:34069\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.160:33091\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-qdpimbeg\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.160:42253\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.253:45821\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.253:45821\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.253:37533\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-c9x7r0ik\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.253:44881\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:   tcp://10.67.22.253:44881\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:         10.67.22.253:40149\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-l1ntb39v\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:   tcp://10.67.22.253:39091\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:    tcp://10.67.22.41:41965\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:    tcp://10.67.22.41:41965\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:    tcp://10.67.22.41:32991\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:          10.67.22.41:45667\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:    tcp://10.67.22.41:32991\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:          10.67.22.41:40839\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-531sytdd\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-dw1r18qi\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:    tcp://10.67.22.41:42055\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          Listening to:    tcp://10.67.22.41:42055\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -          dashboard at:          10.67.22.41:44353\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - Waiting to connect to:    tcp://10.67.22.41:46517\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -               Threads:                          1\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -                Memory:                   1.94 GiB\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Local Directory: /root/dask-worker-space/worker-x0ehf_qd\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO - -------------------------------------------------\n",
      "distributed.deploy.ssh - INFO - distributed.worker - INFO -       Start worker at:    tcp://10.67.22.41:34169\n"
     ]
    }
   ],
   "source": [
    "from dask.distributed import Client, SSHCluster\n",
    "cluster = SSHCluster(\n",
    "     [\"10.67.22.41\", \"10.67.22.41\", \"10.67.22.145\", \"10.67.22.146\", '10.67.22.160', '10.67.22.253'],\n",
    "     worker_options={\"nthreads\": 1, 'n_workers':4},\n",
    "     scheduler_options={\"dashboard_address\": \":9455\"}\n",
    ")\n",
    "client = Client(cluster)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-7cebf326-ff8d-11ec-95ed-91e8aa8c8b10</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> distributed.SpecCluster</td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard: </strong> <a href=\"http://10.67.22.41:9455/status\" target=\"_blank\">http://10.67.22.41:9455/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\"></td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <details>\n",
       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
       "    </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">SpecCluster</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">SSHCluster</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard:</strong> <a href=\"http://10.67.22.41:9455/status\" target=\"_blank\">http://10.67.22.41:9455/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Workers:</strong> 20\n",
       "                </td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total threads:</strong> 20\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total memory:</strong> 38.85 GiB\n",
       "                </td>\n",
       "            </tr>\n",
       "            \n",
       "        </table>\n",
       "\n",
       "        <details>\n",
       "            <summary style=\"margin-bottom: 20px;\">\n",
       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
       "            </summary>\n",
       "\n",
       "            <div style=\"\">\n",
       "    <div>\n",
       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
       "        <div style=\"margin-left: 48px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-073eb049-01bf-4764-b94d-5b640648eec7</p>\n",
       "            <table style=\"width: 100%; text-align: left;\">\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Comm:</strong> tcp://10.67.22.41:46517\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Workers:</strong> 20\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Dashboard:</strong> <a href=\"http://10.67.22.41:9455/status\" target=\"_blank\">http://10.67.22.41:9455/status</a>\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total threads:</strong> 20\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Started:</strong> Just now\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total memory:</strong> 38.85 GiB\n",
       "                    </td>\n",
       "                </tr>\n",
       "            </table>\n",
       "        </div>\n",
       "    </div>\n",
       "\n",
       "    <details style=\"margin-left: 48px;\">\n",
       "        <summary style=\"margin-bottom: 20px;\">\n",
       "            <h3 style=\"display: inline;\">Workers</h3>\n",
       "        </summary>\n",
       "\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.145:36567</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.145:36567\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.145:41049/status\" target=\"_blank\">http://10.67.22.145:41049/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.145:36981\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-u0yult7i\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.145:37847</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.145:37847\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.145:33843/status\" target=\"_blank\">http://10.67.22.145:33843/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.145:43525\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-ruv9yxu5\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.145:42289</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.145:42289\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.145:45839/status\" target=\"_blank\">http://10.67.22.145:45839/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.145:35837\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-wwcrkbb4\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.145:45183</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.145:45183\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.145:35327/status\" target=\"_blank\">http://10.67.22.145:35327/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.145:33165\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-_t4mlrni\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.146:33977</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.146:33977\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.146:32885/status\" target=\"_blank\">http://10.67.22.146:32885/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.146:46689\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-vjuu2qd0\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.146:38237</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.146:38237\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.146:38955/status\" target=\"_blank\">http://10.67.22.146:38955/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.146:42939\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-x0fku75j\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.146:44605</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.146:44605\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.146:38251/status\" target=\"_blank\">http://10.67.22.146:38251/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.146:40835\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-2a9umbsj\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.146:45481</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.146:45481\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.146:40469/status\" target=\"_blank\">http://10.67.22.146:40469/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.146:43183\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-gauydi1o\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.160:34069</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.160:34069\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.160:33091/status\" target=\"_blank\">http://10.67.22.160:33091/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.160:44677\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-qdpimbeg\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.160:36929</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.160:36929\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.160:35975/status\" target=\"_blank\">http://10.67.22.160:35975/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.160:45441\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-5vbr96x8\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.160:38703</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.160:38703\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.160:39641/status\" target=\"_blank\">http://10.67.22.160:39641/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.160:36245\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-69nj1maq\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.160:42253</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.160:42253\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.160:34257/status\" target=\"_blank\">http://10.67.22.160:34257/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.160:41191\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-69t0spoi\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.253:39091</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.253:39091\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.253:39345/status\" target=\"_blank\">http://10.67.22.253:39345/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.253:46563\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-o2ck7kwb\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.253:42649</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.253:42649\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.253:39719/status\" target=\"_blank\">http://10.67.22.253:39719/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.253:45579\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-mdntj9cj\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.253:44881</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.253:44881\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.253:40149/status\" target=\"_blank\">http://10.67.22.253:40149/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.253:46309\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-l1ntb39v\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.253:45821</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.253:45821\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.253:37533/status\" target=\"_blank\">http://10.67.22.253:37533/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.253:44319\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-c9x7r0ik\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.41:32991</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.41:32991\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.41:40839/status\" target=\"_blank\">http://10.67.22.41:40839/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.41:45607\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-dw1r18qi\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.41:34169</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.41:34169\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.41:46251/status\" target=\"_blank\">http://10.67.22.41:46251/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.41:36747\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-xw0amty2\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.41:41965</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.41:41965\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.41:45667/status\" target=\"_blank\">http://10.67.22.41:45667/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.41:40151\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-531sytdd\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.67.22.41:42055</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://10.67.22.41:42055\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"http://10.67.22.41:44353/status\" target=\"_blank\">http://10.67.22.41:44353/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 1.94 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://10.67.22.41:38381\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /root/dask-worker-space/worker-x0ehf_qd\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "\n",
       "        </details>\n",
       "    </div>\n",
       "</div>\n",
       "            </details>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.67.22.41:46517' processes=4 threads=4, memory=7.77 GiB>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask\n",
    "import dask.dataframe as dd\n",
    "import io\n",
    "from functools import partial\n",
    "\n",
    "from fsspec.core import open_files\n",
    "from tlz import concat\n",
    "\n",
    "from dask.bag.core import from_delayed\n",
    "from dask.bytes import read_bytes\n",
    "from dask.delayed import delayed\n",
    "from dask.utils import parse_bytes, system_encoding\n",
    "import pandas as pd\n",
    "import re\n",
    "import dask.bag as db\n",
    "import numpy as np\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "from matplotlib import cm\n",
    "from operator import add\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_travel = db.read_text('/dataNFS/koln.txt', blocksize='9.455MB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_round_row(row):\n",
    "    \n",
    "    pattern = '[0-9]+\\.?[0-9]*'\n",
    "    findings = re.findall(pattern, row[:-1])\n",
    "    \n",
    "    if(len(findings) == len(row[:-1].split(' '))):\n",
    "        round_findings = [round(float(x),4) for x in findings]\n",
    "        return round_findings\n",
    "    else:\n",
    "        return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_cleaned = db_travel.map(clean_round_row).filter(lambda x: len(x)>0)\\\n",
    "                      .map(lambda x: {'time':int(x[0]), 'ID':int(x[1]), \n",
    "                                       'x':float(x[2]), 'y':float(x[3]),\n",
    "                                       'velocity':float(x[4])})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#upper, lower, right, left\n",
    "upper, lower, right, left = pd.read_csv(\"/dataNFS/grid_params.csv\", header=0)[['upper','lower', 'right','left']].values[0]\n",
    "\n",
    "# delta_x, delta_y\n",
    "delta_x, delta_y = pd.read_csv(\"/dataNFS/grid_params.csv\", header=0)[['delta_x','delta_y']].values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>base_ID</th>\n",
       "      <th>horizontal</th>\n",
       "      <th>vertical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3769.0203</td>\n",
       "      <td>1826.646344</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5427.4841</td>\n",
       "      <td>1826.646344</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7085.9479</td>\n",
       "      <td>1826.646344</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8744.4117</td>\n",
       "      <td>1826.646344</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10402.8755</td>\n",
       "      <td>1826.646344</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            x            y  base_ID  horizontal  vertical\n",
       "0   3769.0203  1826.646344        0           0         0\n",
       "1   5427.4841  1826.646344        1           1         0\n",
       "2   7085.9479  1826.646344        2           2         0\n",
       "3   8744.4117  1826.646344        3           3         0\n",
       "4  10402.8755  1826.646344        4           4         0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_BS = pd.read_csv(\"/dataNFS/df_BS.csv\", header=0)\n",
    "df_BS.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2939.7884</td>\n",
       "      <td>847.189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4598.2522</td>\n",
       "      <td>847.189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6256.7160</td>\n",
       "      <td>847.189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7915.1798</td>\n",
       "      <td>847.189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9573.6436</td>\n",
       "      <td>847.189</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           x        y\n",
       "0  2939.7884  847.189\n",
       "1  4598.2522  847.189\n",
       "2  6256.7160  847.189\n",
       "3  7915.1798  847.189\n",
       "4  9573.6436  847.189"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_grid = pd.read_csv(\"/dataNFS/df_grid.csv\", header=0)\n",
    "df_grid.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Foldby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hor_ver_func(row):\n",
    "    \n",
    "    row['horizontal'] = int((row['x'] - left)//delta_x)\n",
    "    row['vertical'] = int((row['y'] - lower)//delta_y)\n",
    "    \n",
    "    return row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_HV = db_cleaned.map(hor_ver_func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def square_match_db(row):\n",
    "    \n",
    "    base_targets = df_BS[df_BS['horizontal'] == row['horizontal']]\n",
    "    base_targets = base_targets[base_targets['vertical'] == row['vertical']]\n",
    "    base_targets['base_ID'] = base_targets.index\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    # TO LOOK AT\n",
    "    if len(base_targets) == 1:\n",
    "        row['base_ID'] = base_targets['base_ID'].values[0]\n",
    "        \n",
    "    \n",
    "    else: \n",
    "        row['base_ID'] = -1\n",
    "        \n",
    "        \n",
    "    return row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_HV_matched = db_HV.map(square_match_db).filter(lambda x: x['base_ID'] >= 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_fold_db_minmax = db_HV_matched.foldby( key = ['base_ID', 'ID'],\n",
    "                                              binop = lambda tot, x: {'min':min(tot['min'],x['time']),\n",
    "                                                                      'max':max(tot['max'],x['time'])},\n",
    "                                              initial = {'min':np.inf, 'max':-np.inf},\n",
    "                                              combine = lambda tot1, tot2: {'min':min(tot1['min'],tot2['min']),\n",
    "                                                                            'max':max(tot1['max'],tot2['max'])},\n",
    "                                              combine_initial = {'min':np.inf, 'max':-np.inf})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "diff_time = result_fold_db_minmax.map(lambda x: {'base_ID':x[0][0],\n",
    "                                                 'ID':x[0][1],\n",
    "                                                 'min':x[1]['min'],\n",
    "                                                 'max':x[1]['max']})\\\n",
    "                                  .map(lambda x: {'base_ID':x['base_ID'],\n",
    "                                                 'ID':x['ID'],\n",
    "                                                 'diff_time':x['max']-x['min']})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/diff_time_foldby.png\" width=300>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_per_base = diff_time.foldby(key = 'base_ID',\n",
    "                                 binop = lambda tot, x: {'sum':tot['sum']+x['diff_time'],\n",
    "                                                         'n':tot['n']+1},\n",
    "                                 initial = {'sum':0, 'n':0},\n",
    "                                 combine = lambda tot1, tot2: {'sum':tot1['sum']+tot2['sum'],\n",
    "                                                               'n':tot1['n']+tot2['n']},\n",
    "                                 combine_initial = {'sum':0, 'n':0})\\\n",
    "                         .map(lambda x: {'base_ID':x[0],\n",
    "                                         'mean':x[1]['sum']/x[1]['n']})\\\n",
    "                         .to_dataframe(meta={'base_ID':int, 'mean':float})\n",
    "                         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/mean_per_base_foldby.png\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The foldby operation does not provide a method to store the result in different partitions, this implementation indeed implies the final data frame to be small enough to fit one single worker's memory.\n",
    "The visualize graphs above show that a more in-depth analysis is required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark\n",
    "\n",
    "In the same way as in the groupby notebook we tried different values of the main parameters of our set-up, varying the number of workers (`n_workers`) for each machine of the cluster, the number of threads per worker (`nthreads`) and the block size (`blocksize`) used to read and import the data.\n",
    "\n",
    "Our aim is to test the performances of the two different DAG optimizations, doing a comparison in order to find the optimal one in terms of computational time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}