04_dataset_augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7aa32761-c997-4a1a-b935-0fa22fd33e96",
   "metadata": {},
   "source": [
    "<font face='Calibri' size='2'> <i>eSBAE - Notebook Series - Part 4, version 0.1, April 2023. Andreas Vollrath, UN-Food and Agricultural Organization, Rome</i>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b64a68-7429-427e-9eb9-fcb16d592cef",
   "metadata": {
    "tags": []
   },
   "source": [
    "![title](images/header.png)\n",
    "\n",
    "# IV - eSBAE Dataset Augmentation\n",
    "### Run various change detection algorithms on previously extracted time-series data\n",
    "-------\n",
    "\n",
    "This notebook takes you through the process of running various change detection algorithms for a set of points using [Google's Earth Engine](https://earthengine.google.com/) as well as python routines. The script is optimized to deal with thousands of points and will use parallelization to efficiently extract the information from the platform.\n",
    "\n",
    "**You will need**:\n",
    "- a valid Earth Engine account ([sign up here](https://code.earthengine.google.com/register))\n",
    "- having successfully executed Notebook 3 of the eSBAE notebook series\n",
    "\n",
    "**This notebook runs best on a r16 instance**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "97ab0346-577d-4a97-a5aa-31e45d1aa944",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/share/jupyter/kernels/venv-eSBAE_notebooks/venv/lib/python3.8/site-packages/bfast/__init__.py:10: UserWarning: cupy is not available in this environment, GPU fonctionnalities won't be available\n",
      "  warn(\"cupy is not available in this environment, GPU fonctionnalities won't be available\")\n"
     ]
    }
   ],
   "source": [
    "# initialize EE    \n",
    "import ee\n",
    "try:\n",
    "    ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')\n",
    "except:\n",
    "    ee.Authenticate()\n",
    "    ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')\n",
    "    \n",
    "from sampling_handler import DatasetAugmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc9b2066-0522-4871-b0f1-c974f1c998e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <style>\n",
       "                .geemap-dark {\n",
       "                    --jp-widgets-color: white;\n",
       "                    --jp-widgets-label-color: white;\n",
       "                    --jp-ui-font-color1: white;\n",
       "                    --jp-layout-color2: #454545;\n",
       "                    background-color: #383838;\n",
       "                }\n",
       "                    \n",
       "                .geemap-dark .jupyter-button {\n",
       "                    --jp-layout-color3: #383838;\n",
       "                }\n",
       "                \n",
       "                .geemap-colab {\n",
       "                    background-color: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "                    \n",
       "                .geemap-colab .jupyter-button {\n",
       "                    --jp-layout-color3: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "            </style>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Using existing project directory at /home/sepal-user/module_results/esbae/sepredd\n",
      "INFO: Using existent config file from project directory /home/sepal-user/module_results/esbae/sepredd\n"
     ]
    }
   ],
   "source": [
    "esbae = DatasetAugmentation(\n",
    "    \n",
    "    # your project name, as set in previous notebooks\n",
    "    project_name = 'sepredd',\n",
    "\n",
    "    # start of calibration period (mainly for bfast)\n",
    "    calibration_start = '2010-01-01',  # YYYY-MM-DD format\n",
    "\n",
    "    # Actual period of interest, i.e. monitoring period\n",
    "    monitor_start =  '2015-01-01',  # YYYY-MM-DD format\n",
    "    monitor_end   =  '2023-01-01',  # YYYY-MM-DD format\n",
    "\n",
    "    # select the band for univariate ts-analysis (has to be inside bands list)\n",
    "    ts_band = 'ndfi'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26756c74-adc4-419e-8275-830f20950918",
   "metadata": {},
   "source": [
    "### Basic settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "baceb4a8-f516-44c4-b137-b9f93045d919",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <style>\n",
       "                .geemap-dark {\n",
       "                    --jp-widgets-color: white;\n",
       "                    --jp-widgets-label-color: white;\n",
       "                    --jp-ui-font-color1: white;\n",
       "                    --jp-layout-color2: #454545;\n",
       "                    background-color: #383838;\n",
       "                }\n",
       "                    \n",
       "                .geemap-dark .jupyter-button {\n",
       "                    --jp-layout-color3: #383838;\n",
       "                }\n",
       "                \n",
       "                .geemap-colab {\n",
       "                    background-color: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "                    \n",
       "                .geemap-colab .jupyter-button {\n",
       "                    --jp-layout-color3: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "            </style>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# select basic pre-processing options\n",
    "esbae.outlier_removal =     False\n",
    "esbae.smooth_ts =           False\n",
    "\n",
    "# Select the algorithms to run\n",
    "esbae.run_cusum =           True\n",
    "esbae.run_bfast =           False\n",
    "esbae.run_ts_metrics =      False\n",
    "esbae.run_bs_slope =        False\n",
    "esbae.run_jrc_nrt =         False     # needs further debugging right now\n",
    "esbae.run_ccdc =            False\n",
    "esbae.run_land_trendr =     False     # not yet implemented\n",
    "esbae.run_global_products = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90323e10-ebc2-420e-ae40-310e9739ef44",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4- Advanced Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "62c5c84a-907b-4dd1-96bf-ef7227620fbf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <style>\n",
       "                .geemap-dark {\n",
       "                    --jp-widgets-color: white;\n",
       "                    --jp-widgets-label-color: white;\n",
       "                    --jp-ui-font-color1: white;\n",
       "                    --jp-layout-color2: #454545;\n",
       "                    background-color: #383838;\n",
       "                }\n",
       "                    \n",
       "                .geemap-dark .jupyter-button {\n",
       "                    --jp-layout-color3: #383838;\n",
       "                }\n",
       "                \n",
       "                .geemap-colab {\n",
       "                    background-color: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "                    \n",
       "                .geemap-colab .jupyter-button {\n",
       "                    --jp-layout-color3: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "            </style>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "esbae.bfast = {\n",
    "    'run': esbae.run_bfast,\n",
    "    'start_monitor': esbae.monitor_start,\n",
    "    'freq': 365,\n",
    "    'k': 3,\n",
    "    'hfrac': 0.25,\n",
    "    'trend': True,\n",
    "    'level': 0.05,\n",
    "    'backend': 'python'\n",
    "}\n",
    "\n",
    "esbae.cusum = {\n",
    "    'run': esbae.run_cusum,\n",
    "    'nr_of_bootstraps': 1000\n",
    "}\n",
    "\n",
    "esbae.bs_slope = {\n",
    "    'run': esbae.run_bs_slope,\n",
    "    'nr_of_bootstraps': 1000\n",
    "}\n",
    "\n",
    "esbae.ts_metrics = {\n",
    "    'run': esbae.run_ts_metrics,\n",
    "    'bands': ['red', 'nir', 'swir1', 'swir2', 'ndfi', 'brightness', 'greenness', 'wetness'],\n",
    "    'metrics': ['mean', 'stddev', 'min', 'max'],  # DO NOT CHANGE YET\n",
    "    'outlier_removal': False,\n",
    "    'z_threshhold': 3\n",
    "}\n",
    "\n",
    "esbae.ccdc = {\n",
    "    'run': esbae.run_ccdc,\n",
    "    'breakpointBands': ['green', 'red', 'nir', 'swir1', 'swir2'],\n",
    "    'tmaskBands': ['green', 'swir2'],\n",
    "    'minObservations': 6,\n",
    "    'chiSquareProbability': 0.99,\n",
    "    'minNumOfYearsScaler': 1,\n",
    "    'dateFormat': 2,\n",
    "    'lambda': 20,\n",
    "    'maxIterations': 1000\n",
    "}\n",
    "\n",
    "esbae.land_trendr = {\n",
    "    'run': esbae.run_land_trendr,\n",
    "    'maxSegments': 6,\n",
    "    'spikeThreshold': 0.9,\n",
    "    'vertexCountOvershoot': 3,\n",
    "    'preventOneYearRecovery': True,\n",
    "    'recoveryThreshold': 0.25,\n",
    "    'pvalThreshold': 0.05,\n",
    "    'bestModelProportion': 0.75,\n",
    "    'minObservationsNeeded': 3\n",
    "}\n",
    "\n",
    "esbae.jrc_nrt = {\n",
    "    'run': esbae.run_jrc_nrt\n",
    "}\n",
    "\n",
    "esbae.global_products = {\n",
    "    'run': esbae.run_global_products,\n",
    "    'gfc': True,\n",
    "    'tmf': True,\n",
    "    'tmf_years': True,\n",
    "    'esa_lc20': True,\n",
    "    'copernicus_lc': True,\n",
    "    'esri_lc': True,\n",
    "    'lang_tree_height': True,\n",
    "    'potapov_tree_height': True,\n",
    "    'elevation': True,\n",
    "    'dynamic_world_tree_prob': True,\n",
    "    'dynamic_world_class_mode': True\n",
    "}\n",
    "\n",
    "esbae.py_workers = 50\n",
    "esbae.ee_workers = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b28743c-e512-4c3e-8907-83b0b4385d95",
   "metadata": {},
   "source": [
    "### 5 - Run the dataset augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c6a92940-b2ae-4640-9287-3ffcdd13be4a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <style>\n",
       "                .geemap-dark {\n",
       "                    --jp-widgets-color: white;\n",
       "                    --jp-widgets-label-color: white;\n",
       "                    --jp-ui-font-color1: white;\n",
       "                    --jp-layout-color2: #454545;\n",
       "                    background-color: #383838;\n",
       "                }\n",
       "                    \n",
       "                .geemap-dark .jupyter-button {\n",
       "                    --jp-layout-color3: #383838;\n",
       "                }\n",
       "                \n",
       "                .geemap-colab {\n",
       "                    background-color: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "                    \n",
       "                .geemap-colab .jupyter-button {\n",
       "                    --jp-layout-color3: var(--colab-primary-surface-color, white);\n",
       "                }\n",
       "            </style>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Verifying parameter settings...\n",
      "INFO: Initializing dataset augmentation routine...\n",
      "INFO: Accumulating batch files of 1/14...\n",
      "INFO: Running the dataset augmentation routines on 6341 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:04.097276\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:51.105370\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 1/14 finished in: : 0:02:01.439596\n",
      "INFO: Accumulating batch files of 10/14...\n",
      "INFO: Running the dataset augmentation routines on 6309 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:03.841685\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:51.049226\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 10/14 finished in: : 0:01:53.100452\n",
      "INFO: Accumulating batch files of 11/14...\n",
      "INFO: Running the dataset augmentation routines on 8671 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:05.438593\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:09.314532\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 11/14 finished in: : 0:02:28.146026\n",
      "INFO: Accumulating batch files of 12/14...\n",
      "INFO: Running the dataset augmentation routines on 8272 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:04.949202\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:05.915415\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 12/14 finished in: : 0:02:23.217659\n",
      "INFO: Accumulating batch files of 13/14...\n",
      "INFO: Running the dataset augmentation routines on 7632 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:04.462976\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:00.573669\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 13/14 finished in: : 0:02:08.959857\n",
      "INFO: Accumulating batch files of 14/14...\n",
      "INFO: Running the dataset augmentation routines on 631 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:00.437207\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:06.381863\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 14/14 finished in: : 0:00:17.000013\n",
      "INFO: Accumulating batch files of 2/14...\n",
      "INFO: Running the dataset augmentation routines on 9531 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:06.085637\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:17.088540\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 2/14 finished in: : 0:03:00.680165\n",
      "INFO: Accumulating batch files of 3/14...\n",
      "INFO: Running the dataset augmentation routines on 6226 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:03.264144\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:49.601177\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 3/14 finished in: : 0:01:44.670029\n",
      "INFO: Accumulating batch files of 4/14...\n",
      "INFO: Running the dataset augmentation routines on 9609 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:06.283417\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:18.449693\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 4/14 finished in: : 0:03:03.561391\n",
      "INFO: Accumulating batch files of 5/14...\n",
      "INFO: Running the dataset augmentation routines on 6082 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:03.326289\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:48.939914\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 5/14 finished in: : 0:01:47.186080\n",
      "INFO: Accumulating batch files of 6/14...\n",
      "INFO: Running the dataset augmentation routines on 5981 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:03.875490\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:00:48.564943\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 6/14 finished in: : 0:01:51.704774\n",
      "INFO: Accumulating batch files of 7/14...\n",
      "INFO: Running the dataset augmentation routines on 9973 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:06.273317\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:21.177328\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 7/14 finished in: : 0:03:08.401606\n",
      "INFO: Accumulating batch files of 8/14...\n",
      "INFO: Running the dataset augmentation routines on 9485 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:06.217400\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:16.670731\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 8/14 finished in: : 0:03:01.337194\n",
      "INFO: Accumulating batch files of 9/14...\n",
      "INFO: Running the dataset augmentation routines on 7920 points...\n",
      "INFO: Creating a subset of the time-series for the full analysis period (calibration & monitoring).\n",
      "INFO: Time-series subsetting finished in: 0:00:05.005736\n",
      "INFO: Running the CuSum algorithm on current batch of points.\n",
      "INFO: CuSum finished in: 0:01:04.363885\n",
      "INFO: Dump results table to output file...\n",
      "INFO: Batch 9/14 finished in: : 0:02:21.214571\n",
      "INFO: Dataset augmentation finished...\n"
     ]
    }
   ],
   "source": [
    "esbae.augment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a78f358-8251-4b3f-b5d2-d660ef2884bd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": " (venv) Sample Based Area Estimation",
   "language": "python",
   "name": "venv-esbae_notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}