Merge pull request #50 from wri/bugfix/pytest

Bugfix/pytest

.github/requirements.txt

@@ -0,0 +1,20 @@
+earthengine-api==0.1.408
+geocube==0.4.2
+geopandas==0.14.1
+rioxarray==0.15.0
+odc-stac==0.3.8
+pystac-client==0.7.5
+pytest==7.4.3
+xarray-spatial==0.3.7
+xee==0.0.3
+utm==0.7.0
+osmnx==1.9.3
+dask[complete]==2023.11.0
+matplotlib==3.8.2
+s3fs==2024.5.0
+geemap==0.32.0
+pip==23.3.1
+boto3==1.34.124
+scikit-learn==1.5.0
+overturemaps==0.6.0
+git+https://github.com/isciences/exactextract

.github/workflows/dev_ci_cd.yml

@@ -0,0 +1,37 @@
+name: Dev CIF API CI/CD
+
+on:
+  push:
+    branches: [ "bugfix/pytest" ]
+
+permissions:
+  contents: read
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 4
+      matrix:
+        python-version: ["3.10"]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install Linux dependencies
+      run: |
+        sudo apt update
+        sudo apt install -y gdal-bin libgdal-dev
+    - name: Install Packages
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r .github/requirements.txt
+        pip install GDAL==`gdal-config --version`
+    - name: Run Tests
+      env:
+        GOOGLE_APPLICATION_USER: ${{ secrets.GOOGLE_APPLICATION_USER }}
+        GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}     
+      run: |
+          pytest tests

README.md

@@ -3,54 +3,61 @@
 The Cities Indicator Framework (CIF) is a set of Python tools to make it easier to calculate zonal statistics for cities by providing a standardized set of data layers for inputs and a common framework for using those layers to calculate indicators.
 
 ## Quick start
+
 * If all you want to do is use the CIF, the quickest way to get started is to use our [WRI Cities Indicator Framework Colab Notebook](https://colab.research.google.com/drive/1PV1H-godxJ6h42p74Ij9sdFh3T0RN-7j#scrollTo=eM14UgpmpZL-)
 
 ## Installation
+
 * `pip install git+https://github.com/wri/cities-cif/releases/latest` gives you the latest stable release.
 * `pip install git+https://github.com/wri/cities-cif` gives you the main branch with is not stable.
 
 ## PR Review
-0. Prerequisites
-  1. Git
-    * On Windows I recommend WSL https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-git
-  3. https://cli.github.com/
-    * On MacOS I recommend the Homebrew option
-    * If you don't have an ssh key, it will install one for you
-  4. Conda (or Mamba) to install dependencies
-    * If you have Homebrew `brew install --cask miniconda`
 
+0. Prerequisites
+1. Git
+   * On Windows I recommend WSL [https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-git](https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-git)
+2. [https://cli.github.com/](https://cli.github.com/)
+   * On MacOS I recommend the Homebrew option
+   * If you don't have an ssh key, it will install one for you
+3. Conda (or Mamba) to install dependencies
+   * If you have Homebrew `brew install --cask miniconda`
 
 ## Dependencies
+
 There are 2 ways to install dependencies. Choose one...
 
 ### Conda
+
 `conda env create -f environment.yml`
 
 ### Setuptools
+
 `python setup.py`
 NOTE: If you are using this method you may want to use something like pyenv to manage Python environments
 
-
 ## Credentials
-To run the module, 
+
+To run the module,
+
   1. You need access to Google Earth Engine
-  2. Install https://cloud.google.com/sdk/docs/install
+  2. Install <https://cloud.google.com/sdk/docs/install>
 
 ### Interactive development
+
 For most people working in a notebook or IDE the script should walk you thourgh an interactive authentication process. You will just need to be logged in to your Google account that has access to GEE in your browser.
 
 ### Programatic access
+
 If you have issues with this or need to run the script as part of an automated workflow we have a GEE-enabled GCP service account that can be used. Get in touch with Saif or Chris to ask about getting the credetials.
 
 Set the following environment variables:
-- GOOGLE_APPLICATION_CREDENTIALS: The path of GCP credentials JSON file containing your private key.
-- GOOGLE_APPLICATION_USER: The email for your GCP user.
-- GCS_BUCKET: The GCS bucket to read and write data from. 
+
+* GOOGLE_APPLICATION_CREDENTIALS: The path of GCP credentials JSON file containing your private key.
+* GOOGLE_APPLICATION_USER: The email for your GCP user.
 
 For example, you could set the following in your `~/.zshrc` file:
 
 ```
-export GCS_BUCKET=gee-exports
 export GOOGLE_APPLICATION_USER=developers@citiesindicators.iam.gserviceaccount.com
 export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials/file
 ```
@@ -60,4 +67,3 @@ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials/file
 All are welcome to contribute by creating a [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). We try to follow the [Github Flow](https://docs.github.com/en/get-started/quickstart/github-flow) workflow.
 
 See the [developer docs](docs/developer.md) to learn more about how to add data layers and indicators.
-

city_metrix/__init__.py

@@ -1,26 +1,37 @@
-from .metrics import *
 import os
-import ee
 import warnings
 
+import ee
+
+from .metrics import *
+
 # initialize ee
-if "GOOGLE_APPLICATION_CREDENTIALS" in os.environ and "GOOGLE_APPLICATION_USER" in os.environ:
+if (
+    "GOOGLE_APPLICATION_CREDENTIALS" in os.environ
+    and "GOOGLE_APPLICATION_USER" in os.environ
+):
     print("Authenticating to GEE with configured credentials file.")
     CREDENTIAL_FILE = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
     GEE_SERVICE_ACCOUNT = os.environ["GOOGLE_APPLICATION_USER"]
-    auth = ee.ServiceAccountCredentials(GEE_SERVICE_ACCOUNT, CREDENTIAL_FILE)
-    ee.Initialize(auth, opt_url='https://earthengine-highvolume.googleapis.com')
+    if CREDENTIAL_FILE.endswith(".json"):
+        auth = ee.ServiceAccountCredentials(
+            GEE_SERVICE_ACCOUNT, key_file=CREDENTIAL_FILE
+        )
+    else:
+        auth = ee.ServiceAccountCredentials(
+            GEE_SERVICE_ACCOUNT, key_data=CREDENTIAL_FILE
+        )
+    ee.Initialize(auth, opt_url="https://earthengine-highvolume.googleapis.com")
 else:
     print("Could not find GEE credentials file, so prompting authentication.")
     ee.Authenticate()
-    ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')
+    ee.Initialize(opt_url="https://earthengine-highvolume.googleapis.com")
 
 
 # set for AWS requests
 os.environ["AWS_REQUEST_PAYER"] = "requester"
 
 # disable warning messages
-warnings.filterwarnings('ignore', module='xee')
-warnings.filterwarnings('ignore', module='dask')
-warnings.filterwarnings('ignore', module='xarray')
-
+warnings.filterwarnings("ignore", module="xee")
+warnings.filterwarnings("ignore", module="dask")
+warnings.filterwarnings("ignore", module="xarray")

environment.yml

@@ -23,5 +23,5 @@ dependencies:
   - boto3=1.34.124
   - scikit-learn=1.5.0
   - pip:
-    - git+https://github.com/isciences/exactextract
-    - overturemaps==0.6.0
+      - git+https://github.com/isciences/exactextract
+      - overturemaps==0.6.0

pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+testpaths = tests

tests/layers.py → tests/test_layers.py

@@ -1,12 +1,35 @@
 import ee
+import numpy as np
+import pytest
 
-from city_metrix.layers import LandsatCollection2, Albedo, LandSurfaceTemperature, EsaWorldCover, EsaWorldCoverClass, TreeCover, AverageNetBuildingHeight, OpenStreetMap, OpenStreetMapClass, UrbanLandUse, OpenBuildings, TreeCanopyHeight, AlosDSM, SmartSurfaceLULC, OvertureBuildings, NasaDEM
+from city_metrix.layers import (
+    Albedo,
+    AlosDSM,
+    AverageNetBuildingHeight,
+    EsaWorldCover,
+    EsaWorldCoverClass,
+    LandSurfaceTemperature,
+    NasaDEM,
+    OpenBuildings,
+    OpenStreetMap,
+    OpenStreetMapClass,
+    OvertureBuildings,
+    SmartSurfaceLULC,
+    TreeCanopyHeight,
+    TreeCover,
+    UrbanLandUse,
+)
 from city_metrix.layers.layer import get_image_collection
-from .conftest import MockLayer, MockMaskLayer, ZONES, LARGE_ZONES, MockLargeLayer, MockGroupByLayer, \
-    MockLargeGroupByLayer
 
-import pytest
-import numpy as np
+from .conftest import (
+    LARGE_ZONES,
+    ZONES,
+    MockGroupByLayer,
+    MockLargeGroupByLayer,
+    MockLargeLayer,
+    MockLayer,
+    MockMaskLayer,
+)
 
 
 def test_count():
@@ -49,29 +72,39 @@ def test_group_by_layer():
 
 
 def test_group_by_large_layer():
-    counts = MockLargeLayer().groupby(LARGE_ZONES, layer=MockLargeGroupByLayer()).count()
+    counts = (
+        MockLargeLayer().groupby(LARGE_ZONES, layer=MockLargeGroupByLayer()).count()
+    )
     assert all([count == {1: 50.0, 2: 50.0} for count in counts])
 
 
-SAMPLE_BBOX = (-38.35530428121955, -12.821710300686393, -38.33813814352424, -12.80363249765361)
+SAMPLE_BBOX = (
+    -38.35530428121955,
+    -12.821710300686393,
+    -38.33813814352424,
+    -12.80363249765361,
+)
 
 
 def test_read_image_collection():
     ic = ee.ImageCollection("ESA/WorldCover/v100")
     data = get_image_collection(ic, SAMPLE_BBOX, 10, "test")
 
     assert data.rio.crs == 32724
-    assert data.dims == {'x': 187, 'y': 200}
+    assert data.dims == {"x": 187, "y": 200}
 
 
 def test_read_image_collection_scale():
     ic = ee.ImageCollection("ESA/WorldCover/v100")
     data = get_image_collection(ic, SAMPLE_BBOX, 100, "test")
-    assert data.dims == {'x': 19, 'y': 20}
+    assert data.dims == {"x": 19, "y": 20}
 
 
 def test_tree_cover():
-    assert pytest.approx(53.84184165912419, rel=0.001) == TreeCover().get_data(SAMPLE_BBOX).mean()
+    assert (
+        pytest.approx(53.84184165912419, rel=0.001)
+        == TreeCover().get_data(SAMPLE_BBOX).mean()
+    )
 
 
 def test_albedo():
@@ -84,39 +117,57 @@ def test_lst():
 
 
 def test_esa():
-    count = EsaWorldCover(land_cover_class=EsaWorldCoverClass.BUILT_UP).get_data(SAMPLE_BBOX).count()
+    count = (
+        EsaWorldCover(land_cover_class=EsaWorldCoverClass.BUILT_UP)
+        .get_data(SAMPLE_BBOX)
+        .count()
+    )
     assert count
 
+
 def test_average_net_building_height():
     assert AverageNetBuildingHeight().get_data(SAMPLE_BBOX).mean()
 
+
 def test_open_street_map():
-    count = OpenStreetMap(osm_class=OpenStreetMapClass.ROAD).get_data(SAMPLE_BBOX).count().sum()
+    count = (
+        OpenStreetMap(osm_class=OpenStreetMapClass.ROAD)
+        .get_data(SAMPLE_BBOX)
+        .count()
+        .sum()
+    )
     assert count
 
+
 def test_urban_land_use():
     assert UrbanLandUse().get_data(SAMPLE_BBOX).count()
 
+
 def test_openbuildings():
     count = OpenBuildings().get_data(SAMPLE_BBOX).count().sum()
     assert count
 
+
 def test_tree_canopy_hight():
     count = TreeCanopyHeight().get_data(SAMPLE_BBOX).count()
     assert count
-
+
+
 def test_alos_dsm():
     mean = AlosDSM().get_data(SAMPLE_BBOX).mean()
     assert mean
 
+
 def test_smart_surface_lulc():
     count = SmartSurfaceLULC().get_data(SAMPLE_BBOX).count()
     assert count
-
+
+
 def test_overture_buildings():
     count = OvertureBuildings().get_data(SAMPLE_BBOX).count().sum()
     assert count
 
+
 def test_nasa_dem():
     mean = NasaDEM().get_data(SAMPLE_BBOX).mean()
     assert mean