pygridmap

Geoprocessing tools for efficient geometric mapping and set operations over geospatial grids.

The pygridmap package enable you to perform some basic geometric/set operations over large/big vector datasets, including regular grid maker (i.e., rasterisation) and grid overlay (i.e. intersection and union). The code takes advantage of multiprocessor capabilities for efficient tile-based processing.

Description

The package pygridmap supports the following methods/algorithms:

• basic vector data handling over geospatial grid (e.g., bounding box manipulations, locations definition, etc...) with the GridProcessor class,

Regular gridding with different reference locations

• user-defined rasterisation of polygon regions (e.g., vector boundaries) into regular grids of various resolutions (i.e., dimension of one-size square grid cells) with the GridMaker class (see also Java implementation GridMaker),

Regular gridding striclty covering a spatial domain (left) or its interior only (right)

• customised vector overlay operations (e.g., intersection, union, overlay/overlap and merging) between any polygon layer and a geospatial grid ("regular" or not) with the Gridoverlay class, including weighted areal interpolation.

Intersection of a regular grid with an arbitrary vector layer

Quick launch

• Run (and update to your convenience) the test/example notebooks from the tests/ folder in binder: . We provide the interactive environments with already installed packages to query and access Eurostat database for notebook resources below (current build with commit ba83e79).

• Run the notebooks in Google colab separately, namely:

  • base.ipynb for basic data handling operations ,
  • gridding.ipynb for regular grid making ,
  • overlay.ipynb for overlay/overlap, merging, intersection and union operations .

Quick install

The pygridmap package can be installed using pip:

>>> pip install pygridmap

If you wish to use the latest available version from github:

>>> pip install git+https://github.com/eurostat/pygridmap.git

Usage

In your Python script

The following classes are available:

>>> from pygridmap.base import FrameProcessor, GridProcessor
>>> from pygridmap.gridding import GridMaker
>>> from pygridmap.overlay import GridOverlay

Through the bash command

Notes

Processing

The implementation of the methods above adopts (customised, using the multiprocessing module) multitprocessor tiling processing together with (native) vector processing whenever possible in order to take advantage of multiprocessor compute capabilities. Note that the FrameProcessor class can be used to run embarassing parallel calculations on dataframe rows.

Example of block-processing tiles

Geometric operations are making an extensive use of the geometric/geospatial Python module geopandas since this module supports essential features (most of them derived from the shapely fiona libraries), like:

• basic set and geometric operations, and
• spatial indexing through the R-Tree algorithm.

These implementations are probably not optimal since they are wrappers to the GEOS library, itself a port of the Java Topology Suite (JTS), similarly to what is used in QGis.

Algorithms

Given a source geospatial representation with various geometries, the simple way to create a regular grid with unit cells of dimension [cellx, celly] covering the spatial domain of the source vector layer is:

>>> from pygridmap import gridding
>>> grid = gridding.grid_maker(source, [cellx, celly]) 

See documentation of method gridding.grid_maker for additional parameters.

On areal interpolation, the package tobler that is distributed with the pysal library enables to perform more generic operations (i.e., not only considering grid as a target layer). Whenever one needs to project/interpolate an extensive feature attribute from a source vector layer onto a target vector layer (e.g., a regular grid), the following command:

>>> from pygridmap import overlay
>>> estimate = overlay.area_interpolate(source, target, attribute) 

is equivalent to running:

>>> from tobler import area_weighted
>>> estimate = area_weighted.area_interpolate(source, target, extensive_variables = attribute)

See documentation of method overlay.area_interpolate for additional parameters.

Areal interpolation of an extensive variable (left) from coarse to fine using tobler code (left) and pygridmap algorithm (right)

With respect to the latter implementation, the current algorithm supports a tile-based multicore approach for the interpolation (through the setting of the tile parameter). The tobler algorithm will however help you "project" intensive variables as well.

About

documentation	in construction
status	since 2020 – ongoing
contributors
license	EUPL

Requirements

• Data handling: numpy and pandas.
• Geospatial data processing: geopandas, fiona, shapely, RTree, geopy and pyproj.
• Map visualisations: ipyleaflet or folium.
• Areal interpolation (alternative implementation), as well as dasymetric mapping, and change of support: tobler, in particular module area_weighted.

Other resources

• Project python-geospatial, a collection of Python packages for geospatial analysis, and earthdatascience, a repository of Python tutorials and workshop.
• Library pysal for geospatial data science with an emphasis on geospatial vector data.
• Software GridMaker for alternative grid maker.

References

• Gimond, M. (2019): Intro to GIS and Spatial Analysis.
• Scheider, S. and Huisjes, M.D. (2018): Distinguishing extensive and intensive properties for meaningful geocomputation and mapping, International Journal of Geographical Information Science, 33(1):28-54, doi: 10.1080/13658816.2018.1514120.
• de Smith M.J., Goodchild M.F. and Longley P.A. (2018): Geospatial Analysis: A Comprehensive Guide to Principles, Techniques and Software Tools, The Winchelsea Press.
• Garrard C. (2016): Geoprocessing with Python, Manning Publications.
• Westra E. (2016): Python Geospatial Development, PACKT Publishing.
• Bahgat K. (2016): Python Geospatial Development Essentials, PACKT Publishing.
• Huyen Do V., Thomas-Agnan C. and Vanhems A. (2015): Spatial reallocation of areal data – another look at basic methods, Revue d’Économie Régionale & Urbaine, pp.27-58, doi: 10.3917/reru.151.0027.
• Lawed J. (2015): QGIS Python Programming CookBook, PACKT Publishing.
• Kresse W. and Danko D.M., eds. (2012): Handbook of Geographic Information, Springer, doi: 10.1007/978-3-540-72680-7.
• Campbell J. and Shin M. (2011): Essentials of Geographic Information Systems (pdf), Saylor Academy Open Textbooks.