Use pg routing (#4245)

* Add TrekGeometry view with get and post methods * Add a parameter to TrekGeometry post method * Add graph and steps info being sent to TrekGeometry view * Add computing of the dijkstra matrix and start and end nodes (only handling whole edges with their default direction) * Add computing of the path to follow (node to node) (handles whole edges with their default direction) * Modify path in response: it's now from start to end * Refacto: nodes idx to id correspondence * Add structure of algorithm handling partial edges and more than 2 steps * ADD handling of edge percentage and more than 2 markers * Add display of a hard-coded polyline sent from the TrekGeometry view * The graph is directly taken from the backend * Add handling of GPS coordinates as input * Add use of Point ewkt when using sql * Add TODOs for next step: obtaining a LineString from dijkstra results * Starting to work on converting the resulting nodes list into LineStrings * Add conversion of dijstra results into a displayed Linestring * Correctng some comments * Fix path not computed if adjacent steps are on the same edge * Fix negative dijkstra weight when we're going backwards relative to a Path direction * Fix path not computed when going backwards on a Path if going through several edges * Starting to display fetched route instead of locally computed route * Preparing topology_helper to handle several linestrings * Automating the geojson generation in the view * Add handling of several steps * Replacing MultiLineString by GeometryCollection in Response * Fixing the organization of layers in the displayed path * Removing old code related to the graph * Removing old code * Removing old code * Making the TrekGeometry post method into a PathViewSet action * Reordering PathRouter methods * Removing graph action of PathViewSet * Moving initial graph generation into the PathRouter class * Removing no longer used imports * Renaming graph.py to path_router.py * Adding parameters error handling for the route_geometry action * Using dict.update instead of a custom method * Adding status codes to Responses * Optimizing the cs_graph generation (generating the upper triangle then filling the lower triangle symmetrically) * Optimizing matrix generation: it is now generated at init and modified depending on the current steps * Optimizing matrix generation by using its symmetrical property * Refactoring: making duplicate code into a method * The route is now fetched only when the marker dragging stops * Saving the dijkstra matrix in the cache * Saving the path graph in the cache * Moving duplicate cache-related code into a single method * Add handling of an impossible path: the view returns a status 204 and the js does not attempt to display the path * Modify error handling in js: using Promise.reject * Correct returns when no path could be found Methods that return an array when a path is found now return [] instead of None if no path can be found * Replaces status 204 by status 400 when no path can be found * Add a spinner when the route is being fetched * Add sending of the modified steps indexes to fetchRoute * Adds fetching of the route for only the modified steps * Add algorithm that updates the displayed route layers * Add use of currently displayed layers indexes to replace them * Fix replacing the currently displayed layers indexes by new ones * Fix updating layers not working after removing a marker * Bump leaflet.textpath.js to v1.1.0 * Move leaflet static files into a vendor directory * Bump leaflet.lineextremities.js to v0.1.1 * Fix missing path layer when an unlinkable via marker is removed * Remove console logs * Add coloring of the markers when one is isolated * Add coloring of all markers to normal appearance when back to normal * Add enabling/disabling of markers drag when a route is correct/incorrect * Add enabling/disabling of markers deletion when a route is correct/incorrect * Add impossibility of creating new via-steps while the route is invalid * Move spinner.stop() into a finally() * Fix one route layer missing when an isolated via-step is corrected * Add deletion of a marker if not dropped on a path * Add resetting of a marker to its previous valid position when not dropped on a path * Fix isolated marker highlight being removed when reset to its previous position * Remove duplicate part of code * Remove hiding of route layer when the start or end marker is moved * Fix out of date step idx used when deleting a via point * Add display of an error toast when a marker is incorrect * Modify error_toast elements class names for genericity * Fix toast stacking: top toast not on the bottom when it's the only one displayed * Fix wrong marker being deleted after trying to create an unsnapped marker * Fix route layer not displayed after start and end markers created and unlinkable * Add disabling of markers when the route is being fetched * Add saving of the route topology through the form * Add modification of the route topology through the edit form * Fix Geotrek object no longer created before use * Remove use of non-sparse matrix for dijkstra (using csr_array) * Modify use of sparse array (using lil_array instead of csr_array) * Optimize dijkstra matrix generation * Remove calls to now deleted js scripts * Remove no longer used methods * Add benchmarking system based on frontend routing version * Fix typo in cypress.config.js * Startng to work on adapting frontend benchmark system for backend * Add record of the number of runs for a benchmark measure * Modify route control clicking via cypress - Path elements' data-test attributes are now set when they are added to the page - Cypress test waits for the data-test attribute to be added before clicking on the route control * Replace broken 3 via-pts topology with a 2 via-pts topology * Modify cypress benchmark: adapt the frontend version to work with backend * Modify python time measures: now in milliseconds instead of seconds * Modify time measurement: startTime is now recorded before the click/drop event * Modify benchmark.sh: making the session id a variable * Add scenario using the big DB and no via point * Add display of current branch name for each measure * Replace big database no via pts scenario with 25 via pts scenario * Add readme for the benchmarking system * Modify benchmarking script: cypress spec file and session id are now passed as command line arguments * Fix crash when computing time averages if a file is missing * Add 2 via-pts topology for big database * Replace 25-via-pts scenario with 2-via-pts scenario for big database * Add unit tests for route_geometry view (check of request body) * Modify cypress timeouts: now 30min * Start to work on impossible and possible path test cases * Add check of lat and lng type and range in route_geometry view * Fix request body not sent in the right format (route_geometry tests) * Add new scenario for big DB with no via point * Add test case: steps not on paths * Add test case: steps one paths not linked * Add tests for route_geometry view * Fix test for the route_geometry view * Remove prints * Move benchmarking directory into tools/ * Fix norm * Modify Cypress 'create trek' test after routing changes * Try to fix cypress e2e tests * Uncomment env_dev settings * Modify Cypress e2e tests: actually click on paths to create a trek * Remove console.log and debugger * Fix cypress e2e tests: id of clicked path * Add test case for route_geometry view: error 500 * Try to fix cypress tests: ensure the topology control is enabled before clicking on it * Comment content of test_graph temporarily * Adapt graph generation tests to the new routing system * Add tests for path_router * Add unit test for cache handling in path_router * Fix cache-related test for PathRouter * Modify test_route_geometry_not_fail_with_via_points_several_paths to include more cases * Modify route_geometry tests: more precision when checking results * Remove old unit test for route_geometry view * Remove redundant unit test for path_router * Add script to take backend time measures * Add steps for route with 100 via-pts for backend time measures script * Fix topology used in 100-via-pts fixture * Add big database topology in backend time measuring script * Improve readability of path_router test: using yied * Update benchmark README to include the new script * Fix requirements.txt after rebase * Modify leaflet map: start spinner and disable markers ASAP * Modify leaflet map: stop spinner and reenable markers as late as possible * Adapt backend benchmark script to new PR * Modify display of route layers: only redisplay those affected by a route modification * Fix route modif not saved if an incorrect marker has been fixed * Fix path layers not displayed at creation * Add computing of the dijkstra matrix and start and end nodes (only handling whole edges with their default direction) * Modify docker-compose to use pgrouting image * Add routing with pgRouting for hard-coded topology * Add use of request latlngs for pgRouting * Generate linestrings for routes with no via-pts * Add display of the route (geometry is wrong on last edge) * Add generation of the serialized topology * Remove old dijkstra-related methods * Remove old dijkstra-related imports * Fix route always containing the whole last path instead of a fraction of it * Replace call to ST_Linesubstring by ST_SmartLineSubstring to split start and end edges * Fix route geometry reversed when a path is taken from its end to its start * Remove LineString rounding before merging * Fix route topology not correct when a path is taken in reverse * Remove prints * Uncomment try except * Add source and target columns to core_path table for pgRouting * Add error handling when no path is found * Remove out of date unit tests * Add use of settings PATH_SNAPPING_DISTANCE for pgRouting tolerance * Add extensions required for database * Modify database extensions * Remove unused imports after rebase * Fix postgres images for github CI * Add setting of source and target to null for a path that is being split * Modify routing: don't try to route on draft paths * Fix routing taking too much time (unnecessary recalculation of graph) * Move setting of pgRouting-related values to null from specific case to trigger after geom is updated * Fix PathViewsTest test_route_geometry_not_fail_no_via_point_one_path * Fix no route found when a marker is on a path's extremity (0 or 1) * Fix verification of serialized positions in tests (wrong format) * Modify how the closest path is found: get it from the front rather than the back * Fix routing tests: create a path before sending the request with a path_id * Fix norm * Delete now empty test_path_router.py * Modify routing conditions: don't route on paths whose visiblity is set to false * Remove modification of geom_3d as a condition for setting pgRouting-related values to null in trigger * Fix incorrect path id test (wrong lng used) * Add checking of error messages for route_geometry unit tests * Add more detailed check of response values for route_geometry tests * Add test cases for route_geometry: draft and invisible paths, editing geom, adding / deleting paths * Add django command to generate pgRouting network topology * Add unit test for command generate_pgr_network_topology * Add entry in doc explaining how to use the generate_pgr_network_topology command * Move tests docstrings descriptions to top * Add docstrings to path_router * Adapt backend benchmark script for new PR * manage pgrouting in nstall scripts * enable pgrouting extension in debian postinst * Update benchmark system for pgRouting * fix tests and viewset permission * prevent flaky test * Update docs/install/import.rst * fix rebase * fix rebase --------- Co-authored-by: JustineFricou <jfr@makina-corpus.com> Co-authored-by: J-E Castagnede <j.e.castagnede@gmail.com>
GeotrekCE · Sep 13, 2024 · 0f81f5d · 0f81f5d
1 parent 7e9c1eb
commit 0f81f5d
Show file tree

Hide file tree

Showing 22 changed files with 1,381 additions and 734 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -57,7 +57,7 @@ jobs:
 
     services:
       postgres:
-        image: postgis/postgis:12-2.5
+        image: pgrouting/pgrouting:12-3.0-2.6.3
         env:
           POSTGRES_DB: ci_test
           POSTGRES_PASSWORD: ci_test
@@ -253,7 +253,7 @@ jobs:
 
     services:
       postgres:
-        image: postgis/postgis:12-2.5
+        image: pgrouting/pgrouting:12-3.0-2.6.3
         env:
           POSTGRES_DB: ci_test
           POSTGRES_PASSWORD: ci_test
@@ -337,7 +337,7 @@ jobs:
 
     services:
       postgres:
-        image: postgis/postgis:12-2.5
+        image: pgrouting/pgrouting:12-3.0-2.6.3
         env:
           POSTGRES_DB: ci_test
           POSTGRES_PASSWORD: ci_test

diff --git a/debian/postinst b/debian/postinst
@@ -65,6 +65,7 @@ if [ "$MANAGE_DB" = "true" ] && [ -z "$2" ]; then
 	# postgis_raster is only useful on postgis 3, it fails on postgis 2 but this is harmless
 	su postgres -c "psql -q -d $POSTGRES_DB -c 'CREATE EXTENSION postgis_raster;'" || true
 	su postgres -c "psql -q -d $POSTGRES_DB -c 'CREATE EXTENSION pgcrypto;'" || true
+	su postgres -c "psql -q -d $POSTGRES_DB -c 'CREATE EXTENSION pgrouting;'" || true
 fi
 
 # Generate secret key

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -15,7 +15,7 @@ x-images:
 
 services:
   postgres:
-    image: postgis/postgis:12-2.5
+    image: pgrouting/pgrouting:12-3.0-2.6.3
     env_file:
       - .env
     ports:

diff --git a/docs/install/import.rst b/docs/install/import.rst
@@ -53,6 +53,16 @@ To import a shapefile containing your paths, use the command ``loadpaths``::
         --srid=2154 --comments-attribute IT_VTT IT_EQ IT_PEDEST \
         --encoding latin9 -i
 
+After importing a large quantity of paths, it is recommended to pre-generate the
+paths graph needed for the routing. This action is not mandatory, but will reduce the time needed for the first
+routing following the import.
+
+To pre-generate the graph, use the ``generate_pgr_network_topology`` command::
+
+    sudo geotrek generate_pgr_network_topology
+    OR
+    docker compose run --rm web ./manage.py generate_pgr_network_topology
+
 .. _import-data-from-touristic-data-systems-sit:
 
 Import data from touristic data systems (SIT)
@@ -144,7 +154,7 @@ Don't forget the u character before strings if they contain non-ascii characters
 
 In this case categories and types in Geotrek database have to be the same as in Esprit parc database. Otherwise missing categories and types will be created in Geotrek database.
 
-Imported contents will be automatically published and approved. 
+Imported contents will be automatically published and approved.
 
 If you use an url that filters a unique category, you can change its name. Example to get only Honey products and set the Geotrek category and type in which import them:
 
@@ -173,7 +183,7 @@ When sensitive areas module is enabled, Geotrek provides 3 parsers to import dat
 
 * **Import sensitive areas from http://biodiv-sports.fr** (``geotrek.sensitivity.parsers.BiodivParser``). By default this
   parser imports all sensitive areas in configured spatial extent.
-* **Import species sensitive areas from a zipped shapefile**. 
+* **Import species sensitive areas from a zipped shapefile**.
   Imported field names are: ``espece`` (required), ``contact`` and ``descriptio``. Species with corresponding names have to be created manually before import.
 * **Import regulatory sensitive areas from a zipped shapefile**. Imported field names are: ``nom`` (required), ``contact``, ``descriptio``, ``periode`` (month numbers separated with comas), ``pratiques`` (separated with comas), and ``url``. Practices with corresponding names have to be created manually before import.
 
@@ -228,7 +238,7 @@ In the following example, ``Provider_1Parser`` and ``Provider_2Parser`` will eac
 
 
 .. danger::
-    It is recommended to use ``provider`` from the first import - Do not add a ``provider`` field to preexisting parsers that already imported objects, or you will have to manually set the same value for ``provider`` on all objects already created by this parser. 
+    It is recommended to use ``provider`` from the first import - Do not add a ``provider`` field to preexisting parsers that already imported objects, or you will have to manually set the same value for ``provider`` on all objects already created by this parser.
 
 
 .. danger::
@@ -450,7 +460,7 @@ You can also use some of Geotrek commands to import data from a vector file hand
 
 Possible data are e.g.: POI, infrastructures, signages, cities, districts, restricted areas, dives, paths.
 
-You must use these commands to import spatial data because of the dynamic segmentation, which will not be computed if you enter the data manually. 
+You must use these commands to import spatial data because of the dynamic segmentation, which will not be computed if you enter the data manually.
 
 Here are the Geotrek commands available to import data from file:
 
@@ -470,7 +480,7 @@ To get help about a command:
 ::
 
     sudo geotrek help <subcommand>
-    
+
 .. _import-dem-altimetry:
 
 
@@ -955,10 +965,10 @@ Merge segmented paths
 A path network is most optimized when there is only one path between intersections.
 If the path database includes many fragmented paths, they could be merged to improve performances.
 
-You can run ``sudo geotrek merge_segmented_paths``. 
+You can run ``sudo geotrek merge_segmented_paths``.
 
 .. danger::
-    This command can take several hours to run. During the process, every topology on a path will be set on the path it is merged with, but it would still be more efficient (and safer) to run it before creating topologies. 
+    This command can take several hours to run. During the process, every topology on a path will be set on the path it is merged with, but it would still be more efficient (and safer) to run it before creating topologies.
 
 Before :
 ::

diff --git a/geotrek/common/templates/common/sql/pre_20_extensions.sql b/geotrek/common/templates/common/sql/pre_20_extensions.sql
@@ -1,3 +1,6 @@
 -- Used to ensure extension is enabled even in test database (migrations disabled)
 -- Otherwise UUIDs on objects can not be generated on insert (including path splits)
-CREATE EXTENSION IF NOT EXISTS "pgcrypto";
+CREATE EXTENSION IF NOT EXISTS "pgcrypto";
+CREATE EXTENSION IF NOT EXISTS "postgis";
+CREATE EXTENSION IF NOT EXISTS "postgis_raster";
+CREATE EXTENSION IF NOT EXISTS "pgrouting" CASCADE;
diff --git a/geotrek/core/management/commands/generate_pgr_network_topology.py b/geotrek/core/management/commands/generate_pgr_network_topology.py
@@ -0,0 +1,13 @@
+from django.core.management.base import BaseCommand
+
+from geotrek.core.path_router import PathRouter
+
+
+class Command(BaseCommand):
+    help = """
+    Generates the paths graph (pgRouting network topology) by filling
+    columns source and target of the core_path table.
+    """
+
+    def handle(self, *args, **options):
+        PathRouter()  # PathRouter's init method builds the network topology
diff --git a/geotrek/core/migrations/0037_path_source_pgr_path_target_pgr.py b/geotrek/core/migrations/0037_path_source_pgr_path_target_pgr.py
@@ -0,0 +1,23 @@
+# Generated by Django 4.2.13 on 2024-07-23 10:16
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0036_auto_20230503_0837'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='path',
+            name='source_pgr',
+            field=models.IntegerField(blank=True, db_column='source', editable=False, help_text='Internal field used by pgRouting', null=True),
+        ),
+        migrations.AddField(
+            model_name='path',
+            name='target_pgr',
+            field=models.IntegerField(blank=True, db_column='target', editable=False, help_text='Internal field used by pgRouting', null=True),
+        ),
+    ]
diff --git a/geotrek/core/models.py b/geotrek/core/models.py
@@ -73,6 +73,17 @@ class Path(CheckBoxActionMixin, ZoningPropertiesMixin, AddPropertyMixin, Geotrek
     draft = models.BooleanField(default=False, verbose_name=_("Draft"), db_index=True)
     uuid = models.UUIDField(default=uuid.uuid4, editable=False, unique=True)
 
+    source_pgr = models.IntegerField(null=True,
+                                     blank=True,
+                                     help_text='Internal field used by pgRouting',
+                                     editable=False,
+                                     db_column='source')
+    target_pgr = models.IntegerField(null=True,
+                                     blank=True,
+                                     help_text='Internal field used by pgRouting',
+                                     editable=False,
+                                     db_column='target')
+
     objects = PathManager()
     include_invisible = PathInvisibleManager()