#587 COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS and per layer refreshInterva…

…ls (#588)

* #587 COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS and per layer refreshIntervals

* #587 Raster and Vector Refresh Intervals

* #587 Update more docs

* #587 Update more docs 2

configure/src/metaconfigs/layer-tile-config.json

@@ -198,6 +198,26 @@
               "defaultChecked": false
             }
           ]
+        },
+        {
+          "components": [
+            {
+              "field": "time.refreshIntervalEnabled",
+              "name": "Refresh Interval Enabled",
+              "description": "If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every 'Refresh Every N Seconds'. This is useful when the layer's data updates at some uniform cadence. Be aware that this may be an expensive operation depending on the amount of data a layer needs and the number of layers that have this enabled.",
+              "type": "switch",
+              "width": 5,
+              "defaultChecked": false
+            },
+            {
+              "field": "time.refreshIntervalAmount",
+              "name": "Refresh Every N Seconds",
+              "description": "If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every n seconds. If null or 0, defaults to 60.",
+              "type": "number",
+              "min": 1,
+              "width": 3
+            }
+          ]
         }
       ]
     },

configure/src/metaconfigs/layer-vector-config.json

@@ -419,6 +419,26 @@
               "width": 6
             }
           ]
+        },
+        {
+          "components": [
+            {
+              "field": "time.refreshIntervalEnabled",
+              "name": "Refresh Interval Enabled",
+              "description": "If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every 'Refresh Every N Seconds'. This is useful when the layer's data updates at some uniform cadence. Be aware that this may be an expensive operation depending on the amount of data a layer needs and the number of layers that have this enabled.",
+              "type": "switch",
+              "width": 5,
+              "defaultChecked": false
+            },
+            {
+              "field": "time.refreshIntervalAmount",
+              "name": "Refresh Every N Seconds",
+              "description": "If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every n seconds. If null or 0, defaults to 60.",
+              "type": "number",
+              "min": 1,
+              "width": 3
+            }
+          ]
         }
       ]
     },

docs/pages/Configure/Layers/Tile/Tile.md

@@ -40,6 +40,17 @@ The format of the tiles.
 - WMS: Web Map Service tiles are a popular way of publishing maps by professional GIS software. This format is similar to the previous two formats, but more generic and not so well optimized for use in web maps. A WMS image is defined by the coordinates of its corners. A layer (or list of layers) should be provided as an options by appending `?layers=<your_layer_name><,another_if_you _want>` to your `URL`. To override WMS parameters append `&<wms_param>=<value>` again to the `URL` after the "layers" parameters. If desired, use `&TILESIZE=` to change the tile size of the request and layer away from the default of 256.
   _Example URL: `http://ows.mundialis.de/services/service?layers=TOPO-WMS,OSM-Overlay-WMS`_
 
+#### Refresh Interval Enabled
+
+_type:_ boolean
+If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every 'Refresh Every N Seconds'. This is useful when the layer's data updates at some uniform cadence. Be aware that this may be an expensive operation depending on the amount of data a layer needs and the number of layers that have this enabled.
+
+#### Refresh Every N Seconds
+
+_type:_ number
+_default:_ 60
+If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every n seconds.
+
 #### Initial Visibility
 
 _type:_ bool  

docs/pages/Configure/Layers/Vector/Vector.md

@@ -79,6 +79,17 @@ Required in `Time Enabled = true` and `Time Type = Local`. The main time propert
 _type:_ string _optional_  
 The string format to be used in the URL for `{starttime}` and `{endtime}`. Defaults to `YYYY-MM-DDTHH:mm:ssZ`.
 
+#### Refresh Interval Enabled
+
+_type:_ boolean
+If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every 'Refresh Every N Seconds'. This is useful when the layer's data updates at some uniform cadence. Be aware that this may be an expensive operation depending on the amount of data a layer needs and the number of layers that have this enabled.
+
+#### Refresh Every N Seconds
+
+_type:_ number
+_default:_ 60
+If 'Time Enabled' and 'Refresh Interval Enabled', this layer will automatically refresh/requery its data every n seconds.
+
 #### Stroke Color
 
 _type:_ CSS color string or a prop _optional_  

docs/pages/Setup/ENVs/ENVs.md

@@ -163,3 +163,7 @@ If true, then also triggers the kernel download when MMGIS starts | boolean | de
 #### `SPICE_SCHEDULED_KERNEL_CRON_EXPR=`
 
 A cron schedule expression for use in the [node-schedule npm library](https://www.npmjs.com/package/node-schedule) | string | default `"0 0 */2 * *"` (every other day)
+
+#### `COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS=`
+
+When using composited time tiles, MMGIS queries the tileset's folder for existing time folders. It caches the results of the these folder listings every COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS milliseconds. Defaults to requerying every 30 minutes. If 0, no caching. If null or NaN, uses default. | number | default `1800000`

sample.env

@@ -92,4 +92,9 @@ SPICE_SCHEDULED_KERNEL_DOWNLOAD=false
 # If true, then also triggers the kernel download when MMGIS starts
 SPICE_SCHEDULED_KERNEL_DOWNLOAD_ON_START=false
 # A cron schedule expression for use in the node-schedule npm library. Defaults to "0 0 */2 * *" which is every other day.
-SPICE_SCHEDULED_KERNEL_CRON_EXPR=
+SPICE_SCHEDULED_KERNEL_CRON_EXPR=
+
+# When using composited time tiles, MMGIS queries the tileset's folder for existing time folders.
+# It caches the results of the these folder listings every COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS milliseconds
+# defaults to requerying every 30 minutes. If 0, no caching. If null or NaN, uses default.
+COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS=

scripts/middleware.js

@@ -6,7 +6,11 @@ const rootDir = `${__dirname}/..`;
 const rootDirMissions = `${rootDir}/Missions`;
 
 const dirStore = {};
-const DIR_STORE_MAX_AGE = 3600000 / 2; // 1hours / 2
+const DIR_STORE_MAX_AGE =
+  process.env.COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS == null ||
+  isNaN(parseInt(process.env.COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS))
+    ? 3600000 / 2
+    : parseInt(process.env.COMPOSITE_TILE_DIR_STORE_MAX_AGE_MS); // defaults to 1hours / 2
 
 async function compositeImageUrls(urls) {
   try {

src/essence/Ancillary/TimeControl.js

@@ -180,7 +180,13 @@ var TimeControl = {
         if (layer.time) return layer.time.end
         return false
     },
-    reloadLayer: async function (layer, evenIfOff, evenIfControlled) {
+    reloadLayer: async function (
+        layer,
+        evenIfOff,
+        evenIfControlled,
+        forceRequery,
+        skipOrderedBringToFront
+    ) {
         // reload layer
         if (typeof layer == 'string') {
             layer = L_.asLayerUUID(layer)
@@ -199,7 +205,8 @@ var TimeControl = {
 
         layer.url = await TimeControl.performTimeUrlReplacements(
             layer.url,
-            layer
+            layer,
+            forceRequery
         )
         let changedUrl = null
         if (layer.url !== originalUrl) changedUrl = layer.url
@@ -218,7 +225,8 @@ var TimeControl = {
             if (layer.time && layer.time.enabled === true) {
                 if (
                     layer.time.type === 'global' ||
-                    layer.time.type === 'requery'
+                    layer.time.type === 'requery' ||
+                    forceRequery
                 ) {
                     layer.url = layer.url
                         .replace(
@@ -250,7 +258,8 @@ var TimeControl = {
             if (
                 layer.type === 'vector' &&
                 layer.time.type === 'local' &&
-                layer.time.endProp != null
+                layer.time.endProp != null &&
+                forceRequery !== true
             ) {
                 if (evenIfControlled === true || layer.controlled !== true)
                     L_.timeFilterVectorLayer(
@@ -262,19 +271,46 @@ var TimeControl = {
                 // refresh map
                 if (evenIfControlled === true || layer.controlled !== true)
                     if (L_.layers.on[layer.name] || evenIfOff) {
-                        return await Map_.refreshLayer(layer, () => {
-                            // put start/endtime keywords back
-                            if (layer.time && layer.time.enabled === true)
-                                layer.url = originalUrl
-                        })
+                        return await Map_.refreshLayer(
+                            layer,
+                            () => {
+                                if (layer.time && layer.time.enabled === true) {
+                                    // put start/endtime keywords back
+                                    layer.url = originalUrl
+
+                                    // if requery was force, remember to timeFilter after load
+                                    if (
+                                        layer.type === 'vector' &&
+                                        layer.time.type === 'local' &&
+                                        layer.time.endProp != null &&
+                                        forceRequery === true
+                                    ) {
+                                        if (
+                                            evenIfControlled === true ||
+                                            layer.controlled !== true
+                                        )
+                                            L_.timeFilterVectorLayer(
+                                                layer.name,
+                                                new Date(
+                                                    layer.time.start
+                                                ).getTime(),
+                                                new Date(
+                                                    layer.time.end
+                                                ).getTime()
+                                            )
+                                    }
+                                }
+                            },
+                            skipOrderedBringToFront
+                        )
                     }
             }
         }
         // put start/endtime keywords back
         if (layer.time && layer.time.enabled === true) layer.url = originalUrl
         return true
     },
-    performTimeUrlReplacements: async function (url, layer) {
+    performTimeUrlReplacements: async function (url, layer, forceRequery) {
         return new Promise(async (resolve, reject) => {
             let layerTimeFormat =
                 layer.time?.format == null
@@ -315,12 +351,17 @@ var TimeControl = {
                     }
                 }
             }
+            if (forceRequery === true) {
+                nextUrl += `${
+                    nextUrl.indexOf('?') === -1 ? '?' : '&'
+                }nocache=${new Date().getTime()}`
+            }
             resolve(nextUrl)
         })
     },
     reloadTimeLayers: function () {
         // refresh time enabled layers
-        var reloadedLayers = []
+        let reloadedLayers = []
         for (let layerName in L_.layers.data) {
             const layer = L_.layers.data[layerName]
             if (
@@ -335,7 +376,7 @@ var TimeControl = {
         return reloadedLayers
     },
     updateLayersTime: function () {
-        var updatedLayers = []
+        let updatedLayers = []
         for (let layerName in L_.layers.data) {
             const layer = L_.layers.data[layerName]
             if (layer.time && layer.time.enabled === true) {

src/essence/Basics/Layers_/Layers_.js

@@ -34,6 +34,7 @@ const L_ = {
         opacity: {}, // opacityArray
         filters: {}, // layerFilters
         nameToUUID: {},
+        refreshIntervals: {}, // In order to reloadLayer
     },
     // ===== Private ======
     //Index -> layer name
@@ -244,7 +245,7 @@ const L_ = {
     //Takes in config layer obj
     //Toggles a layer on and off and accounts for sublayers
     //Takes in a config layer object
-    toggleLayer: async function (s) {
+    toggleLayer: async function (s, skipOrderedBringToFront) {
         if (s == null) return
 
         const wasNeverOn = L_.layers.layer[s.name] === false
@@ -253,7 +254,7 @@ const L_ = {
         if (L_.layers.on[s.name] === true) on = true
         else on = false
 
-        await L_.toggleLayerHelper(s, on)
+        await L_.toggleLayerHelper(s, on, null, null, skipOrderedBringToFront)
 
         Object.keys(L_._onLayerToggleSubscriptions).forEach((k) => {
             L_._onLayerToggleSubscriptions[k](s.name, !on)
@@ -291,7 +292,8 @@ const L_ = {
         s,
         on,
         ignoreToggleStateChange,
-        globeOnly
+        globeOnly,
+        skipOrderedBringToFront
     ) {
         if (s.type !== 'header') {
             if (on) {
@@ -302,7 +304,7 @@ const L_ = {
                     try {
                         $('.drawToolContextMenuHeaderClose').click()
                     } catch (err) {}
-                    L_.Map_.map.removeLayer(L_.layers.layer[s.name])
+                    L_.Map_.rmNotNull(L_.layers.layer[s.name])
                     if (L_.layers.attachments[s.name]) {
                         for (let sub in L_.layers.attachments[s.name]) {
                             switch (L_.layers.attachments[s.name][sub].type) {
@@ -587,7 +589,11 @@ const L_ = {
 
             if (s.type === 'vector') L_._updatePairings(s.name, !on)
 
-            if (!on && s.type === 'vector') {
+            if (
+                !on &&
+                s.type === 'vector' &&
+                skipOrderedBringToFront !== true
+            ) {
                 L_.Map_.orderedBringToFront()
             }
             L_._refreshAnnotationEvents()
@@ -1104,7 +1110,7 @@ const L_ = {
                         })
                         layer.options = savedOptions2
                     }
-                }, 1)
+                }, 100)
             }
         } catch (err) {
             if (layer._icon)
@@ -1638,6 +1644,10 @@ const L_ = {
             }
         }
         L_.layers.opacity[name] = newOpacity
+
+        if (L_.activeFeature?.layer && L_.activeFeature.layerName === name) {
+            L_.highlight(L_.activeFeature.layer)
+        }
     },
     getLayerOpacity: function (name) {
         var l = L_.layers.layer[name]
@@ -3528,6 +3538,51 @@ function parseConfig(configData, urlOnLayers) {
 
             //Create parsed layers named
             L_.layers.data[d[i].name] = d[i]
+
+            if (
+                d[i].time &&
+                d[i].time.enabled === true &&
+                d[i].time.refreshIntervalEnabled === true
+            ) {
+                if (L_.layers.refreshIntervals[d[i].name])
+                    clearInterval(L_.layers.refreshIntervals[d[i].name])
+                L_.layers.refreshIntervals[d[i].name] = setInterval(
+                    async () => {
+                        if (L_.layers.on[d[i].name] === true) {
+                            let savedActiveFeature
+                            if (
+                                L_.activeFeature &&
+                                L_.activeFeature.layerName === d[i].name
+                            ) {
+                                savedActiveFeature = {
+                                    layerName: L_.activeFeature.layerName,
+                                    feature: JSON.parse(
+                                        JSON.stringify(L_.activeFeature.feature)
+                                    ),
+                                }
+                            }
+                            await L_.TimeControl_.reloadLayer(
+                                d[i].name,
+                                false,
+                                false,
+                                true,
+                                true
+                            )
+                            // Reselect activeFeature
+                            if (
+                                savedActiveFeature &&
+                                savedActiveFeature.layerName === d[i].name
+                            ) {
+                                L_.selectFeature(
+                                    savedActiveFeature.layerName,
+                                    savedActiveFeature.feature
+                                )
+                            }
+                        }
+                    },
+                    (d[i].time.refreshIntervalAmount || 60) * 1000
+                )
+            }
             //Save the prevName for easy tracing back
             L_._layersParent[d[i].name] = prevName