Merge pull request #246 from Chicago/dev

Merge Milestone 1.2.0 changes into Master branch

README.md

@@ -4,26 +4,37 @@
 
 OpenGrid an open-source, interactive map platform that allows users to explore multiple data sources in an easy-to-use interface. Developed to support situational awareness, incident monitoring and responses, historical data retrieval, and real-time advanced analytics. Users can perform advanced queries to filter data, search within custom boundaries, or based on the users location. Other GIS data, such as weather and Shapefiles can be overlaid on the map with other data. OpenGrid is natively compatible with desktops and mobile devices.
 
+OpenGrid uses a service layer to retrieve data from an underlying data store.
+
 ## Important Links
-* User Documentation: http://opengrid.readthedocs.io
-* Repository: https://github.com/Chicago/opengrid
+
+* User Documentation: http://docs.opengrid.io
+* Repository for User Interface: https://github.com/Chicago/opengrid
+* Repository for 
 * Planning Documentation and Meeting Notes: https://github.com/Chicago/opengrid/wiki
 * Issues: https://github.com/Chicago/opengrid/issues
 * Mailing List & Discussion: https://groups.google.com/forum/#!forum/opengrid-chicago
 
-##  System Requirements <br>
+##  System Requirements
+
 Web server for deployment and Open Grid Service Package. 
 
-**Required Software**
+### Required Software
 
   * Java Script IDE
   * Node.js
   * Npm
 
-**Browser Requirements:**
+### Browser Requirements:
 
 OpenGrid has been tested on IE 10+, Chrome, Firefox, and Safari on the desktop. It has also been tested on iOS using Safari and Android using the Chrome browser.
 
+### Architecture
+
+This repository contains the code which users interact with in the web browser. OpenGrid depends on a service layer, which then communicates with a data storage layer. Currently, OpenGrid is compatible with [MongoDB](http://mongodb.com) or [Plenario](http://plenar.io) using their respective service layers.
+
+Read the documentation for a description of the service layer and available test instances.
+
 ## Installation
 
 Download OpenGrid Code:

docs/Architecture.md

@@ -0,0 +1,31 @@
+# Application Architecture
+
+OpenGrid consists of a user interface layer that is seen by the user in the web browser and a service layer. The service layer is responsible for interpreting commands from the user interface to an underlying data layer. In turn, the user interface communicates with the service layer through an [Application Programmer Interface (API)](http://docs.opengrid.io/en/latest/OpenGrid%20API/). 
+
+![App Architecture](media/application-architecture-diagram.png)
+
+The service layer share the same API to communicate with the user interface, but are compatible with different data store engines. This approach will make it easier to maintain a consistent user interface, but allow it to be compatible with different databases and APIs as these tend to change or different data storage engines often change over time.
+
+OpenGrid is compatible with [MongoDB](https://www.mongodb.com) or [Plenario](http://www.plenar.io). MongoDB's service layer was created by the City of Chicago and [available in this repository](https://www.github.com/Chicago/opengrid-svc-template). The service layer for Plenario was created by Smart Chicago Collaborative and [available here](https://github.com/smartchicago/opengrid-svc-plenario).
+
+## MongoDB or Plenario?
+
+MongoDB is a flexible, JSON-based data engine that lets you easily store data with minimal concerns about designing a schema. Plenario is a storage engine which quickly and easily stores open data from [Socrata](https://www.socrata.com), such as [City of Chicago](https://data.cityofchicago.org), or CKAN like the [U.S. government](http://catalog.data.gov/dataset) data portals. Plenario makes all of these data available through a simple API.
+
+MongoDB is a great solution for on-site deployments or when using data not readily available on Socrata or CKAN. It is an open-source, supurbly documented database that can be installed on-site and behind a firewall. The City of Chicago, for instance, uses OpenGrid + MongoDB to power an internal version of OpenGrid, dubbed [WindyGrid](http://datasmart.ash.harvard.edu/news/article/chicagos-windygrid-taking-situational-awareness-to-a-new-level-259), that loads data from internal systems.
+
+Plenario quickly ingests data from popular open data portal platforms, Socrata and CKAN. But it is more difficult to upload data from other sources, such as databases. Therefore, pairing OpenGrid with Plenario (with the corresponding service layer) works great to visualize open data. Adding data from an open data portal is simple, see [Plenario's documentation for adding a data set](http://plenar.io/add). An example of OpenGrid + Plenario can be seen at [https://chicago.opengrid.io](https://chicago.opengrid.io). 
+
+## Plenar.io test service layer
+
+A publicly-available instance of the Plenar.io service layer is at [http://opengrid-service-dev-1134290206.us-west-2.elb.amazonaws.com/opengrid-service/](http://opengrid-service-dev-1134290206.us-west-2.elb.amazonaws.com/opengrid-service/)
+
+Use this end-point to test any changes you have made to the OpenGrid application that relies on Plenario, but does not require the setup of the service layer.
+
+## Links
+
+* [OpenGrid user interface repo](https://github.com/Chicago/opengrid)
+* [OpenGrid Serivce layer API](OpenGrid%20API/)
+* [MongoDB-compatible service layer repo](https://github.com/Chicago/opengrid-svc-layer)
+* [Plenario service layer repo](https://github.com/smartchicago/opengrid-svc-plenario)
+* [Plenario serice layer development end-point](http://opengrid-service-dev-1134290206.us-west-2.elb.amazonaws.com/opengrid-service/)

docs/OpenGrid API.md

@@ -94,10 +94,11 @@ The JavaScript Web Token expire after 4 hours. The authentication token needs to
 <p>No response is returned but the authentication token, with key X-AUTH-TOKEN, is appended to the response header such as below:
 </p>
 
-<code> 
-<b>X-AUTH-TOKEN:</b><br>
-<p>eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsImV4cCI6MTQzOTMzNjQwNywianRpIjoiYWRtaW4iLCJyb2xlcyI6Im9wZW5ncmlkX2FkbWlucyIsImZuYW1lIjoiT3BlbkdyaWQiLCJsbmFtZSI6IkFkbWluIn0.nShqceUs52ykIxu0RBRp4vZ8zaQqfdZ2haZn8AWMqyq5GJHRQkddoOaaLtKABktr32C0zha1pMJJBrjuYoPHIQ
-</p></code>
+<b>X-AUTH-TOKEN:</b> <br>
+```
+eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsImV4cCI6MTQzOTMzNjQwNywianRpIjoiYWRtaW4iLCJyb2xlcyI6Im9wZW5ncmlkX2FkbWlucyIsImZuYW1lIjoiT3BlbkdyaWQiLCJsbmFtZSI6IkFkbWluIn0.nShqceUs52ykIxu0RBRp4vZ8zaQqfdZ2haZn8AWMqyq5GJHRQkddoOaaLtKABktr32C0zha1pMJJBrjuYoPHIQ
+
+```
 
 <p>
 This token can be parsed using the <b><i>jwt_decode</i></b> JavaScript Web Token library (See <a href="https://github.com/auth0/jwt-decode"> https://github.com/auth0/jwt-decode</a>)
@@ -796,17 +797,17 @@ It is recommended that this value be URL encoded.
 </table>
 </html>
 
-<p><b>Sample Request</b><br>
-<code>
+<p><b>Sample Request</b></p>
+
+```
 &lt;Service URL&gt;/datasets/twitter/query?q={"$and":[{"text":{"$regex":"happy"}}]}&n=1&opts={"geoFilter":{"type":"MultiPolygon","coordinates":[[[[-87.63304710388184,41.89278978584501],[-87.61206150054932,41.89278978584501],[-87.61206150054932,41.88140002416609],[-87.63304710388184,41.88140002416609],[-87.63304710388184,41.89278978584501]]]]}}
-</code> 
-</p>
+```
 
-<p><b><small>Or when URL encoded:</small></b><br>
-<code>
+
+<p><b><small>Or when URL encoded:</small></b></p>
+```
 &lt;Service URL&gt;/datasets/twitter/query?q=%7B%22$and%22:%5B%7B%22text%22:%7B%22$regex%22:%22happy%22%7D%7D%5D%7D&n=6000&opts=%7B%22geoFilter%22:%7B%22type%22:%22MultiPolygon%22,%22coordinates%22:%5B%5B%5B%5B-87.63304710388184,41.89278978584501%5D,%5B-87.61206150054932,41.89278978584501%5D,%5B-87.61206150054932,41.88140002416609%5D,%5B-87.63304710388184,41.88140002416609%5D,%5B-87.63304710388184,41.89278978584501%5D%5D%5D%5D%7D%7D
-</code>
-</p>
+```
 
 <p><b>Sample Response</b></br>
 <code>

docs/User Documentation.md

@@ -38,15 +38,18 @@ The quick search help feature assist users in performing a valid search. Each se
 <p>Advanced Search allows you to combine search terms by setting specific parameters for your results.It gives the user the ability to narrow their searches by a series of different filters such as adding additional rules, groups,datasets and Geo spatial-filters.</p>
 
 ### Selecting data and date range
-<p>To query by data and date range...Click on add datasets, select the saved dataset from the list <b>(for ex: business license)</b>; click submit. Add a rule or group to your dataset <b><small>(Adding a rule or group gives the user an option to query by date, city, name, address, etc. depending on the search criteria.)</b></small>. Date range can be specified with the parameters of:</p>
+<p>To query by data and date range...Click on add datasets, select a dataset from the list <b>(for ex: business license)</b>; click submit. Add a rule or group to your dataset <b><small>(Adding a rule or group is optional and gives the user an option to query by date, city, name, address, etc. depending on the search criteria.)</b></small>. Date range can be specified with the parameters and relative dates of:</p>
 
-- equal
-- not equal 
-- less
-- less or equal
-- greater 
-- greater or equal
 - between
+- greater 
+- less
+- yesterday
+- tomorrow
+- today
+- day (i.e. a day ago, 2 days ago, etc.)
+- week (i.e. a week ago, 2 weeks ago, etc.)
+- month (i.e. a month ago, 2 months ago, etc.)
+- year (i.e. a year ago, 2 years ago, etc.)
 <br>
 
 ![combine](media/combine21.jpg)
@@ -77,7 +80,7 @@ There are two filters to search by <b>“Within”</b> and <b>“Near”</b>. <b
 ## Table Functions
 <p>The table functionality is used to organized information that has been displayed via query. The table displays the data that was pulled from the queried results and in return structures it in a readable format for the user to interpret. Each dropdown are intertwined with one another.The table consist of five drop down sections: </p>
 
-![table](media/combine33.png)
+![table](media/combine33.jpg)
 #### 1. Columns<br>
 The columns section displays the list of columns from the query data-sets and places them in order by default. The columns can be interchangeable and removed to fit the user preference.
 

lib/jquery-1.11.2/opengrid-query-builder.js

@@ -238,7 +238,8 @@
             add_group: 'glyphicon glyphicon-plus-sign',
             add_rule: 'glyphicon glyphicon-plus',
             remove_group: 'glyphicon glyphicon-remove',
-            remove_rule: 'glyphicon glyphicon-remove',
+            //OpenGrid customization
+            remove_rule: 'glyphicon glyphicon-remove-circle',
             error: 'glyphicon glyphicon-warning-sign'
         }
     };
@@ -1355,6 +1356,20 @@
         return false;
     };
 
+	 QueryBuilder.prototype.isNaturalLangChronoTimeValue = function(value) {
+        var o = chrono.parse(value);
+        if (o.length > 0) {
+            var v = o[0];
+            if (v.tags) {
+                if (v.tags.ENTimeAgoFormatParser)
+                    return true;
+                else
+                    return false;
+            }
+        }
+        return false;
+    };
+
 
     /**
      * Returns an incremented group ID
@@ -2726,20 +2741,23 @@
 
 
         _getRuleValue: function(rule) {
+
             //OpenGrid extension, translate system rule value
             if (rule.type === 'date') {
                 //OpenGrid extension, translate system rule value
                 if (this.isNaturalLangChronoValue(rule.value) ) {
-                    //map to Mongo's current date time token
-                    //this does not work for datetime values stored as numeric
-                    //rule.value = '$currentDate';
-
                     //assume: datetime on target is stored in Mongo as a numeric value representing Unix epoch in ms
-                    return moment(chrono.parseDate(rule.value)).startOf('day').valueOf();
+                    if (this.isNaturalLangChronoTimeValue(rule.value))
+                        //include time value
+                        return moment(chrono.parseDate(rule.value)).valueOf();
+                    else
+                        //start of day (assume > to be only operator)
+                        return moment(chrono.parseDate(rule.value)).startOf('day').valueOf();
                 } else
                     return moment(rule.value, "MM/DD/YYYY hh:mm:ss a").valueOf();
             } else
                 return rule.value;
+
         },
 
         //assume only 'between' operator can be applied
@@ -2748,10 +2766,18 @@
                 //map to Mongo's current date time token
 
                 //assume: datetime on target is stored in Mongo as a numeric value representing Unix epoch in ms
-                if (i==1)
-                    value = moment(chrono.parseDate(value)).endOf('day').valueOf();
-                else
-                    value = moment(chrono.parseDate(value)).startOf('day').valueOf();
+                if (this.isNaturalLangChronoTimeValue(value)) {
+                    //include time value, e.g. 5 hours ago, 30 minutes ago, etc
+                    value =  moment(chrono.parseDate(value)).valueOf();
+                } else {
+                    //between operation
+                    if (i==1)
+                        //2nd argument, use end of day time
+                        value = moment(chrono.parseDate(value)).startOf('day').add(1, 'day').valueOf();
+                    else
+                        //1st argument, use start of day time
+                        value = moment(chrono.parseDate(value)).startOf('day').valueOf();
+                }
                 return value;
             } else {
                 //assume: datetime on target is stored in Mongo as a numeric value representing Unix epoch in ms
@@ -3527,7 +3553,8 @@
         "__author": "Damien \"Mistic\" Sorel, http://www.strangeplanet.fr",
         "add_rule": "Add rule",
         "add_group": "Add group",
-        "delete_rule": "Delete",
+       //OpenGrid customization: removed Delete text
+        "delete_rule": "",
         "delete_group": "Delete",
         "conditions": {
             "AND": "AND",

mkdocs.yml

@@ -10,6 +10,7 @@ pages:
 - User Guide:
     - End-user documentation: User Documentation.md
 - System administration:
+    - App Architecture: Architecture.md
     - Installation and Setup: Installation and Setup.md
     - Build Procedures: Build Procedures.md
     - Configuration: OpenGrid Configuration.md
@@ -22,4 +23,4 @@ markdown_extensions:
         permalink: 
     - admonition:
 
-copyright: Copyright &copy; 2015 <a href="http://www.cityofchicago.org">City of Chicago</a> and licensed under <a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International</a>.
+copyright: Copyright &copy; 2015-2016 <a href="http://www.cityofchicago.org">City of Chicago</a> and licensed under <a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International</a>.

src/css/ogrid-theme-blue.css

@@ -65,14 +65,17 @@ html, body {
 
 
 .query-builder .rules-group-container {
-    padding: 10px 10px 6px;
+   padding: 5px 5px 0px 0px;
+
     border:none !important;
     background: none !important;
 }
 .query-builder .rules-group-container .btn-success {background-color: #28303D !important; border-color:#73C4E1 !important;}
 #ogrid-ds-tabs .regtab a {background-color:transparent !important; border:1px #FFFFFF solid !important; border-radius: 10px; }
-.query-builder .rules-group-container .btn-danger {color: #28303D; background-color:transparent !important; border-color: #28303D;}
+.query-builder .rules-group-container .btn-danger {color: #28303D; background-color:transparent !important; border: none;
+    font-size: 1.5em;}
 #advSearchSave {color:#fff !important; background-color:#28303D !important; border-color:#2E6DA4 !important;}
+.query-builder .rules-group-container .btn-danger:hover {opacity: 0.6; }
 
 	::-webkit-scrollbar-button:start:decrement,
     ::-webkit-scrollbar-button:end:increment { height: 0px; display: block; background-color: black;}
@@ -181,13 +184,25 @@ html, body {
 }
 
 #advSearchSubmit, #advSearchReset, #advSearchSave {
-    width: 100px;
+    width: 120px;
 }
 
+
 #advSearchSubmit {
-    margin-left: 10px;
+    margin-left: 35px;
+	position: fixed;
+	bottom: 35px;
+}
+
+ #advSearchReset {
+	position: fixed;
+	bottom: 35px;
+	margin-left: 165px;
 }
 
+.ogrid-color-panel {
+    padding: 0px 10px 0px 10px !important;
+}
 
 
 #autorefresh-panel .control-label {
@@ -258,7 +273,10 @@ html, body {
 .ogrid-task-pane .rule-value-container .form-control {
     font-weight: 300;
 }
-
+.ogrid-color-options {
+    font-size: 12px !important;
+    text-decoration: underline;
+}
 
 div#ogrid-dataset-filters.panel-body,  div#ogrid-geo-filters.panel-body {
     padding-left: 0px;
@@ -276,6 +294,17 @@ div#ogrid-dataset-filters.panel-body,  div#ogrid-geo-filters.panel-body {
     font-size: 152%;
 }
 
+.adv-bottom-commandbar {
+    background-color: #2480a0;
+    margin-left: -25px;
+    position: fixed;
+    bottom: 0px;
+    height: 80px;
+    width: 100%;
+    opacity: 1.0;
+    z-index: 8000;
+}
+
 /* Size when the advanced tabs header breaks */
 @media (max-width: 342px) {
     #ogrid-adv-tabs a {
@@ -315,4 +344,36 @@ div#ogrid-dataset-filters.panel-body,  div#ogrid-geo-filters.panel-body {
         padding-left: 10px;
         font-size: 1.5em;
     }
+}
+
+.header-center {
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    height: 25px;
+    margin-right: 0 !important;
+    margin-top: 0 !important;
+}
+
+#ogrid-footer .panel-title a {
+    padding-top: 0;
+    padding-bottom: 0;
+}
+
+.ogrid-footer-panel {
+    border-width: 0;
+}
+
+.ogrid-footer-panel-heading, .ogrid-footer-panel-heading .panel-title {
+    color: #fff !important;
+    background-color: #292e35 !important;
+    border-width: 0 !important;
+}
+.ogrid-nav-btn .active {
+    background-color: rgba(129, 131, 132, 0.97) !important;
+}
+
+.leaflet-control-attribution {
+    /* adjusted due to increase in search results panel height */
+    margin-bottom: 4px !important;
 }