DataStream Open Data Schema for Water Quality Data

Download the latest version in JSON Schema, Google Sheets, Excel template formats.

DataStream

DataStream (DataStream.org) is an online open-access platform for sharing water quality and sediment quality data. Data is uploaded, stored and shared in DataStream’s Open Data Schema -- a model based on the WQX standard for the Exchange of Water Quality Data. DataStream is free to use and allows users to query, visualize, and download data in this standardized format. Water monitoring groups across Canada — including communities, researchers, Indigenous nations and governments at all levels — are using DataStream to publish their data.

DataStream was developed by The Gordon Foundation and is carried out in collaboration with regional partners and monitoring networks. Data contributors maintain ownership of their data which are published under open data licenses.

DataStream Open Data Schema (DS-WQX)

To ensure consistent formatting of water data and to avoid ambiguous or missing information, we developed an observation-level data schema based on the WQX standard for the Exchange of Water Quality Data.

The WQX schema was developed by the US Environmental Protection Agency (EPA) and the US Geological Society (USGS) and is an implementation of the ESAR (Environmental Sampling, Analysis and Results) data standard. It was designed to enable multiple monitoring entities to share results in a common format. In the US, the WQX schema is used on the US EPA’s Water Quality Portal to share over 340 million water quality data records data from 400 federal, state, tribal and other partners.

In 2018, The Gordon Foundation led a comprehensive review process to determine how this model could be adapted to best meet the needs of diverse water monitoring initiatives in Canada. DataStream’s science and technical advisory team, regional partners and collaborators, data contributors, government representatives as well as other members of the water community were engaged in this process.

DataStream’s open data schema (DS-WQX v1.0) was fully implemented across all regional DataStream platforms in 2019.

Changelog

The DataStream open data schema will continue to evolve to meet user needs and therefore is subject to various updates over time (e.g. addition of new allowed values). To view the most recent version number and record of changes please see the DataStream Upload Template.

Dataset Metadata

In addition to the observation-level information in DS-WQX, DataStream uses dataset level metadata. DataStream Metadata

Special Case Tests

In addition to our schema enforcing allowed values the column conditional logic; we have included additional check for common errors to the frontend flavour of our schema.

• Dissolved oxygen (DO) should not be in %
• Dissolved oxygen saturation should not be less than 0%
• Hardness should not be less than or equal to 0
• pH should be within 0 and 14
• Temperature should be within -100 degC and 100 degC

Testing

We aim to have our test be as robust as possible. This is accomplished by having acceptance and rejection tests.

Install

You can download the compiled DS-WQX schema from above.

Alternately, you can build it from the source to include in your project.

# Public
$ npm i
$ npm run build
$ cat primary.json

# Private
$ npm i @datastreamapp/schema

Use

CSV Template

The csv template follows R import/export best practices.

JavaScript

import validate from '@datastreamapp/schema'
const data = {}; // Single row of data
const valid = validate(data);
if (!valid) console.error(validate.errors)

Schema Flavours

Supports JSON Schema Draft 2019-09 Specification in non-strict mode. Strict mode removes if, then, additionalProperties from the schema.

• primary: This includes only JSON schema specification supported parameters
• frontend: Includes value coercion and supplementary conditional checks
• backend: Includes all possible allowed values, value coercion and no conditional logic.

Publish

# change version in package.json
npm test
cd dist
npm publish

Contributing

Commenting on the Schema

If you wish to comment on the schema please open an issue.

• For information on opening an issue review github's creating an issue document

You can also email us at team@datastream.org or visit us at:

• DataStream
• Mackenzie DataStream
• Atlantic DataStream
• Lake Winnipeg DataStream
• Great Lakes DataStream
• Pacific DataStream

Development

brew install nvm
nvm get 12
npm i

Publishing

# update version in `package.json`
npm run test
cd dist
npm publish

References

• WQX
• R Import/Export

Citation

If you use this schema, please cite it as below.

DataStream Initiative. ([Year]). DataStream WQX Schema ([current version]). GitHub. https://github.com/datastreamapp/schema