Skip to content

Commit

Permalink
Merge pull request #52 from PermafrostDiscoveryGateway/develop
Browse files Browse the repository at this point in the history
Develop
  • Loading branch information
julietcohen authored Jun 4, 2024
2 parents 6d0aaeb + 55751df commit 321c05e
Show file tree
Hide file tree
Showing 10 changed files with 329 additions and 60 deletions.
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -129,4 +129,7 @@ dmypy.json
.pyre/

# macOS
.DS_Store
.DS_Store

# Poetry
poetry.lock
7 changes: 7 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"python.testing.pytestArgs": [
"tests"
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true
}
204 changes: 204 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Contributing

:tada: First off, thanks for contributing! :tada:

- [Types of contributions](#types-of-contributions)
- [Pull Requests](#pull-requests)
- [Development Workflow](#development-workflow)
- [Release process](#release-process)
- [Testing](#testing)
- [Code style](#code-style)
- [Contributor license agreement](#contributor-license-agreement)

## Types of contributions

We welcome all types of contributions, including bug fixes, feature enhancements,
bug reports, documentation, graphics, and many others. You might consider contributing by:

- Report a bug or request a new feature in our [issue tracker](https://github.com/PermafrostDiscoveryGateway/viz-staging/issues)
- Fix a bug and contribute the code with a Pull Request
- Write or edit some documentation
- Sharing helpful tips or FAQ-type answers to users or future contributors
- ...

This is an open source project, and we welcome full
participation in the project. Contributions are reviewed and suggestions are
made to increase the value of this software to the community. We strive to
incorporate code, documentation, and other useful contributions quickly and
efficiently while maintaining a high-quality software product.

## Pull Requests
We use the pull-request model for contributions. See [GitHub's help on pull-requests](https://help.github.com/articles/about-pull-requests/).

In short:

- add an [issue](https://github.com/PermafrostDiscoveryGateway/viz-staging/issues) describing your planned changes, or add a comment to an existing issue;
- on GitHub, fork the [repository](https://github.com/PermafrostDiscoveryGateway/viz-staging)
- on your computer, clone your forked copy of the repository
- base your work on the `develop` branch and commit your changes
- push your branch to your forked repository, and submit a pull-request
- our team will be notified of your Pull Request and will review your changes
- our team may request changes before we will approve the Pull Request, or we will make them for you
- once the code is reviewed, our team will merge in your changes to `develop` for the next planned release

## Development Workflow

Development is managed through the git repository at https://github.com/PermafrostDiscoveryGateway/viz-staging. The repository is organized into several branches, each with a specific purpose.

**main**. The `main` branch represents the stable branch that is constantly maintained with the current release. It should generally be safe to install and use the `main` branch the same way as binary releases. The version number in all configuration files and the README on the `main` branch follows [semantic versioning](https://semver.org/) and should always be set to the current stable release, for example `2.8.5`.

**develop**. Development takes place on a single branch for integrated development and testing of the set of features
targeting the next release. Commits should only be pushed to this branch once they are ready to be deployed to
production immediately after being pushed. This keeps the `develop` branch in a state of readiness for the next release.
Any unreleased code changes on the `develop` branch represent changes that have been tested and staged for the next
release.
The tip of the `develop` branch always represents the set of features that are awaiting the next release. The develop
branch represents the opportunity to integrate changes from multiple features for integrated testing before release.

Version numbers on the `develop` branch represent either the planned next release number (e.g., `2.9.0`), or the planned next release number with a `beta` designator or release candidate `rc` designator appended as appropriate. For example, `2.8.6-beta1` or `2.9.0-rc1`.

**feature**. To isolate development on a specific set of capabilities, especially if it may be disruptive to other
developers working on the `develop` branch, feature branches should be created.

Feature branches are named as `feature-` + `{issue}` + `-{short-description}`, with `{issue}` being the GitHub issue number related to that new feature. e.g. `feature-23-refactor-storage`.

All `feature-*` branches should be frequently merged with changes from `develop` to
ensure that the branch stays up to date with other features that have
been tested and are awaiting release. Thus, each `feature-*` branch can be tested on its own before it is merged with other features on develop, and afterwards as well. Once a feature is complete and ready for full integration testing, it is generally merged into the `develop` branch after review through a pull request.

**bugfix**. A final branch type are `bugfix` branches, which work the same as feature branches, but fix bugs rather than adding new functionality. Sometimes it is hard to distinguish features from bug fixes, so some repositories may choose to use `feature` branches for both types of change. Bugfix branches are named similarly, following the pattern: `bugfix-` + `{issue}` + `-{short-description}`, with `{issue}` being the GitHub issue number related to that bug. e.g. `bugfix-83-fix-name-display`.

### Development flow overview

```mermaid
%%{init: { 'theme': 'base',
'gitGraph': {
'rotateCommitLabel': false,
'showCommitLabel': false
},
'themeVariables': {
'commitLabelColor': '#ffffffff',
'commitLabelBackground': '#000000'
}
}}%%
gitGraph
commit id: "1" tag: "v1.0.0"
branch develop
checkout develop
commit id: "2"
branch feature-A
commit id: "3"
commit id: "4"
checkout develop
merge feature-A id: "5"
commit id: "6"
commit id: "7"
branch feature-B
commit id: "8"
commit id: "9"
checkout develop
merge feature-B id: "10" type: NORMAL
checkout main
merge develop id: "11" tag: "v1.1.0"
```

## Release process

1. Our release process starts with integration testing in a `develop` branch. Once all
changes that are desired in a release are merged into the `develop` branch, we run
the full set of tests on a clean checkout of the `develop` branch.
2. After testing, the `develop` branch is merged to main, and the `main` branch is tagged with
the new version number (e.g. `2.11.2`). At this point, the tip of the `main` branch will
reflect the new release and the `develop` branch can be fast-forwarded to sync with `main` to
start work on the next release.
3. Releases can be downloaded from the [GitHub releases page](https://github.com/PermafrostDiscoveryGateway/viz-staging/releases).

## Testing

**Unit and integration tests**. We strive for a full suite of `pytest` tests in the `tests` subdirectory.
Any new code developed should include a robust set of tests for each public
method, as well as integration tests from new feature sets. Tests should fully
exercise the feature to ensure that it responds correctly to both good data inputs
and various classes of corrupt or bad data. All tests should pass before submitting a PR
or merging to `develop`.

Tests are automatically run via GitHub Actions. Check the root `README.md` file
for this GitHub Actions status badge and make sure it says "Passing":

## Code style

Code should be written to professional standards to enable clean, well-documented,
readable, and maintainable software. While there has been significant variability
in the coding styles applied historically, new contributions should strive for
clean code formatting. We generally follow PEP8 guidelines for Python code formatting,
typically enforced through the `black` code formatting package.

## Contributor license agreement

In order to clarify the intellectual property license
granted with Contributions from any person or entity, you agree to
a Contributor License Agreement ("CLA") with the Regents of the University of
California (hereafter, the "Regents").

1. Definitions.
"You" (or "Your") shall mean the copyright owner or legal entity
authorized by the copyright owner that is making this Agreement
with the Regents. For legal entities, the entity making a
Contribution and all other entities that control, are controlled
by, or are under common control with that entity are considered to
be a single Contributor. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"Contribution" shall mean any original work of authorship,
including any modifications or additions to an existing work, that
is intentionally submitted by You to the Regents for inclusion
in, or documentation of, any of the products owned or managed by
the Regents (the "Work"). For the purposes of this definition,
"submitted" means any form of electronic, verbal, or written
communication sent to the Regents or its representatives,
including but not limited to communication on electronic mailing
lists, source code control systems, and issue tracking systems that
are managed by, or on behalf of, the Regents for the purpose of
discussing and improving the Work, but excluding communication that
is conspicuously marked or otherwise designated in writing by You
as "Not a Contribution."
2. Grant of Copyright License. Subject to the terms and conditions of
this Agreement, You hereby grant to the Regents and to
recipients of software distributed by the Regents a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare derivative works of,
publicly display, publicly perform, sublicense, and distribute Your
Contributions and such derivative works.
3. Grant of Patent License. Subject to the terms and conditions of
this Agreement, You hereby grant to the Regents and to
recipients of software distributed by the Regents a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have
made, use, offer to sell, sell, import, and otherwise transfer the
Work, where such license applies only to those patent claims
licensable by You that are necessarily infringed by Your
Contribution(s) alone or by combination of Your Contribution(s)
with the Work to which such Contribution(s) was submitted. If any
entity institutes patent litigation against You or any other entity
(including a cross-claim or counterclaim in a lawsuit) alleging
that your Contribution, or the Work to which you have contributed,
constitutes direct or contributory patent infringement, then any
patent licenses granted to that entity under this Agreement for
that Contribution or Work shall terminate as of the date such
litigation is filed.
4. You represent that you are legally entitled to grant the above
license. If your employer(s) has rights to intellectual property
that you create that includes your Contributions, you represent
that you have received permission to make Contributions on behalf
of that employer, that your employer has waived such rights for
your Contributions to the Regents, or that your employer has
executed a separate Corporate CLA with the Regents.
5. You represent that each of Your Contributions is Your original
creation (see section 7 for submissions on behalf of others). You
represent that Your Contribution submissions include complete
details of any third-party license or other restriction (including,
but not limited to, related patents and trademarks) of which you
are personally aware and which are associated with any part of Your
Contributions.
49 changes: 45 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,27 @@
# PDG Staging
# Viz-staging: vector data tiling for geospatial visualization

Divides vector files into tiled vector files according to a specified [OGC Two Dimensional Tile Matrix Set](http://docs.opengeospatial.org/is/17-083r2/17-083r2.html) in preparation for processing into other formats in the PDG workflow.
- **Authors**: Robyn Thiessen-Bock ; Juliet Cohen ; Matthew B. Jones ; Kastan Day ; Lauren Walker
- **DOI**: [10.18739/A2RV0D26C](https://ezid.cdlib.org/id/doi:10.18739/A2RV0D26C)
- **License**: [Apache 2](https://opensource.org/license/apache-2-0/)
- [Package source code on GitHub](https://github.com/PermafrostDiscoveryGateway/viz-staging)
- [Submit bugs and feature requests](https://github.com/PermafrostDiscoveryGateway/viz-staging/issues/new)

The `pdgstaging` package divides vector files into tiled vector files according to a specified [OGC Two Dimensional Tile Matrix Set](http://docs.opengeospatial.org/is/17-083r2/17-083r2.html) in preparation for processing into other formats in the Permafrost Discovery Gateway (PDG) workflow.

![PDG staging summary](docs/images/staging_tldr.png)

## Citation

Cite this software as:

> Robyn Thiessen-Bock, Juliet Cohen, Matt Jones, Kastan Day, Lauren Walker. 2023. Viz-staging: vector data tiling for geospatial visualization (version 0.9.1). Arctic Data Center. doi: 10.18739/A2RV0D26C
## Install

Requires Python version `3.9` and `libspatialindex` or `libspatialindex-dev`
Requires Python version `3.9` or `3.10` and `libspatialindex` or `libspatialindex-dev`

1. Follow the instructions to install [`libspatialindex`](https://libspatialindex.org/en/latest/) or [`libspatialindex-dev`](https://packages.ubuntu.com/bionic/libspatialindex-dev)
2. Make sure that Python version 3.9 is installed (try `which python3.9`).
2. Make sure that Python version 3.9 or 3.10 is installed (try `which python3.9`).
3. Install `pdgstaging` from GitHub repo using pip: `pip install git+https://github.com/PermafrostDiscoveryGateway/viz-staging.git`

## Usage
Expand All @@ -20,6 +32,7 @@ Requires Python version `3.9` and `libspatialindex` or `libspatialindex-dev`
- run: `python -m pdgstaging -c '/path/to/config.json'`

**In python:**

```python
import pdgstaging
stager = pdgstaging.TileStager('/path/to/config.json')
Expand All @@ -29,6 +42,8 @@ stager.stage_all()
stager.stage('path/to/input/file.shp')
```

See more example code in [`PermafrostDiscoveryGateway/viz-info/helpful-code`](https://github.com/PermafrostDiscoveryGateway/viz-info/tree/main/helpful-code)

## Vector file staging for the PDG tiling pipeline

This repository contains code that prepares vector data (e.g. shapefiles, geopackages) for subsequent steps in the [PDG](https://permafrost.arcticdata.io/) tiling pipeline (such as [viz-3dtiles](https://github.com/PermafrostDiscoveryGateway/viz-3dtiles) and [viz-raster](https://github.com/PermafrostDiscoveryGateway/viz-raster)). The staging step creates output vector files that conform to a specified [OGC Two Dimensional Tile Matrix Set](http://docs.opengeospatial.org/is/17-083r2/17-083r2.html) ("TMS"). Specifically, for each input file, the staging process:
Expand Down Expand Up @@ -78,3 +93,29 @@ The staging process will also output a summary CSV file with one row for each ti
- If the deduplication method specified in the configuration is `footprints`, the footprint file(s) are provided with a structure that follows the [docs](https://github.com/PermafrostDiscoveryGateway/viz-staging/blob/main/docs/footprints.md).
- In order for logging to work properly, the node running the script that uses this package has a `/tmp` directory so the `log.log` file can populate there.

## Development

Build and test using poetry and pytest.

- To build, run `poetry build`
- To test, run `pytest` from the root of the package directory
- VS Code configuration is setup to configure tests as well

## License

```
Copyright [2013] [Regents of the University of California]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

8 changes: 3 additions & 5 deletions pdgstaging/ConfigManager.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,9 +275,7 @@ class ConfigManager():
the intersecting polygons to be considered a duplicate. If
False, then the overlap_tolerance proportion must be True
for only one of the intersecting polygons to be considered
a duplicate. Default is True. Note that with release 0.9.0,
the 'neighbor' method has been not been thoroughly tested
and should not be applied to input data.
a duplicate. Default is True.
- deduplicate_centroid_tolerance : float, optional
For the 'neighbor' deduplication method only. The maximum
distance between the centroids of two polygons to be
Expand All @@ -299,8 +297,8 @@ class ConfigManager():
- deduplicate_clip_to_footprint : bool, optional
For the 'footprints' deduplication method only. If True,
then polygons that fall outside the bounds of the
associated footprint will be removed. Default is True for
release version 0.9.0, but will be false for future releases.
associated footprint will be removed. Default is True for this
release, but will be false for future releases.
- deduplicate_clip_method: str, optional
For the 'footprints' deduplication method only, when
deduplicate_clip_to_footprint is True. The method to use to
Expand Down
6 changes: 3 additions & 3 deletions pdgstaging/Deduplicator.py
Original file line number Diff line number Diff line change
Expand Up @@ -478,8 +478,8 @@ def deduplicate_by_footprint(
return_intersections : bool, optional
If true, the polygons that represent the intersections between
footprints will be returned. Default is False. Not currently available
in this release 0.9.0. return_intersections is to be integrated again
in future releases.
in this release. return_intersections is to be integrated again
in a future release.
prop_duplicated : str, optional
Defaults to "staging_duplicated". The column name / property to use to
flag duplicates.
Expand All @@ -494,7 +494,7 @@ def deduplicate_by_footprint(
`intersections` represents the polygon area where the footprints overlap.
It has not been integrated into the function again since the deduplication
approach changed from returning a dictionary to returning a labeled GDF.
This will be integrated again in releases after 0.9.0.
This will be integrated again in a future release.
"""

logger.info(f"Executing footprint deduplication.")
Expand Down
Loading

0 comments on commit 321c05e

Please sign in to comment.