diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..8dc6c8b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,192 @@ +# Contributing + +:tada: First off, thanks for contributing! :tada: + +- [Types of contributions](#types-of-contributions) +- [Pull Requests](#pull-requests) +- [Development Workflow](#development-workflow) +- [Release process](#release-process) +- [Testing](#testing) +- [Code style](#code-style) +- [Contributor license agreement](#contributor-license-agreement) + +## Types of contributions + +We welcome all types of contributions, including bug fixes, feature enhancements, +bug reports, documentation, graphics, and many others. You might consider contributing by: + +- Report a bug or request a new feature in our [issue tracker](https://github.com/PermafrostDiscoveryGateway/viz-staging/issues) +- Fix a bug and contribute the code with a Pull Request +- Write or edit some documentation +- Sharing helpful tips or FAQ-type answers to users or future contributors +- ... + +This is an open source project, and we welcome full +participation in the project. Contributions are reviewed and suggestions are +made to increase the value of this software to the community. We strive to +incorporate code, documentation, and other useful contributions quickly and +efficiently while maintaining a high-quality software product. + +## Pull Requests +We use the pull-request model for contributions. See [GitHub's help on pull-requests](https://help.github.com/articles/about-pull-requests/). + +In short: + +- add an [issue](https://github.com/PermafrostDiscoveryGateway/viz-staging/issues) describing your planned changes, or add a comment to an existing issue; +- on GitHub, fork the [repository](https://github.com/PermafrostDiscoveryGateway/viz-staging) +- on your computer, clone your forked copy of the repository +- base your work on the `develop` branch and commit your changes +- push your branch to your forked repository, and submit a pull-request +- our team will be notified of your Pull Request and will review your changes +- our team may request changes before we will approve the Pull Request, or we will make them for you +- once the code is reviewed, our team will merge in your changes to `develop` for the next planned release + +## Development Workflow + +Development is managed through the git repository at https://github.com/PermafrostDiscoveryGateway/viz-workflow. The repository is organized into several branches, each with a specific purpose. + +**main**. The `main` branch represents the stable branch that is constantly maintained with the current release. It should generally be safe to install and use the `main` branch the same way as binary releases. The version number in all configuration files and the README on the `main` branch follows [semantic versioning](https://semver.org/) and should always be set to the current stable release, for example `2.8.5`. + +**develop**. Development takes place on a single branch for integrated development and testing of the set of features +targeting the next release. Commits should only be pushed to this branch once they are ready to be deployed to +production immediately after being pushed. This keeps the `develop` branch in a state of readiness for the next release. +Any unreleased code changes on the `develop` branch represent changes that have been tested and staged for the next +release. +The tip of the `develop` branch always represents the set of features that are awaiting the next release. The develop +branch represents the opportunity to integrate changes from multiple features for integrated testing before release. + +Version numbers on the `develop` branch represent either the planned next release number (e.g., `2.9.0`), or the planned next release number with a `beta` designator or release candidate `rc` designator appended as appropriate. For example, `2.8.6-beta1` or `2.9.0-rc1`. + +**feature**. To isolate development on a specific set of capabilities, especially if it may be disruptive to other +developers working on the `develop` branch, feature branches should be created. + +Feature branches are named as `feature-` + `{issue}` + `-{short-description}`, with `{issue}` being the GitHub issue number related to that new feature. e.g. `feature-23-refactor-storage`. + +All `feature-*` branches should be frequently merged with changes from `develop` to +ensure that the branch stays up to date with other features that have +been tested and are awaiting release. Thus, each `feature-*` branch can be tested on its own before it is merged with other features on develop, and afterwards as well. Once a feature is complete and ready for full integration testing, it is generally merged into the `develop` branch after review through a pull request. + +**bugfix**. A final branch type are `bugfix` branches, which work the same as feature branches, but fix bugs rather than adding new functionality. Sometimes it is hard to distinguish features from bug fixes, so some repositories may choose to use `feature` branches for both types of change. Bugfix branches are named similarly, following the pattern: `bugfix-` + `{issue}` + `-{short-description}`, with `{issue}` being the GitHub issue number related to that bug. e.g. `bugfix-83-fix-name-display`. + +### Development flow overview + +```mermaid +%%{init: { 'theme': 'base', + 'gitGraph': { + 'rotateCommitLabel': false, + 'showCommitLabel': false + }, + 'themeVariables': { + 'commitLabelColor': '#ffffffff', + 'commitLabelBackground': '#000000' + } +}}%% +gitGraph + commit id: "1" tag: "v1.0.0" + branch develop + checkout develop + commit id: "2" + branch feature-A + commit id: "3" + commit id: "4" + checkout develop + merge feature-A id: "5" + commit id: "6" + commit id: "7" + branch feature-B + commit id: "8" + commit id: "9" + checkout develop + merge feature-B id: "10" type: NORMAL + checkout main + merge develop id: "11" tag: "v1.1.0" +``` + +## Release process + +1. Our release process starts with integration testing in a `develop` branch. Once all +changes that are desired in a release are merged into the `develop` branch, we run +the full set of tests on a clean checkout of the `develop` branch. +2. After testing, the `develop` branch is merged to main, and the `main` branch is tagged with +the new version number (e.g. `2.11.2`). At this point, the tip of the `main` branch will +reflect the new release and the `develop` branch can be fast-forwarded to sync with `main` to +start work on the next release. +3. Releases can be downloaded from the [GitHub releases page](https://github.com/PermafrostDiscoveryGateway/viz-staging/releases). + +## Code style + +Code should be written to professional standards to enable clean, well-documented, +readable, and maintainable software. While there has been significant variability +in the coding styles applied historically, new contributions should strive for +clean code formatting. We generally follow PEP8 guidelines for Python code formatting, +typically enforced through the `black` code formatting package. + +## Contributor license agreement + +In order to clarify the intellectual property license +granted with Contributions from any person or entity, you agree to +a Contributor License Agreement ("CLA") with the Regents of the University of +California (hereafter, the "Regents"). + +1. Definitions. + "You" (or "Your") shall mean the copyright owner or legal entity + authorized by the copyright owner that is making this Agreement + with the Regents. For legal entities, the entity making a + Contribution and all other entities that control, are controlled + by, or are under common control with that entity are considered to + be a single Contributor. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + "Contribution" shall mean any original work of authorship, + including any modifications or additions to an existing work, that + is intentionally submitted by You to the Regents for inclusion + in, or documentation of, any of the products owned or managed by + the Regents (the "Work"). For the purposes of this definition, + "submitted" means any form of electronic, verbal, or written + communication sent to the Regents or its representatives, + including but not limited to communication on electronic mailing + lists, source code control systems, and issue tracking systems that + are managed by, or on behalf of, the Regents for the purpose of + discussing and improving the Work, but excluding communication that + is conspicuously marked or otherwise designated in writing by You + as "Not a Contribution." +2. Grant of Copyright License. Subject to the terms and conditions of + this Agreement, You hereby grant to the Regents and to + recipients of software distributed by the Regents a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare derivative works of, + publicly display, publicly perform, sublicense, and distribute Your + Contributions and such derivative works. +3. Grant of Patent License. Subject to the terms and conditions of + this Agreement, You hereby grant to the Regents and to + recipients of software distributed by the Regents a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have + made, use, offer to sell, sell, import, and otherwise transfer the + Work, where such license applies only to those patent claims + licensable by You that are necessarily infringed by Your + Contribution(s) alone or by combination of Your Contribution(s) + with the Work to which such Contribution(s) was submitted. If any + entity institutes patent litigation against You or any other entity + (including a cross-claim or counterclaim in a lawsuit) alleging + that your Contribution, or the Work to which you have contributed, + constitutes direct or contributory patent infringement, then any + patent licenses granted to that entity under this Agreement for + that Contribution or Work shall terminate as of the date such + litigation is filed. +4. You represent that you are legally entitled to grant the above + license. If your employer(s) has rights to intellectual property + that you create that includes your Contributions, you represent + that you have received permission to make Contributions on behalf + of that employer, that your employer has waived such rights for + your Contributions to the Regents, or that your employer has + executed a separate Corporate CLA with the Regents. +5. You represent that each of Your Contributions is Your original + creation (see section 7 for submissions on behalf of others). You + represent that Your Contribution submissions include complete + details of any third-party license or other restriction (including, + but not limited to, related patents and trademarks) of which you + are personally aware and which are associated with any part of Your + Contributions. diff --git a/README.md b/README.md index 6fe006d..ccfeb33 100644 --- a/README.md +++ b/README.md @@ -1,81 +1,28 @@ -# PDG Ray Workflow +# Viz-workflow: the Permafrost Discovery Gateway geospatial data visualization workflow -Permafrost Discovery Gateway visualization workflow that uses [viz-staging](https://github.com/PermafrostDiscoveryGateway/viz-staging), [viz-raster](https://github.com/PermafrostDiscoveryGateway/viz-raster/tree/main), and [viz-3dtiles](https://github.com/PermafrostDiscoveryGateway/viz-3dtiles) in parallel using Ray Core and Ray workflows. +- **Authors**: Robyn Thiessen-Bock ; Juliet Cohen ; Matthew B. Jones ; Kastan Day ; Lauren Walker +- **DOI**: [10.18739/A2NS0M04C](https://ezid.cdlib.org/id/doi:10.18739/A2NS0M04C) +- **License**: [Apache 2](https://opensource.org/license/apache-2-0/) +- [Package source code on GitHub](https://github.com/PermafrostDiscoveryGateway/viz-workflow) +- [Submit bugs and feature requests](https://github.com/PermafrostDiscoveryGateway/viz-workflow/issues/new) -## Running the Workflow +The Permafrost Discovery Gateway visualization workflow uses [viz-staging](https://github.com/PermafrostDiscoveryGateway/viz-staging), [viz-raster](https://github.com/PermafrostDiscoveryGateway/viz-raster/tree/main), and [viz-3dtiles](https://github.com/PermafrostDiscoveryGateway/viz-3dtiles) in parallel using Ray Core and Ray workflows. An alternative workflow that uses `Docker` and `parsl` for parallelization is currently under development. -For release 0.9.0, scripts must be executed in a particular order for staging, rasterization, and web-tiling steps. 3d-tiling has not been tested for this release. +![PDG workflow summary](docs/images/viz_workflow.png) -- ssh into the Delta server +## Citation -- Pull updates from the `main` branch for each of the 4 PDG repositories: - - [`PermafrostDiscoveryGateway/viz-workflow`](https://github.com/PermafrostDiscoveryGateway/viz-workflow/tree/main) - - [`PermafrostDiscoveryGateway/viz-staging`](https://github.com/PermafrostDiscoveryGateway/viz-staging) - - [`PermafrostDiscoveryGateway/viz-raster`](https://github.com/PermafrostDiscoveryGateway/viz-raster) - - [`PermafrostDiscoveryGateway/viz-3dtiles`](https://github.com/PermafrostDiscoveryGateway/viz-3dtiles) (Note: that release 0.9.0, 3D-tiling has not been fully implemented) +Cite this software as: -- Create a virtual environment with `python=3.9` and install __local__ versions of the PDG packages (using `pip install -e {LOCAL PACKAGE}`). Also install the packages: - - `ray` - - `glances` - - `pyfastcopy` +> Robyn Thiessen-Bock, Juliet Cohen, Matthew B. Jones, Kastan Day, Lauren Walker. 2023. Viz-workflow: the Permafrost Discovery Gateway geospatial data visualization workflow (version 0.9.2). Arctic Data Center. doi: 10.18739/A2NS0M04C -- Prepare one of the two `slurm` scripts to claim some worker nodes on which we will launch a job. - - Open the appropriate script that will soon be run to claim the nodes: either `viz-workflow/slurm/BEST-v2_gpu_ray.slurm` if you're using GPU, or `BEST_cpu_ray_double_srun.slurm` for CPU. - - **Change the line `#SBATCH --nodes={NUMBER}`** which represents the number of nodes that will process the IWP data. - - **Change the line `#SBATCH --time=24:00:00`** which represents the total amount of time a job is allowed to run (and charge credits based on minutes and cores) before it is cancelled. The full 24 hours should be set if doing a full IWP run. - - **Change the line `#SBATCH --account={ACCOUNT NAME}`** and enter the account name for the appropriate allocation. Note that we do __not__ store these private account names on GitHub, so pay attention to this line when you are pushing. - - **Find the `# global settings section` and change `conda activate {ENVIRONMENT}` or `source path/to/{ENVIRONMENT}/bin/activate`** by entering your virtual environment for this workflow. +## Usage -- Open a new terminal, start a `tmux` session, then activate your virtual environment. - -- Switch into the slurm dir by running `cd viz-workflow/slurm`. Then run `sbatch BEST-v2_gpu_ray.slurm` or `sbatch BEST_cpu_ray_double_srun.slurm` to launch the job on the number of nodes you specified within that file. - -- Sync the most recent footprints from `/scratch` to all relevant nodes' `/tmp` dirs on Delta. Within a `tmux` session, switch into the correct environment, and ssh into the head node (e.g. `ssh gpub059`) and run `python viz-workflow/rsync_footprints_to_nodes.py`. - -- Adjust `viz-workflow/PRODUCTION_IWP_CONFIG.py` as needed: - - Change the variable `head_node` to the head node. - - Specify the `INPUT` path and `FOOTPRINTS_REMOTE` paths. - - Specify the `OUTPUT` path (which serves as the _start_ of the path for all output files). This includes a variable `output_subdir` which should be changed to something unique, such as any subfolders the user wants the data to be written to. Create this folder manually. - - Within the subdir just created, created a subdirectory called `staged`. - -- Adjust `viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py` as needed: - - Within the `try:` part of the first function `main()`, comment out / uncomment out steps depending on your stage in the workflow. `step0_staging()` is first, so only uncomment this step. - -- Within the `tmux` session with your virtual environment activated, ssh into the head node associated with that job by running `ssh gpub059` or `ssh cn059`, for example. Then run: `python viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py`. - -- Once staging is complete, run `python viz-workflow/rsync_staging_to_scratch.py`. - -- Adjust the file `merge_staged_vector_tiles.py` as needed: - - Set the variables `staged_dir_path_list` and `merged_dir_path`: - - Change the last string part of `merged_dir_path` (where merged staged files will live) to the lowest number node of the nodes you're using (the head node). - - Change the hard-coded nodes specified in `staged_dir_paths_list` to the list of nodes you're using for this job, except **do not include the head node in this list** because it was already assigned to `merged_dir_path`. - - Within a `tmux` session, with your virtual environment activated, and ssh'd into the head node, run `python viz-workflow/merge_staged_vector_tiles.py`. - -- Pre-populate your `/scratch` with a `geotiff` dir. - -- Return to the file `viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py` and comment out `step0_staging()`, and uncomment out the next step: `step2_raster_highest(batch_size=100)` (skipping 3d-tiling for release 0.9.0). Run `python viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py` in a `tmux` session with the virtual environment activated and ssh'd into the head node. - -- Run `python viz-workflow/rsync_step2_raster_highest_to_scratch.py`. - -- Return to the file `viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py` and comment out `step2_raster_highest(batch_size=100)`, and uncomment out the next step: `step3_raster_lower(batch_size_geotiffs=100)`. Run `python viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py`. - -- Check that the file `rasters_summary.csv` was written to `/scratch`. Download this file locally. If the top few lines look oddly fomatted, delete these lines and re-upload the file to the same directory (overwriting the misformatted one there). - -- Create a new directory called `web_tiles` in your `/scratch` dir. - -- Return to `IN_PROGRESS_VIZ_WORKFLOW.py` and comment out the step: `step4_webtiles(batch_size_web_tiles=250)`. Run `python viz-workflow/IN_PROGRESS_VIZ_WORKFLOW.py`. - -- Transfer the `log.log` in each nodes' `/tmp` dir to that respective nodes' subdir within `staging` dir: run `python viz-workflow/rsync_log_to_scratch.py` - -- Cancel the job: `scancel {JOB ID}`. The job ID can be found on the left column of the output from `squeue | grep {USERNAME}`. No more credits are being used. Recall that the job will automatically be cancelled after 24 hours even if this command is not run. - -- Remember to remove the `{ACCOUNT NAME}` for the allocation in the slurm script before pushing to GitHub. - -- Move output data from Delta to the Arctic Data Center as soon as possible. +To run the visualization workflow with Ray on the National Center for Supercomputing Applications Delta server, see documentation in [`PermafrostDiscoveryGateway/viz-info/09_iwp-workflow`](https://github.com/PermafrostDiscoveryGateway/viz-info/blob/main/09_iwp-workflow.md) ### Port Forward Ray Dashboard -1. Login to a login node on delta server +1. Login to a login node on Delta server ``` ssh <πŸ‘‰YOUR_NCSA_USERNAMEπŸ‘ˆ>@login.delta.ncsa.illinois.edu @@ -103,12 +50,20 @@ ssh -L 8265:localhost:8265 @ # Navigate your web browser to: localhost:8265/ ``` -### Contributing +## License -Please contribute via pull requests. +``` +Copyright [2013] [Regents of the University of California] -Documenting an environment can be done as follows: +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at -``` -conda env export | grep -v "^prefix: " > environment.yml +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. ``` diff --git a/infrastructure_config.py b/infrastructure_config.py new file mode 100644 index 0000000..0dbf3f2 --- /dev/null +++ b/infrastructure_config.py @@ -0,0 +1,75 @@ +from datetime import datetime +import subprocess +import numpy as np + +# always include the tailing slash "/" +# define user on Delta, avoid writing files to other user's dir +user = subprocess.check_output("whoami").strip().decode("ascii") +head_node = 'cn102/' +#head_node = 'gpub___' + +INPUT = '/scratch/bbou/julietcohen/infrastructure/input/' +output_subdir = 'infrastructure/output' +OUTPUT = f'/scratch/bbou/{user}/{output_subdir}/' + +STAGING_LOCAL = '/tmp/staged/' +STAGING_REMOTE = OUTPUT + 'staged/' +STAGING_REMOTE_MERGED = STAGING_REMOTE + head_node + +GEOTIFF_LOCAL = '/tmp/geotiff/' +GEOTIFF_REMOTE = OUTPUT + 'geotiff/' + +WEBTILE_REMOTE = OUTPUT + 'web_tiles/' + +""" final config is exported here, and imported in the workflow python file. """ +CONFIG = { + "deduplicate_clip_to_footprint": False, + "deduplicate_method": None, + "deduplicate_at": None, + "deduplicate_keep_rules": None, + "dir_output": OUTPUT, + "dir_input": INPUT, + "ext_input": ".gpkg", + "dir_geotiff_remote": GEOTIFF_REMOTE, + "dir_geotiff_local": GEOTIFF_LOCAL, + "dir_web_tiles": WEBTILE_REMOTE, + "dir_staged_remote": STAGING_REMOTE, + "dir_staged_remote_merged": STAGING_REMOTE_MERGED, + "dir_staged_local": STAGING_LOCAL, + "filename_staging_summary": STAGING_REMOTE + "staging_summary.csv", + "filename_rasterization_events": GEOTIFF_REMOTE + "raster_events.csv", + "filename_rasters_summary": GEOTIFF_REMOTE + "raster_summary.csv", + "version": datetime.now().strftime("%B%d,%Y"), + "simplify_tolerance": 0.1, + "tms_id": "WGS1984Quad", + "z_range": [ + 0, + 12 + ], + "geometricError": 57, + "z_coord": 0, + "statistics": [ + { + "name": "infrastructure_code", + "weight_by": "area", + "property": "DN", + "aggregation_method": "max", + "resampling_method": "nearest", + "val_range": [ + 11, + 50 + ], + "palette": [ + "#f48525", + "#f4e625", + "#47f425", + "#25f4e2", + "#2525f4", + "#f425c3", + "#f42525" + ], + "nodata_val": 0, + "nodata_color": "#ffffff00" + } + ] +} diff --git a/rsync_staging_to_scratch.py b/rsync_staging_to_scratch.py index 6684166..484c33c 100644 --- a/rsync_staging_to_scratch.py +++ b/rsync_staging_to_scratch.py @@ -14,25 +14,25 @@ # IWP_CONFIG = PRODUCTION_IWP_CONFIG.IWP_CONFIG # IWP_CONFIG2 = IWP_CONFIG.copy() -# for processing lake data: -import lake_change_config -IWP_CONFIG = lake_change_config.IWP_CONFIG -IWP_CONFIG2 = IWP_CONFIG.copy() - # for testing branches with IWP data: # import branch_testing_iwp_config # IWP_CONFIG = branch_testing_iwp_config.IWP_CONFIG # IWP_CONFIG2 = IWP_CONFIG.copy() + +# for infrastructure data: +import infrastructure_config +CONFIG = infrastructure_config.CONFIG +CONFIG2 = CONFIG.copy() # ----------------------------------------------------- # set config properties for current context -IWP_CONFIG2['dir_staged'] = IWP_CONFIG2['dir_staged_local'] -SOURCE = IWP_CONFIG2['dir_staged'] -IWP_CONFIG2['dir_staged'] = IWP_CONFIG2['dir_staged_remote'] -DESTINATION = IWP_CONFIG2['dir_staged'] +CONFIG2['dir_staged'] = CONFIG2['dir_staged_local'] +SOURCE = CONFIG2['dir_staged'] +CONFIG2['dir_staged'] = CONFIG2['dir_staged_remote'] +DESTINATION = CONFIG2['dir_staged'] print("Using config: ") -pprint.pprint(IWP_CONFIG2) +pprint.pprint(CONFIG2) # define user on Delta, avoid writing files to other user's dir user = subprocess.check_output("whoami").strip().decode("ascii") diff --git a/rsync_step2_raster_highest_to_scratch.py b/rsync_step2_raster_highest_to_scratch.py index d19cede..91f8877 100644 --- a/rsync_step2_raster_highest_to_scratch.py +++ b/rsync_step2_raster_highest_to_scratch.py @@ -10,20 +10,20 @@ #import PRODUCTION_IWP_CONFIG #IWP_CONFIG = PRODUCTION_IWP_CONFIG.IWP_CONFIG -# for processing lake data: -import lake_change_config -IWP_CONFIG = lake_change_config.IWP_CONFIG - # for testing branches with IWP data: # import branch_testing_iwp_config # IWP_CONFIG = branch_testing_iwp_config.IWP_CONFIG + +# for infrastructure data: +import infrastructure_config +CONFIG = infrastructure_config.CONFIG # ----------------------------------------------------- # set config properties for current context -IWP_CONFIG['dir_geotiff'] = IWP_CONFIG['dir_geotiff_local'] -SOURCE = IWP_CONFIG['dir_geotiff'] -IWP_CONFIG['dir_geotiff'] = IWP_CONFIG['dir_geotiff_remote'] -DESTINATION = IWP_CONFIG['dir_geotiff'] +CONFIG['dir_geotiff'] = CONFIG['dir_geotiff_local'] +SOURCE = CONFIG['dir_geotiff'] +CONFIG['dir_geotiff'] = CONFIG['dir_geotiff_remote'] +DESTINATION = CONFIG['dir_geotiff'] # define user on Delta, avoid writing files to other user's dir user = subprocess.check_output("whoami").strip().decode("ascii") @@ -57,28 +57,4 @@ process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE) print("All jobs launched! They will work in the background WITHOUT stdout printing. ") - -# OLD CODE THAT IS WRONG BECAUSE WE DO NOT WANT HOSTNAMES TO BE SUBDIRS OF THE GEOTIFF DIR IN SCRATCH BC -# ALL RASTER HIGHEST NEED TO BE IN THE SAME SUBDIR TO CORRECTLY EXECUTE THE RASTER LOWER STEP -# count = 0 -# for hostname in hostnames: -# # to use ssh in rsync (over a remote sheel) use the following: `rsync -rv --rsh=ssh hostname::module /dest`` -# # see https://manpages.ubuntu.com/manpages/focal/en/man1/rsync.1.html (USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION) - -# # mkdir then sync -# mkdir = ['mkdir', '-p', f'{DESTINATION}{hostname}'] -# process = Popen(mkdir, stdin=PIPE, stdout=PIPE, stderr=PIPE) -# time.sleep(0.2) - -# ssh = ['ssh', f'{hostname}',] -# rsync = ['rsync', '-r', '--update', SOURCE, f'{DESTINATION}{hostname}'] -# cmd = ssh + rsync -# print(f"'{count} of {len(hostnames)}'. running command: {cmd}") -# count += 1 - -# process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE) - -# print("All jobs launched! They will work in the background WITHOUT stdout printing. ") - -# otpional improvement -# shlex.split(s) -- turn cmd line args into a list. \ No newline at end of file + \ No newline at end of file diff --git a/setup.py b/setup.py index 6adde00..c86f0a2 100644 --- a/setup.py +++ b/setup.py @@ -50,6 +50,6 @@ test_suite='tests', tests_require=test_requirements, url='https://github.com/PermafrostDiscoveryGateway/viz-workflow', - version='0.9.0', + version='0.9.2', zip_safe=False, ) diff --git a/slurm/BEST_cpu_ray_double_srun.slurm b/slurm/BEST_cpu_ray_double_srun.slurm index 27fac58..034db86 100644 --- a/slurm/BEST_cpu_ray_double_srun.slurm +++ b/slurm/BEST_cpu_ray_double_srun.slurm @@ -3,11 +3,11 @@ #SBATCH --job-name=pdg_viz #SBATCH --partition=cpu #SBATCH --account= -#SBATCH --time=24:00:00 +#SBATCH --time=48:00:00 #SBATCH --export=ALL,RAY_worker_register_timeout_seconds=120 -#SBATCH --nodes=20 +#SBATCH --nodes=1 #SBATCH --mem=0 #SBATCH --exclusive @@ -39,7 +39,8 @@ set -x echo "This is BEST_cpu_ray_double_srun.slurm" # venv init -source /scratch/bbou/julietcohen/venv/iwp_3/bin/activate +source /scratch/bbou/julietcohen/venv/infrastructure/bin/activate +# source /scratch/bbou/julietcohen/venv/iwp_3/bin/activate # set file soft limit to maximum value (not unlimited because that's not permitted) # before any srun's are executed ulimit -n 32768 diff --git a/lake_change_viz_workflow.py b/viz_workflow.py similarity index 83% rename from lake_change_viz_workflow.py rename to viz_workflow.py index 041c9d5..e4f3e96 100644 --- a/lake_change_viz_workflow.py +++ b/viz_workflow.py @@ -2,7 +2,6 @@ import json import logging -#import logging.config import logging.handlers import os @@ -28,8 +27,8 @@ # define user on Delta, avoid writing files to other user's dir user = subprocess.check_output("whoami").strip().decode("ascii") -import lake_change_config -IWP_CONFIG = lake_change_config.IWP_CONFIG +import infrastructure_config +CONFIG = infrastructure_config.CONFIG # configure logger logger = logging.getLogger("logger") @@ -45,7 +44,6 @@ logger.addHandler(handler) logger.setLevel(logging.INFO) -print(logger.handlers) def main(): result = subprocess.run(["hostname", "-i"], capture_output=True, text=True) @@ -59,8 +57,6 @@ def main(): print("🎯 Ray initialized.") print_cluster_stats() - # create file workflow_log.txt in output dir - # start_logging() # removed this because trying to get logger to work with special config to log.log rather than Kastan's file start = time.time() try: @@ -77,8 +73,6 @@ def main(): # todo: immediately after initiating above step, start rsync script to continuously sync geotiff files, # or immediately after the above step is done, rsync all files at once if there is time left in job # step3_raster_lower(batch_size_geotiffs=100) # rasterize all LOWER Z levels - # todo: immediately after initiating above step, start rsync script to continuously sync geotiff files, - # or immediately after the above step is done, rsync all files at once if there is time left in job step4_webtiles(batch_size_web_tiles=250) # convert to web tiles. except Exception as e: @@ -91,7 +85,6 @@ def main(): ############### πŸ‘‡ MAIN STEPS FUNCTIONS πŸ‘‡ ############### -# @workflow.step(name="Step0_Stage_All") def step0_staging(): FAILURES = [] FAILURE_PATHS = [] @@ -103,15 +96,14 @@ def step0_staging(): logger.info("step0_staging() has initiated.") # update the config for the current context: write staged files to local /tmp dir - iwp_config = deepcopy(IWP_CONFIG) - iwp_config['dir_staged'] = iwp_config['dir_staged_local'] + config = deepcopy(CONFIG) + config['dir_staged'] = config['dir_staged_local'] # make directory /tmp/staged on each node - # not really necessary cause Robyn's functions are set up to do this + # not really necessary cause functions are set up to do this # and /tmp allows dirs to be created to write files - os.makedirs(iwp_config['dir_staged'], exist_ok = True) + os.makedirs(config['dir_staged'], exist_ok = True) - # OLD METHOD "glob" all files. - stager = pdgstaging.TileStager(iwp_config, check_footprints=False) + stager = pdgstaging.TileStager(config, check_footprints=False) staging_input_files_list = stager.tiles.get_filenames_from_dir('input') @@ -119,7 +111,7 @@ def step0_staging(): # with open(os.path.join(iwp_config['dir_output'], "workflow_log.txt"), "a+") as file: # file.write(f"Number of filepaths in staging_input_files_list: {len(staging_input_files_list)}\n\n") # removed this because trying to get logger to work with special config to log.log rather than Kastan's file - with open(os.path.join(iwp_config['dir_output'], "staging_input_files_list.json") , "w") as f: + with open(os.path.join(config['dir_output'], "staging_input_files_list.json") , "w") as f: json.dump(staging_input_files_list, f) """ @@ -145,7 +137,7 @@ def step0_staging(): for itr, filepath in enumerate(staging_input_files_list): # if itr <= 6075: # create list of remote function ids (i.e. futures) - app_futures.append(stage_remote.remote(filepath, iwp_config)) + app_futures.append(stage_remote.remote(filepath, config)) # record how many app_futures were created to determine if it is the full number of input paths # with open(os.path.join(iwp_config['dir_output'], "workflow_log.txt"), "a+") as file: @@ -220,13 +212,13 @@ def prep_only_high_ice_input_list(): try: staging_input_files_list_raw = json.load(open('./iwp-file-list.json')) except FileNotFoundError as e: - print("❌❌❌ Hey you, please specify a πŸ‘‰ json file containing a list of input file paths πŸ‘ˆ (relative to `BASE_DIR_OF_INPUT`).", e) + print("❌❌❌ Specify a json file containing a list of input file paths.", e) #if ONLY_SMALL_TEST_RUN: # for testing only # staging_input_files_list_raw = staging_input_files_list_raw[:TEST_RUN_SIZE] # make paths absolute (not relative) - staging_input_files_list = prepend(staging_input_files_list_raw, IWP_CONFIG['dir_input']) + staging_input_files_list = prepend(staging_input_files_list_raw, CONFIG['dir_input']) # ONLY use high_ice for a test run. print("⚠️ WARNING: ONLY USING HIGH_ICE FOR A TEST RUN!!!!! ⚠️") @@ -330,25 +322,21 @@ def step2_raster_highest(batch_size=100): This is a BLOCKING step (but it can run in parallel). Rasterize all staged tiles (only highest z-level). """ - # import gc - # from random import randrange - # from pympler import muppy, summary, tracker #os.makedirs(iwp_config['dir_geotiff'], exist_ok=True) - iwp_config = deepcopy(IWP_CONFIG) + config = deepcopy(CONFIG) - print(f"Using config {iwp_config}") + print(f"Using config {config}") # update the config for the current context: write geotiff files to local /tmp dir - iwp_config['dir_geotiff'] = iwp_config['dir_geotiff_local'] + config['dir_geotiff'] = config['dir_geotiff_local'] - print(f"2️⃣ Step 2 Rasterize only highest Z, saving to {iwp_config['dir_geotiff']}") + print(f"2️⃣ Step 2 Rasterize only highest Z, saving to {config['dir_geotiff']}") - iwp_config['dir_staged'] = iwp_config['dir_staged_remote_merged'] - stager = pdgstaging.TileStager(iwp_config, check_footprints=False) - stager.tiles.add_base_dir('output_of_staging', iwp_config['dir_staged'], '.gpkg') - # i dont think this next line is necessary cause dedup occurs based on the flag that is inserted during staging: + config['dir_staged'] = config['dir_staged_remote_merged'] + stager = pdgstaging.TileStager(config, check_footprints=False) + stager.tiles.add_base_dir('output_of_staging', config['dir_staged'], '.gpkg') #rasterizer = pdgraster.RasterTiler(iwp_config) # make directories in /tmp so geotiffs can populate there @@ -364,17 +352,15 @@ def step2_raster_highest(batch_size=100): # first define the dir that contains the staged files into a base dir to pull all filepaths correctly #stager.tiles.add_base_dir('output_of_staging', iwp_config['dir_staged'], '.gpkg') - print(f"Collecting all STAGED files from `{iwp_config['dir_staged']}`...") + print(f"Collecting all STAGED files from `{config['dir_staged']}`...") # Get paths to all the newly staged tiles staged_paths = stager.tiles.get_filenames_from_dir(base_dir = 'output_of_staging') - - # stager.kas_check_footprints(staged_paths) #if ONLY_SMALL_TEST_RUN: # staged_paths = staged_paths[:TEST_RUN_SIZE] # save a copy of the files we're rasterizing. - staged_path_json_filepath = os.path.join(iwp_config['dir_output'], "staged_paths_to_rasterize_highest.json") + staged_path_json_filepath = os.path.join(config['dir_output'], "staged_paths_to_rasterize_highest.json") print(f"Writing a copy of the files we're rasterizing to {staged_path_json_filepath}...") with open(staged_path_json_filepath, "w") as f: json.dump(staged_paths, f, indent=2) @@ -382,7 +368,7 @@ def step2_raster_highest(batch_size=100): print(f"Step 2️⃣ -- Making batches of staged files... batch_size: {batch_size}") staged_batches = make_batch(staged_paths, batch_size) - print(f"The input to this step, Rasterization, is the output of Staging.\n Using Staging path: {iwp_config['dir_staged']}") + print(f"The input to this step, Rasterization, is the output of Staging.\n Using Staging path: {config['dir_staged']}") print(f"πŸŒ„ Rasterize total files {len(staged_paths)} gpkgs, using batch size: {batch_size}") print(f"🏎 Parallel batches of jobs: {len(staged_batches)}...\n") @@ -409,13 +395,10 @@ def step2_raster_highest(batch_size=100): app_futures = [] for i, batch in enumerate(staged_batches): - # inserted to try to get geotiffs to stop trying to write to scratch: - #iwp_config['dir_footprints'] = iwp_config['dir_footprints_local'] #iwp_config['dir_geotiff'] = iwp_config['dir_geotiff_local'] - #stager = pdgstaging.TileStager(iwp_config, check_footprints=False) # maybe remove this, added when troubleshooting #rasterizer = pdgraster.RasterTiler(iwp_config) - os.makedirs(iwp_config['dir_geotiff'], exist_ok=True) - app_future = rasterize.remote(batch, iwp_config) + os.makedirs(config['dir_geotiff'], exist_ok=True) + app_future = rasterize.remote(batch, config) app_futures.append(app_future) for i in range(0, len(app_futures)): @@ -465,32 +448,29 @@ def step2_raster_highest(batch_size=100): # @workflow.step(name="Step3_Rasterize_lower_z_levels") def step3_raster_lower(batch_size_geotiffs=20): ''' - STEP 3: Create parent geotiffs for all z-levels (except highest) - THIS IS HARD TO PARALLELIZE multiple zoom levels at once..... sad, BUT - πŸ‘‰ WE DO PARALLELIZE BATCHES WITHIN one zoom level. + STEP 3: Create parent geotiffs for all z-levels except highest + (parallelize batches within one zoom level) ''' print("3️⃣ Step 3: Create parent geotiffs for all lower z-levels (everything except highest zoom)") - iwp_config = deepcopy(IWP_CONFIG) + config = deepcopy(CONFIG) - iwp_config['dir_geotiff'] = iwp_config['dir_geotiff_remote'] + config['dir_geotiff'] = config['dir_geotiff_remote'] # update the config for the current context: pull stager that represents staged files in /scratch - # next line is likely not necessary but can't hurt - iwp_config['dir_staged'] = iwp_config['dir_staged_remote_merged'] - stager = pdgstaging.TileStager(iwp_config, check_footprints=False) + config['dir_staged'] = config['dir_staged_remote_merged'] + stager = pdgstaging.TileStager(config, check_footprints=False) # find all Z levels min_z = stager.config.get_min_z() max_z = stager.config.get_max_z() parent_zs = range(max_z - 1, min_z - 1, -1) - # next line is likely not necessary but can't hurt (we already defined this a few lines above) - iwp_config['dir_geotiff'] = iwp_config['dir_geotiff_remote'] - rasterizer = pdgraster.RasterTiler(iwp_config) + config['dir_geotiff'] = config['dir_geotiff_remote'] + rasterizer = pdgraster.RasterTiler(config) - print(f"Collecting all Geotiffs (.tif) in: {iwp_config['dir_geotiff']}") - # removing this next line bc robyn removed it from kastan's script: - stager.tiles.add_base_dir('geotiff_remote', iwp_config['dir_geotiff'], '.tif') # had to change this from geotiff to geotiff_remote bc got error that the geotiff base dir already existed + print(f"Collecting all Geotiffs (.tif) in: {config['dir_geotiff']}") + # had to change next line from 'geotiff' to 'geotiff_remote' bc got error that the geotiff base dir already existed + stager.tiles.add_base_dir('geotiff_remote', config['dir_geotiff'], '.tif') start = time.time() # Can't start lower z-level until higher z-level is complete. @@ -528,10 +508,10 @@ def step3_raster_lower(batch_size_geotiffs=20): # I dont think theres a need to set rasterizer with new config after chaning this property cause # that is done within the function create_composite_geotiffs() but troubleshooting # so lets do it anyway - rasterizer = pdgraster.RasterTiler(iwp_config) + rasterizer = pdgraster.RasterTiler(config) # MANDATORY: include placement_group for better stability on 200+ cpus. - app_future = create_composite_geotiffs.remote(parent_tile_batch, iwp_config, logging_dict=None) + app_future = create_composite_geotiffs.remote(parent_tile_batch, config, logging_dict=None) app_futures.append(app_future) # Don't start the next z-level (or move to step 4) until the @@ -554,7 +534,6 @@ def step3_raster_lower(batch_size_geotiffs=20): print(f"⏰ Total time to create parent geotiffs: {(time.time() - start)/60:.2f} minutes\n") return "Done step 3." -# @workflow.step(name="Step4_create_webtiles") def step4_webtiles(batch_size_web_tiles=300): ''' STEP 4: Create web tiles from geotiffs @@ -562,49 +541,36 @@ def step4_webtiles(batch_size_web_tiles=300): ''' print("4️⃣ -- Creating web tiles from geotiffs...") - iwp_config = deepcopy(IWP_CONFIG) + config = deepcopy(CONFIG) # instantiate classes for their helper functions # not sure that the next line is necessary but might as well keep the config up to date # with where files currently are - iwp_config['dir_staged'] = iwp_config['dir_staged_remote_merged'] + config['dir_staged'] = config['dir_staged_remote_merged'] # pull all z-levels of rasters from /scratch for web tiling - iwp_config['dir_geotiff'] = iwp_config['dir_geotiff_remote'] + config['dir_geotiff'] = config['dir_geotiff_remote'] # define rasterizer here just to updates_ranges() - rasterizer = pdgraster.RasterTiler(iwp_config) + rasterizer = pdgraster.RasterTiler(config) # stager = pdgstaging.TileStager(iwp_config, check_footprints=False) remove this line, no point in defining the stager right before we update the config, do it after! start = time.time() # Update color ranges print(f"Updating ranges...") rasterizer.update_ranges() - iwp_config_new = rasterizer.config.config - print(f"Defined new config: {iwp_config_new}") - - # define the stager so we can pull filepaths from the geotiff base dir in a few lines - # stager = pdgstaging.TileStager(iwp_config_new, check_footprints=False) + config_new = rasterizer.config.config + print(f"Defined new config: {config_new}") # Note: we also define rasterizer later in each of the 3 functions: # raster highest, raster lower, and webtile functions - print(f"Collecting all Geotiffs (.tif) in: {iwp_config['dir_geotiff']}...") # the original config here is correct, it is just printing the filepath we are using for source of geotiff files - #print(f"Collecting all Geotiffs (.tif) in: {IWP_CONFIG_NEW['dir_geotiff']}...") - - #stager.tiles.add_base_dir('geotiff_path', IWP_CONFIG_NEW['dir_geotiff'], '.tif') # don't think we need this line because we are using the same geotiff base dir set in the raster lower step (remote geotiff dir with geotiffs of all z-levels) - #stager = pdgstaging.TileStager(IWP_CONFIG, check_footprints=False) - #stager.tiles.add_base_dir('geotiff_remote', IWP_CONFIG['dir_geotiff'], '.tif') - #geotiff_paths = stager.tiles.get_filenames_from_dir(base_dir = 'geotiff_path') + # the original config here is fine, just printing filepath for source of geotiff files + print(f"Collecting all Geotiffs (.tif) in: {config['dir_geotiff']}...") # set add base dir with rasterizer instead of stager #rasterizer.tiles.add_base_dir('geotiff_remote', IWP_CONFIG['dir_geotiff'], '.tif') #geotiff_paths = rasterizer.tiles.get_filenames_from_dir(base_dir = 'geotiff_remote') - # added next 2 lines 20230214: - rasterizer.tiles.add_base_dir('geotiff_remote_all_zs', iwp_config_new['dir_geotiff'], '.tif') # call it something different than geotiff_remote because we already made that base dir earlier and it might not overwrite and might error cause already exists - # and also remove line just below bc it was that line's replacement for time being: - # rasterizer.tiles.add_base_dir('geotiff_remote_all_zs', IWP_CONFIG['dir_geotiff'], '.tif') + rasterizer.tiles.add_base_dir('geotiff_remote_all_zs', config_new['dir_geotiff'], '.tif') # call it something different than geotiff_remote because we already made that base dir earlier and it might not overwrite and might error cause already exists geotiff_paths = rasterizer.tiles.get_filenames_from_dir(base_dir = 'geotiff_remote_all_zs') - # check if the rasterizer can add_base_dir?? robyn added that so must be correct - # change made 2023-02-14: change rasterizer to stager to pull filenames from stager base dir created in raster lower step #geotiff_paths = stager.tiles.get_filenames_from_dir(base_dir = 'geotiff_remote') #if ONLY_SMALL_TEST_RUN: @@ -628,8 +594,7 @@ def step4_webtiles(batch_size_web_tiles=300): # MANDATORY: include placement_group for better stability on 200+ cpus. # app_future = create_web_tiles.options(placement_group=pg).remote(batch, IWP_CONFIG) # app_future = create_web_tiles.remote(batch, IWP_CONFIG) # remove this line - app_future = create_web_tiles.remote(batch, iwp_config_new) - #app_future = create_web_tiles.remote(batch, IWP_CONFIG) # remove this line when line "reintroduce" lines are reintroduced # update 8/22: dont remember why I wrote that comment + app_future = create_web_tiles.remote(batch, config_new) app_futures.append(app_future) for i in range(0, len(app_futures)): @@ -663,10 +628,6 @@ def rasterize(staged_paths, config, logging_dict=None): # print(staged_paths) try: - #IWP_CONFIG['dir_geotiff'] = IWP_CONFIG['dir_geotiff_remote'] - #os.makedirs(IWP_CONFIG['dir_geotiff'], exist_ok = True) - #IWP_CONFIG['dir_footprints'] = IWP_CONFIG['dir_footprints_local'] - #IWP_CONFIG['dir_staged'] = IWP_CONFIG['dir_staged_remote_merged'] rasterizer = pdgraster.RasterTiler(config) # todo: fix warning `python3.9/site-packages/geopandas/base.py:31: UserWarning: The indices of the two GeoSeries are different.` # with suppress(UserWarning): @@ -687,7 +648,6 @@ def create_composite_geotiffs(tiles, config, logging_dict=None): import logging.config logging.config.dictConfig(logging_dict) try: - #iwp_config['dir_footprints'] = iwp_config['dir_footprints_local'] rasterizer = pdgraster.RasterTiler(config) rasterizer.parent_geotiffs_from_children(tiles, recursive=False) except Exception as e: @@ -704,7 +664,6 @@ def create_web_tiles(geotiff_paths, config, logging_dict=None): import logging.config logging.config.dictConfig(logging_dict) try: - #IWP_CONFIG['dir_geotiff'] = IWP_CONFIG['dir_geotiff_remote'] using new config here...not sure if it will have the property 'dir_geotiff_remote'??? If so, uncomment this line rasterizer = pdgraster.RasterTiler(config) rasterizer.webtiles_from_geotiffs(geotiff_paths, update_ranges=False) except Exception as e: @@ -725,9 +684,7 @@ def stage_remote(filepath, config, logging_dict = None): Parallelism at the per-shape-file level. """ # deepcopy makes a realy copy, not using references. Helpful for parallelism. - # config_path = deepcopy(IWP_CONFIG) - - iwp_config = deepcopy(IWP_CONFIG) + #config = deepcopy(CONFIG) try: stager = pdgstaging.TileStager(config=config, check_footprints=False) @@ -737,8 +694,6 @@ def stage_remote(filepath, config, logging_dict = None): # file.write(f"Successfully staged file:\n") # file.write(f"{filepath}\n\n") # removed this because trying to get logger to work with special config to log.log rather than Kastan's file - #logging.info(f"Juliet's logging: Successfully staged tile: {filepath}.") - if 'Skipping' in str(ret): print(f"⚠️ Skipping {filepath}") @@ -746,12 +701,8 @@ def stage_remote(filepath, config, logging_dict = None): # file.write(f"SKIPPING FILE:\n") # file.write(f"{filepath}\n\n") # removed this because trying to get logger to work with special config to log.log rather than Kastan's file - #logging.info(f"Juliet's logging: Skipping staging tile: {filepath}.") - except Exception as e: - #logging.info(f"Juliet's logging :Failed to stage tile: {filepath}.") - # with open(os.path.join(iwp_config['dir_output'], "workflow_log.txt"), "a+") as file: # file.write(f"FAILURE TO STAGE FILE:\n") # file.write(f"{filepath}\n") @@ -777,10 +728,7 @@ def make_workflow_id(name: str) -> str: return f"{name}-{str(curr_time.strftime('%h_%d,%Y@%H:%M'))}" def build_filepath(input_file): - # Demo input: /home/kastanday/output/staged/WorldCRS84Quad/13/1047/1047.gpkg - # Replace 'staged' -> '3d_tiles' in path - # In: /home/kastanday/output/staged/WorldCRS84Quad/13/1047/1047.gpkg # new_path: /home/kastanday/output/3d_tiles/WorldCRS84Quad/13/1047/1047.gpkg path = pathlib.Path(input_file) index = path.parts.index('staged') @@ -906,10 +854,10 @@ def start_logging(): ''' Writes file workflow_log.txt in output directory. ''' - filepath = pathlib.Path(IWP_CONFIG['dir_output'] + 'workflow_log.txt') + filepath = pathlib.Path(CONFIG['dir_output'] + 'workflow_log.txt') filepath.parent.mkdir(parents=True, exist_ok=True) filepath.touch(exist_ok=True) - logging.basicConfig(level=logging.INFO, filename= IWP_CONFIG['dir_output'] + 'workflow_log.txt', filemode='w', format='%(asctime)s - %(levelname)s - %(message)s') + logging.basicConfig(level=logging.INFO, filename= CONFIG['dir_output'] + 'workflow_log.txt', filemode='w', format='%(asctime)s - %(levelname)s - %(message)s') @ray.remote def rsync_raster_to_scatch(rsync_python_file='utilities/rsync_merge_raster_to_scratch.py'):