ABSTRACT:

(This collection holds major CJW announcements with full-text of the most recent and important ones repeated in the Abstract section)
(For the latest features and example notebooks please refer to the links to Release Announcement in "Collection Content" down below.)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updated on 07/13/2022
CJW 2022-Q2 release is live. Check it out at http://go.illinois.edu/cybergis-jupyter-water
For release notes: https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

---------
5/18/2022 (Updated on 12PM CT)

Globus service interruption has been resolved on SDSC Expanse HPC. Job submission to Expanse is back online.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03/2022
CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This notebook demonstrates the setup for a typical WRF-Hydro model on HydroShare leveraging different tools or services throughout the entire end-to-end modelling workflow. The notebook is designed in such a way that the user/modeler is able to retrieve datasets only relevant to a user-defined spatial domain (space domain), for example, a watershed domain of interest and time domain using a graphical user interface (GUI) linked to HPC. In order to help users submitting a job on HPC to run the model, they are provided with a user-friendly interface that abstracts away details and complexities involved in the HPC use such as authorization, authentication, monitoring and scheduling of the jobs, data and job management, and transferring data back and forth. Users can interact with this GUI to perform modeling work. This GUI is designed in such a way to allow users/modeler to 1) select the remote server where the HPC job will run, 2) upload the simulation directory, which contains the configuration files, 3) specify the parameters of the HPC job that the user is allowed to utilize, 4) set some parameters related to the model compilation, 5) follow-up on the submitted job status and 6) retrieve the model output files back to local workspace. Once the model execution is completed, users can easily have access to the model outputs on HPC and retrieve them to the local workspace for visualization and analysis.

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q3)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

(1) Cern Virtual Machine File System (CVMFS): We have redesigned how we deliver software within CyberGIS-Jupyter. This new design drastically increases computational performance and reproducibility, and allows the platform to make the software environment available in a variety of settings. From an end-user perspective, there should be no change to your accessing and utilizing the CJW services.

(2) Improved user experience for CyberGIS-Compute: In previous versions, we introduced the capability for users to “Restore” their previously submitted jobs of interest. Based on user feedback, we’ve further refined the interface to support viewing and downloading outputs of all previously submitted jobs by simply navigating to the “Past Results” section. The result/output of any completed job can be accessed with a single click.

(3) Support for new High Performance Computing (HPC) backend in CyberGIS-Compute: Anvil is now available as a new HPC resource for CyberGIS-Compute. Supported by NSF, Anvil is a HPC system hosted at Purdue University that contains 1000 CPU nodes based on the third generation AMD EPYC "Milan" processor, delivering a peak performance of 5.3 petaflops. Allocations on Anvil are managed by NSF's ACCESS program (https://access-ci.org/). The large numbers of CPU nodes and cores (i.e., 128) enable superior computational performance for scalable codes, short queuing times, and fast execution for hydrologic models via CyberGIS-Compute. For more information on Anvil, refer to the documentation at: https://www.rcac.purdue.edu/anvil. The WRFHydro model is supported on Anvil via CyberGIS-Compute. Please refer to the example notebook below.

Please refer to the following resources for details and examples:

A Brief Overview Of Cern Virtual Machine File System (CVMFS)
http://www.hydroshare.org/resource/ab1555c0c8d34d3496997353ba8060d9

CyberGIS-Compute updates - 2022-Q3
http://www.hydroshare.org/resource/3b472641c3504161bb13a19d4c9fbc87

Submission of WRFHydro model to Anvil HPC
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

See Release Notes on HydroShare
http://www.hydroshare.org/resource/bf463f07e1244de4a17b3ea7b2d95916

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.
Best regards,
CyberGIS-Hydro team

ABSTRACT:

A Brief Overview Of Cern Virtual Machine File System (CVMFS) in Jupyter environment

ABSTRACT:

Jupyter Notebook listing use cases and code snippets relevant to CyberGIS-Compute Q3-2022 updates.

ABSTRACT:

These are the HUC 12 watershed boundaries for the Great Basin.

ABSTRACT:

News articles collected by searching words ‘aging dams’ in Google News and news websites. 188 news articles in total.

ABSTRACT:

Datasets for I-GUIDE Data Catalog

ABSTRACT:

The Semantic Scholar Open Research Corpus (S2ORC) is a general-purpose corpus for NLP and text mining research over scientific papers. The corpus covers 136M+ paper nodes with 12.7M+ full text papers and connected by 467M+ citation edges by unifying data from many different sources.
This resource contains aging dams part of research corpus filtered by IGUIDE GeoAI & DS team

ABSTRACT:

This data repository is connected with a manuscript entitled "Socioeconomic characteristics of at-risk populations impacted by the aging dam infrastructure in the USA - Who is facing the risk of potential dam failures? "

Abstract of the manuscript:
The dam infrastructure in the conterminous United States (CONUS) has exceeded its designed service lives to a large extent, posing an increased risk of failures that can cause catastrophic disasters with substantial economic and human losses. However, limited attention has been paid to the characteristics of at-risk populations, hindering adequate understanding and preparedness for emergency planning. Our study proposes a framework employing spatial metrics to discover where and whether socially vulnerable populations are more exposed to flood inundation risks induced by dam failures. By applying the framework to 345 dams in the CONUS, we found that characteristics of at-risk populations vary extensively across space. To better understand this spatial variability, we categorized the dams into five clusters based on at-risk population characteristics. We find that of the dams analyzed, those in California, New England, and the Upper Mississippi basin, pose particularly high consequential risks for socially vulnerable populations.

The naming convention of inundation maps:
Syntax: [Water Level]_[Breach Condition]_[NID Dam ID]
* Water Level:
- MH: Maximum Height
- TAS: Top of Active Storage
- NH: Normal Height

* Breach Condition:
- F: Fail
- S: Stable

For example, 'MH_F_CA10022' indicates inundation maps induced by the failure of the dam (CA10022) under the Maximum Height (MH) and Breach (F) scenario. The original data is available at the National Inventory of Dams (https://nid.sec.usace.army.mil/viewer/index.html).

ABSTRACT:

This data repository is connected with a manuscript entitled "Socioeconomic characteristics of at-risk populations impacted by the aging dam infrastructure in the USA - Who is facing the risk of potential dam failures? "

Abstract of the manuscript:
The dam infrastructure in the conterminous United States (CONUS) has exceeded its designed service lives to a large extent, posing an increased risk of failures that can cause catastrophic disasters with substantial economic and human losses. However, limited attention has been paid to the characteristics of at-risk populations, hindering adequate understanding and preparedness for emergency planning. Our study proposes a framework employing spatial metrics to discover where and whether socially vulnerable populations are more exposed to flood inundation risks induced by dam failures. By applying the framework to 345 dams in the CONUS, we found that characteristics of at-risk populations vary extensively across space. To better understand this spatial variability, we categorized the dams into five clusters based on at-risk population characteristics. We find that of the dams analyzed, those in California, New England, and the Upper Mississippi basin, pose particularly high consequential risks for socially vulnerable populations.

Directory Structure:
In addition to the notebooks and other data, the analysis ready output data for all dams is in "results/N_*" directories.

ABSTRACT:

The objective of this HydroShare resource is to query AORC v1.0 Forcing data stored on HydroShare's Thredds server and create a subset of this dataset for a designated watershed and timeframe. The user is prompted to define their temporal and spatial frames of interest, which specifies the start and end dates for the data subset. Additionally, the user is prompted to define a spatial frame of interest, which could be a bounding box or a shapefile, to subset the data spatially.

Before the subsetting is performed, data is queried, and geospatial metadata is added to ensure that the data is correctly aligned with its corresponding location on the Earth's surface. To achieve this, two separate notebooks were created - [this notebook](https://github.com/CUAHSI/notebook-examples/blob/main/thredds/query-aorc-thredds.ipynb) and [this notebook] (https://github.com/CUAHSI/notebook-examples/blob/main/thredds/aorc-adding-spatial-metadata.ipynb) - which explain how to query the dataset and add geospatial metadata to AORC v1.0 data in detail, respectively. In this notebook, we call functions from the AORC.py script to perform these preprocessing steps, resulting in a cleaner notebook that focuses solely on the subsetting process.