Shaowen Wang

University of Illinois at Urbana-Champaign | Professor

Subject Areas: Computational Science; CyberGIS; Geography; Geospatial Data Science; Spatial Analysis

 Recent Activity

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Support for NCAR/UCAR WRFHydro Training Notebooks: Users are now able to set up the necessary environment to run WRFHydro hands-on training notebooks (https://ral.ucar.edu/projects/wrf_hydro/training-materials) through simple clicks on CJW. For details on how to run these training notebooks please review the following HydroShare resources: (a) WRFHydro Hands-on Training v5.2.x on CJW; and (b) WRF&WRFHydro Coupled Training v5.1.2 on CJW. This provides an alternative solution to the traditional Docker-based local setup (https://hub.docker.com/r/wrfhydro/training/) that makes it easy for users to complete this training as it does not require installation or downloading of any software or data onto the user’s local computer. Additionally, it enables users to access more powerful computing resources in a cloud-based Jupyter environment. All necessary materials required to complete the training are retrieved from official data sources managed by the NCAR/UCAR WRFHydro Development Team and accessible on CJW via a browser.

2) Transition from XSEDE Comet to Expanse: Comet HPC will be decommissioned on July 31, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW will be transferred to the Expanse HPC (https://portal.xsede.org/sdsc-expanse). As a result, the CyberGIS-Compute service will also drop the support on Comet and replace it with Expanse. Despite this change, we do not expect any action is required for the majority of users. The CyberGIS-Compute service and its SDK will redirect all jobs submitted to Comet to Expanse with a warning message showing up. All previously developed notebooks that use Comet will continue to run after this transition. Please contact us for solutions if you have models or notebooks that access Comet without going through the CyberGIS-Compute service.

3) Extended user testing of Kubernetes-based CJW instance: In the previous release, we announced a Kubernetes-based (Aka K8s: https://kubernetes.io/) CJW instance deployed at https://go.illinois.edu/cjw-k8s for user testing with a preliminary migration plan on deprecation of the current DockerSwarm-based CJW. Due to the growing complexity of K8s and more features being developed and added for a smooth user experience, we have decided to continue conducting extensive testing in this release. We encourage all users to join this testing process and would greatly appreciate your feedback. The current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan is implemented.

Please refer to the following HydroShare resources for details and examples:
Run WRFHydro Hands-on Training v5.2.x on CJW
https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/
Run WRF&WRFHydro Coupled Training v5.1.2 on CJW
https://www.hydroshare.org/resource/c2389a2f05564da08ab218e59bdf1e81/
User testing on Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s
Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/6a1ddb17155a4b27b885f442ad14e344/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

Show More

ABSTRACT:

This is a collection that holds all the WRFHydro official training materials you can run on CyberGIS-Jupyter for Water without installing or downloading any software or data onto your local computer. This collection will expand as new training lessons be added.

Show More

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRF&WRF-Hydro Coupled Testcase Online Lesson (v5.1.2). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, two new folders will be created (wrf-hydro-training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

 Contact

Resources
All 0
Collection 0
Composite Resource 0
Generic 0
Geographic Feature 0
Geographic Raster 0
HIS Referenced Time Series 0
Model Instance 0
Model Program 0
MODFLOW Model Instance Resource 0
Multidimensional (NetCDF) 0
Script Resource 0
SWAT Model Instance 0
Time Series 0
Web App 0
Composite Resource Composite Resource
TauDEM Processing of Logan Watershed
Created: May 17, 2017, 1:38 p.m.
Authors: Shaowen Wang

ABSTRACT:

A digital elevation model encompassing the Logan River watershed in northern Utah.

[Modified in JupyterHub on 2017-05-17 13:38:02.413771]
This a group of TauDEM processing results that were derived using the Logan River DEM.

Show More
Web App Resource Web App Resource

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

Show More
Collection Resource Collection Resource

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2020 Q2)

Dear HydroShare Users,

We are pleased to announce a new quarterly release of CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes new capabilities to support the geoanalytics suite of GRASS for model pre/post-processing, PostGIS database, and Landlab Earth surface modelling toolkit along with several enhancements to job submission middleware, system security as well as service infrastructure. Please refer to the following list for details and examples.

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

GRASS GIS for model pre/post-processing:
Learn how to consolidate the features of the GRASS geoanalytics suite to support pre/post-processing for SUMMA and RHESSYs models in CJW.
Example notebooks: https://www.hydroshare.org/resource/4cbcfdd6e7f943e2969dd52e780bc52d/

Manage geospatial data with PostGIS:
PostGIS is an extension to the PostgreSQL object-relational database system which allows geospatial data to be efficiently stored while providing various advanced functions for in-situ data analysis and processing.
Example notebooks: https://www.hydroshare.org/resource/bb779d4cce564dd6afcf463c8910786f/

Security and service infrastructure enhancements
Trusted group: Starting from this release, all users are required to join the “CyberGIS-Jupyter for Water” trusted group at https://www.hydroshare.org/group/157 in order to access the CJW platform, which is a preventive measure to protect the shared computing resources from being abused by malicious users. A complete user profile page is highly recommended to expedite the approval process.
User metric submission to XSEDE: CJW, as a science gateway, is now sending unique user usage metrics to XSEDE to comply with its requirements.

Landlab for enabling collaborative numerical modeling in Earth sciences using knowledge infrastructure
Example notebooks: https://www.hydroshare.org/resource/370c288b61b84794b847ef85c4dd4ffb/
https://www.hydroshare.org/resource/6add6bee06bb4050bfe23e1081627614/

Job submission enhancements
Refactored the structure of the cyberGIS job submission system
Data-driven implementation for avoiding excessive data transmission between HydroShare and CJW
Add the specification of input parameters into a JSON file to improve the flexibility and generality of model management
Enable HPC-SUMMA object that can directly call SUMMA
Example notebooks: https://www.hydroshare.org/resource/4a4a22a69f92497ead81cc48700ba8f8/

Show More
Web App Resource Web App Resource

ABSTRACT:

A newly deployed CyberGIS-Jupyter for Water (CJW) instance powered by Kubernetes (k8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology would significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user sign-up. We welcome all users to join the public testing and give us feedback. We anticipate the public testing on the new CJW would take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final “migration plan” is announced (TBD).

Please send comments and bug reports to help@cybergis.org

URL for direct access: https://go.illinois.edu/cjw-k8s

How to setup OpenWith for "Kubernetes-based CJW (user testing)"
1) Login HydroShare
2) Visit this resource landing page: https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
3) In the upper-right corner, click on the 3rd icon from the left (the one with 3x3 small squares), which should prompt "Add WebApp to Open With List"
4) Refresh the landing page of the resource that has notebooks you are interested in, and "Kubernetes-based CJW (user testing)" should show up in the OpenWith list now

Show More
Collection Resource Collection Resource

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks created for the study "An Approach for Creating Immutable and Interoperable End-to-End Hydrological Modeling Computational Workflows" led by researcher Young-Don Choi submitted to the 2021 EarthCube Annual meeting, Notebook Sessions.

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file provided in this resource.

For the sake of completeness, the abstract for the study submitted to the EarthCube session is mentioned below:

"Reproducibility is a fundamental requirement to advance science. Creating reproducible hydrological models that include all required data, software, and workflows, however, is often burdensome and requires significant work. Computational hydrology is a rapidly advancing field with fast-evolving technologies to support increasingly complex computational hydrologic modeling. The growing model complexity in terms of variety of software and cyberinfrastructure capabilities makes achieving computational reproducibility extremely challenging. Through recent reproducibility research, there have been efforts to integrate three components: 1) (meta)data, 2) computational environments, and 3) workflows. However, each component is still separate, and researchers must interoperate between these three components. These separations make verifying end-to-end reproducibility challenging. Sciunit was developed to assist scientists, who are not programming experts, with encapsulating these three components into a container to enable reproducibility in an immutable form. However, there were still limitations to support interoperable computational environments and apply end-to-end solutions, which are an ultimate goal of reproducible hydrological modeling. Therefore, the objective of this research is to advance the existing Sciunit capabilities to not only support immutable, but also interoperable computational environments and apply an end-to-end modeling workflow using the Regional Hydro-Ecologic Simulation System (RHESSys) hydrologic model as an example. First, we create an end-to-end workflow for RHESSys using pyRHESSys on the CyberGIS-Jupyter for Water platform. Second, we encapsulate the aforementioned three components and create configurations that include lists of encapsulated dependencies using Sciunit. Third, we create two HydroShare resources, one for immutable reproducibility evaluation using Sciunit and the other for interoperable reproducibility evaluation using library configurations created by Sciunit. Finally, we evaluate the reproducibility of Sciunit in MyBinder, which is a different computational environment, using these two resources. This research presents a detailed example of a user-centric case study demonstrating the application of an open and interoperable containerization approach from a hydrologic modeler’s perspective."

Show More
Composite Resource Composite Resource
Run WRFHydro Hands-on Training v5.2.x on CyberGIS-Jupyter for Water
Created: May 25, 2021, 8:30 p.m.
Authors: ·

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRFHydro Hands-on Training v5.2.x (Nov 2020). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook will retrieve the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and put them in a similar directory structure as the Docker-based local setup.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource
Run WRF&WRFHydro Coupled Training v5.1.2 on CyberGIS-Jupyter for Water
Created: May 26, 2021, 9:05 p.m.
Authors: ·

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRF&WRF-Hydro Coupled Testcase Online Lesson (v5.1.2). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, two new folders will be created (wrf-hydro-training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Collection Resource Collection Resource

ABSTRACT:

This is a collection that holds all the WRFHydro official training materials you can run on CyberGIS-Jupyter for Water without installing or downloading any software or data onto your local computer. This collection will expand as new training lessons be added.

Show More
Collection Resource Collection Resource

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Support for NCAR/UCAR WRFHydro Training Notebooks: Users are now able to set up the necessary environment to run WRFHydro hands-on training notebooks (https://ral.ucar.edu/projects/wrf_hydro/training-materials) through simple clicks on CJW. For details on how to run these training notebooks please review the following HydroShare resources: (a) WRFHydro Hands-on Training v5.2.x on CJW; and (b) WRF&WRFHydro Coupled Training v5.1.2 on CJW. This provides an alternative solution to the traditional Docker-based local setup (https://hub.docker.com/r/wrfhydro/training/) that makes it easy for users to complete this training as it does not require installation or downloading of any software or data onto the user’s local computer. Additionally, it enables users to access more powerful computing resources in a cloud-based Jupyter environment. All necessary materials required to complete the training are retrieved from official data sources managed by the NCAR/UCAR WRFHydro Development Team and accessible on CJW via a browser.

2) Transition from XSEDE Comet to Expanse: Comet HPC will be decommissioned on July 31, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW will be transferred to the Expanse HPC (https://portal.xsede.org/sdsc-expanse). As a result, the CyberGIS-Compute service will also drop the support on Comet and replace it with Expanse. Despite this change, we do not expect any action is required for the majority of users. The CyberGIS-Compute service and its SDK will redirect all jobs submitted to Comet to Expanse with a warning message showing up. All previously developed notebooks that use Comet will continue to run after this transition. Please contact us for solutions if you have models or notebooks that access Comet without going through the CyberGIS-Compute service.

3) Extended user testing of Kubernetes-based CJW instance: In the previous release, we announced a Kubernetes-based (Aka K8s: https://kubernetes.io/) CJW instance deployed at https://go.illinois.edu/cjw-k8s for user testing with a preliminary migration plan on deprecation of the current DockerSwarm-based CJW. Due to the growing complexity of K8s and more features being developed and added for a smooth user experience, we have decided to continue conducting extensive testing in this release. We encourage all users to join this testing process and would greatly appreciate your feedback. The current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan is implemented.

Please refer to the following HydroShare resources for details and examples:
Run WRFHydro Hands-on Training v5.2.x on CJW
https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/
Run WRF&WRFHydro Coupled Training v5.1.2 on CJW
https://www.hydroshare.org/resource/c2389a2f05564da08ab218e59bdf1e81/
User testing on Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s
Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/6a1ddb17155a4b27b885f442ad14e344/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

Show More
Composite Resource Composite Resource

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More