ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1)Community Commons for Easy Access to HydroShare Notebooks:
Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/. Interested in learning how CJW enables computational reproducibility of such notebooks? Read on!

2)Enhanced Reproducibility of CJW:
Starting from this release, CJW has implemented a new solution in kernel and software management such that each notebook is now associated with a specific versioned kernel. This approach significantly enhances reproducibility in the CJW environment as each notebook is always opened with the original kernel it is tied to. We encourage users to test their notebooks and reach out to us if you experience any problems.

3)Transferring Large Model Output Datasets:
CJW has introduced a new Globus-based (https://www.globus.org/) data-transfer capability to enhance the performance and stability of transferring a large number of model outputs from high-performance computing (HPC) resources back to CJW, which is a common scenario in running hydrologic models.

4)Seamless Transition from XSEDE Comet to Expanse:
Comet was decommissioned on July 15, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW have been transferred to the Expanse (https://portal.xsede.org/sdsc-expanse). The CyberGIS-Compute service and its SDK now automatically redirect all jobs submitted to Comet to Expanse while prompting users with a simple warning message.

Please refer to the following HydroShare resources for details and examples:
Community Commons for Easy Access to HydroShare Notebooks
https://cybergisxhub.cigi.illinois.edu/hydroshare/
Enhanced Reproducibility of CJW
https://www.hydroshare.org/resource/df8244042e5445edb106d93e5a491d29/
Transferring Large Model Output Datasets
https://www.hydroshare.org/resource/5aa3d27ec5ef4b1b8dfa621ea284af14/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/f3315ec8c1df4f4ab5d2274220de0351/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

An example notebook walks you through how to setup customized kernels on CyberGIS-Jupyter for Water (CJW)

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

Transition to CyberGIS-Compute V2:
CyberGIS-Compute V2 is a new development phase of the CyberGIS-Compute framework that was initially released (denoted as V1) one year ago through CJW 2020-Q4. Compared with V1, V2 includes several major enhancements: 1) a new workflow for model contribution to facilitate adding new hydrologic models by community developers; 2) a GUI in the notebook environment to simplify and guide users through the job submission process; 3) transparent and bi-directional data transfers between CJW and high-performance computing (HPC) resources using Globus by default, and 4) detailed tracking of usage and metrics. It is worth noting that due to the upgraded architecture in V2, existing models implemented in V1 would need to be migrated. For a smooth transition and backward compatibility, services in V1 will remain available in parallel to those in V2, and all the old notebooks that use V1 remain functional.

SUMMA Model Migrated to CyberGIS-Compute V2:
We have migrated the SUMMA model to CyberGIS-Compute V2, and end users can now benefit from the new features mentioned above in SUMMA modeling work. Please refer to the example notebook below for details. In addition, the implementation of SUMMA in CyberGIS-Compute V2 is accessible on a Github repo (https://github.com/cybergis/cybergis-compute-v2-summa), which can serve as a real-world example to model developers who may want to contribute their models for sharing with the community. A “HelloWorld” implementation is also available serving as a model-agnostic example (https://github.com/cybergis/cybergis-compute-mpi-helloworld).

New Modules and Kernel Customization:
Upon user requests, two new easybuild-based modules have been added to the CJW toolbox and are now ready to use: NCL (https://www.ncl.ucar.edu/) for scientific data analysis and visualization (e.g., NetCDF, GRID, HDF); and CDO (https://code.mpimet.mpg.de/projects/cdo) for manipulation of climate and Numerical Weather Prediction (NWP) data. Furthermore, for advanced users who may want to customize the provided software environment and kernels, an example notebook (see below) is available for users to walk through the basics on how to install new libraries on top of existing environments or set up a Conda environment from scratch.

New UI Elements on CJW:
CJW has further customized the Jupyter Notebook user interface to include a virtual Announcement Board (in the header area) for timely communicating with users on upcoming events including downtimes and new releases, and a Bug Report button (at the upper-right corner) that opens an issue tracker page in a publicly accessible Github repo for quick bug reporting.

Please refer to the following resources for details and examples:
Run ensemble SUMMA 3.0 model with CyberGIS-Compute V2
https://www.hydroshare.org/resource/deac1b0b5b46415aaedb886b9dc16f45/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

Implementation of SUMMA model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-v2-summa

Implementation of HelloWorld model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-mpi-helloworld

See Release Notes on HydroShare
https://www.hydroshare.org/resource/2086b241b16b453d827db847e8640475/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

(This collection holds major CJW announcements with full-text of the most recent and important ones repeated in the Abstract section)
(For the latest features and example notebooks please refer to the links to Release Announcement in "Collection Content" down below.)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updated on 07/13/2022
CJW 2022-Q2 release is live. Check it out at http://go.illinois.edu/cybergis-jupyter-water
For release notes: https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

---------
5/18/2022 (Updated on 12PM CT)

Globus service interruption has been resolved on SDSC Expanse HPC. Job submission to Expanse is back online.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03/2022
CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q2)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) CJW moved to a new home. Jetstream-1, an NSF-funded high-performance cloud computing resource where CJW was hosted for the past 3 years, was permanently shut down on July 31, 2022. Its successor, Jetstream-2, which offers much more powerful capabilities, has become the new home of CJW. All existing CJW user data and notebooks have been migrated to Jetstream-2. We do not expect users to experience any change in usage due to this transition but to enjoy a faster and smoother Jupyter environment backed by the latest hardware and cloud technology. In exceptional cases, the previous CJW instance on Jetstream-1 could be accessible upon user request.

2) Improved user experience in CyberGIS-Compute job submission: Have you ever had a long-running job submitted to high-performance computing (HPC) resources but found your Jupyter session died after the browser was idle for too long? The latest CyberGIS-Compute SDK now allows you to reinstate job submission sessions for all previous jobs you submitted. Just switch to the new “Your Jobs” tab page in the user environment and “Restore” the jobs you are interested in. This also gives you a chance to re-download model outputs from previous jobs.

3) WRFHydro model integration supports merging model outputs: A new option “Merge_Output” is added to the WRFHydro workflow supported by CyberGIS-Compute. If enabled, single-timestep NetCDF files can be merged on the “Time” dimension after model execution. Currently supported output types include CHANOBS, LDASOUT, GWOUT, LAKEOUT, RTOUT, and LSMOUT. This optional merging step can reduce data transfer size and speed up post-processing work on CJW. The merged files are put into a separate folder called “Outputs_Merged” alongside the original model outputs. Users can choose to download either or both. Please refer to the example notebook for more information.

4) Enhanced support for user customization to CJW kernels: While CJW has pre-installed a large collection of common libraries and tools to support a suite of hydrologic analysis and modeling workflows, users may still want to install something specific to certain use cases. CJW now allows users to directly use “!pip install XXX” in notebook cells to customize existing kernels. CJW supports flexible additions or changes on a per-kernel basis, which does not affect other existing kernels. Please refer to this example notebook for more information.

5) Updates on CJW backend (kernel, plugin, and bugfix): A new general-purpose kernel, Python3-2022-06, is added, which incorporates a rich set of new geospatial packages. The ‘StickyLand” JupyterLab plugin is installed that allows users to create customizable dashboards and linear notebooks; A bug specific to Apple Safari browser in the OpenWith operation has been fixed.

Please refer to the following resources for details and examples:

Run WRFHydro model on HPC resources using CyberGIS-Compute V2 (updated 2022-07)
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Customization to CJW Kernels with Pip
https://www.hydroshare.org/resource/d18886d2aedf4a2e8c6302165b8fe10f/

CyberGIS-Compute SDK new features
https://cybergis.github.io/cybergis-compute-python-sdk/release-notes.html

CJW 2022-Q2 Release Notes on HydroShare
https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Compute is a scalable middleware framework for enabling high-performance and data-intensive geospatial research and education on CyberGISX. This API can be used to send supported jobs to various supported HPC & computing resources.

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q3)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

(1) Cern Virtual Machine File System (CVMFS): We have redesigned how we deliver software within CyberGIS-Jupyter. This new design drastically increases computational performance and reproducibility, and allows the platform to make the software environment available in a variety of settings. From an end-user perspective, there should be no change to your accessing and utilizing the CJW services.

(2) Improved user experience for CyberGIS-Compute: In previous versions, we introduced the capability for users to “Restore” their previously submitted jobs of interest. Based on user feedback, we’ve further refined the interface to support viewing and downloading outputs of all previously submitted jobs by simply navigating to the “Past Results” section. The result/output of any completed job can be accessed with a single click.

(3) Support for new High Performance Computing (HPC) backend in CyberGIS-Compute: Anvil is now available as a new HPC resource for CyberGIS-Compute. Supported by NSF, Anvil is a HPC system hosted at Purdue University that contains 1000 CPU nodes based on the third generation AMD EPYC "Milan" processor, delivering a peak performance of 5.3 petaflops. Allocations on Anvil are managed by NSF's ACCESS program (https://access-ci.org/). The large numbers of CPU nodes and cores (i.e., 128) enable superior computational performance for scalable codes, short queuing times, and fast execution for hydrologic models via CyberGIS-Compute. For more information on Anvil, refer to the documentation at: https://www.rcac.purdue.edu/anvil. The WRFHydro model is supported on Anvil via CyberGIS-Compute. Please refer to the example notebook below.

Please refer to the following resources for details and examples:

A Brief Overview Of Cern Virtual Machine File System (CVMFS)
http://www.hydroshare.org/resource/ab1555c0c8d34d3496997353ba8060d9

CyberGIS-Compute updates - 2022-Q3
http://www.hydroshare.org/resource/3b472641c3504161bb13a19d4c9fbc87

Submission of WRFHydro model to Anvil HPC
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

See Release Notes on HydroShare
http://www.hydroshare.org/resource/bf463f07e1244de4a17b3ea7b2d95916

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.
Best regards,
CyberGIS-Hydro team