Zhiyu/Drew Li

University of Illinois at Urbana-Champaign

 Recent Activity

ABSTRACT:

This resource provides a quick way to discover and navigate through different example notebooks developed in the CyberGIS-Jupyter for Water (CJW) webapp.
The collection is being updated periodically as new resources will be added and old ones gets deprecated.

How to launch a notebook with CJW:
0) Request to join the CyberGIS-Jupyter for Water Group on HydroShare (one-time effort): Click on the “Ask to join” button in the lower left corner, and a request will normally get approved very quick if your user profile is complete and up-to-date.
1) Click on a resource below to open up the landing page of selected resource;
2) Click on the OpenWith button in the upper-right corner and select "CyberGIS-Jupyter for Water";
3) Follow the instructions in the notebook;

Show More

ABSTRACT:

Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/.

Show More

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1)Community Commons for Easy Access to HydroShare Notebooks:
Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/. Interested in learning how CJW enables computational reproducibility of such notebooks? Read on!

2)Enhanced Reproducibility of CJW:
Starting from this release, CJW has implemented a new solution in kernel and software management such that each notebook is now associated with a specific versioned kernel. This approach significantly enhances reproducibility in the CJW environment as each notebook is always opened with the original kernel it is tied to. We encourage users to test their notebooks and reach out to us if you experience any problems.

3)Transferring Large Model Output Datasets:
CJW has introduced a new Globus-based (https://www.globus.org/) data-transfer capability to enhance the performance and stability of transferring a large number of model outputs from high-performance computing (HPC) resources back to CJW, which is a common scenario in running hydrologic models.

4)Seamless Transition from XSEDE Comet to Expanse:
Comet was decommissioned on July 15, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW have been transferred to the Expanse (https://portal.xsede.org/sdsc-expanse). The CyberGIS-Compute service and its SDK now automatically redirect all jobs submitted to Comet to Expanse while prompting users with a simple warning message.

Please refer to the following HydroShare resources for details and examples:
Community Commons for Easy Access to HydroShare Notebooks
https://cybergisxhub.cigi.illinois.edu/hydroshare/
Enhanced Reproducibility of CJW
https://www.hydroshare.org/resource/df8244042e5445edb106d93e5a491d29/
Transferring Large Model Output Datasets
https://www.hydroshare.org/resource/5aa3d27ec5ef4b1b8dfa621ea284af14/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/f3315ec8c1df4f4ab5d2274220de0351/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

Show More

 Contact

Resources
All 0
Collection 0
Composite Resource 0
Generic 0
Geographic Feature 0
Geographic Raster 0
HIS Referenced Time Series 0
Model Instance 0
Model Program 0
MODFLOW Model Instance Resource 0
Multidimensional (NetCDF) 0
Script Resource 0
SWAT Model Instance 0
Time Series 0
Web App 0
Composite Resource Composite Resource
US States
Created: Nov. 3, 2015, 5:43 p.m.
Authors: Drew Li

ABSTRACT:

US States polygon shapefiles

Show More
Composite Resource Composite Resource
NHD Flowlines Colorado Region
Created: Dec. 11, 2015, 8:07 p.m.
Authors: Drew Li

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

Show More
Composite Resource Composite Resource
NHD Flowlines Colorado Region
Created: Aug. 11, 2016, 4:50 p.m.
Authors: Drew Li

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

Show More
Composite Resource Composite Resource
gdal-2.1.2-binary
Created: Nov. 17, 2016, 7:22 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This gdal 2.1.2 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains a sqlite/spatialite geodatabase for assisting the workflow of subsetting NWM Ver1.1 netcdf.
1) grid cell polygon for land and forcing files
2) stream polyline (huc 8, 10, 12)
3) reservoir polygon

Download all the 5 split zip files into one folder and unzip the first one (nwm.zip.001) using 7z

Show More
Composite Resource Composite Resource

ABSTRACT:

Subset National Water Model (NWM) Ver1.1 20170327 results for Bear River-Frontal Great Salt Lake HUC12 watershed (comid: 160102040504).
Python library used to prepare this data: https://pypi.python.org/pypi/subset_nwm_netcdf/

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.3 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.
xy_land_NAD1983.tif is for querying gird cell indices of NWM forcing and land files
xy_terrain_NAD1983.tif is for querying gird cell indices of NWM terrain files

Show More
Composite Resource Composite Resource

ABSTRACT:

A subset of National Water Model (NWM) Ver1.1 20170404 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

Show More
Composite Resource Composite Resource
TwoMileCreek watershed at Tuscaloosa, Alabama
Created: April 12, 2017, 6:03 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

TwoMileCreek watershed at Tuscaloosa, Alabama

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.4-1.1.9 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.

Download and unzip on Ubuntu
sudo apt-get install wget unzip
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.001
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.002
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.003
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.004
cat nwm.zip.* > nwm.zip
rm nwm.zip.*
unzip nwm.zip
rm nwm.zip

Show More
Composite Resource Composite Resource

ABSTRACT:

A Subset of National Water Model (NWM) Ver1.1 20170419 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

Show More
Composite Resource Composite Resource
bash script to download NWM v1.1 outputs
Created: April 29, 2017, 3:59 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

bash script to download NWM v1.1 outputs

Show More
Composite Resource Composite Resource

ABSTRACT:

Hand drawn Utah state border in geojson featurecollection format Projection: WGS84 (EPSG: 4326). This resource was created to test NWM viewer app.

Show More
Composite Resource Composite Resource

ABSTRACT:

Hand drawn Alabama state border in geojson polygon format. This resource was created to test NWM viewer app.

Show More
Composite Resource Composite Resource

ABSTRACT:

NWM v1.1 forcing_analysis_assimilation files of TwoMileCreek watershed region for 20170419

Show More
Collection Resource Collection Resource
HydroShare App Demos
Created: June 7, 2017, 12:56 a.m.
Authors: Christina Bandaragoda

ABSTRACT:

This is a collection of step by step demonstrations on how to use HydroShare Apps.

Show More
Composite Resource Composite Resource
Demo: View multimedia from maps
Created: June 7, 2017, 12:59 a.m.
Authors: Jimmy Phuong · Christina Bandaragoda

ABSTRACT:

This is a step-by-step demonstration of how to Add Images, PDFs, and Videos to digital maps using the HydroShare GIS App using an example from this related HydroShare resource: Ames, D. (2016). Algae Growth in Utah Lake Time-lapse, HydroShare, http://www.hydroshare.org/resource/4c8ecb05a72647339df0df6e9a87718f

Show More
Composite Resource Composite Resource

ABSTRACT:

This is a step-by-step demonstration of how to view and download forecasts from any stream in the National Hydrography Dataset with the National Water Model App.

Show More
Composite Resource Composite Resource
stream_sld
Created: June 20, 2017, 11:48 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

stream_sld

Show More
Composite Resource Composite Resource
Thunder Creek Watershed Boundary
Created: July 25, 2017, 9:07 p.m.
Authors: Christina Bandaragoda

ABSTRACT:

Thunder Creek, Skagit River Basin, State of Washington, USA.

Show More
Web App Resource Web App Resource
National Water Model Forecast Viewer App
Created: Dec. 7, 2017, 3:33 p.m.
Authors: · michael souffront · Zhiyu (Drew) Li · Jim Nelson · Dan Ames

ABSTRACT:

The NWM Viewer app has 2 main features provided in Home Mode and Subset Mode respectively:
Home Mode: Retrieve and View NWM Time Series for a single stream reach, reservoir or grid cell.
Subset Mode: Subset NWM outputs (NetCDF files) with a watershed polygon to get 'shrunken' NetCDFs that only contain data for a specific area.

Show More
Web App Resource Web App Resource
Data Rods Explorer App
Created: Dec. 7, 2017, 3:38 p.m.
Authors: Gonzalo E. Espinoza · David Arctur ·

ABSTRACT:

The Data Rods Explorer (DRE) is a web client app that enables users to browse several NASA-hosted data sets. The interface enables visualization and download of NASA observation retrievals and land surface model (LSM) outputs by variable, space and time. The key variables are precipitation, wind, temperature, surface downward radiation flux, heat flux, humidity, soil moisture, groundwater, runoff, and evapotranspiration. These variables describe the main components of the water cycle over land masses.

Show More
Composite Resource Composite Resource
Rocky River HUC10 0303000305 at Raleigh NC
Created: Dec. 19, 2017, 5:05 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

Rocky River HUC10 0303000305 at Raleigh NC

Show More
Composite Resource Composite Resource
GDAL2.2.3-ubuntu14-x64-binary
Created: Dec. 29, 2017, 5:57 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This gdal 2.2.3 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python

GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal223build
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=gnu++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++11 support: yes

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

Show More
Composite Resource Composite Resource
GDAL2.3-dev-bb4c395-ubuntu16-x64-binary
Created: Dec. 30, 2017, 12:50 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This GDAL2.3-dev-bb4c395 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python
GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal_23_dev_bb4c395_binary
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=c++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++14 support: no

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
crypto/openssl support: yes
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
JPEG-Lossless/CharLS: no
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

Show More
Composite Resource Composite Resource
NWM Viewer App GIS data
Created: Jan. 9, 2018, 6:28 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This resource contains GIS data for National Water Model Viewer App in ArcGIS geodatabase format:
channels : line--2.7 million NHD+ stream reaches
reservoirs: point--1260 reservoirs locations
usgs_gauge: point--NHD+ USGS gauges locations
grid_land: polygon--grid cell polygons for land and forcing outputs

nwm_app_data.mxd : ArcMap 10 project file

Show More
Composite Resource Composite Resource
LittleWashita HUC10 1113030208 Polygon in OH
Created: Jan. 18, 2018, 12:06 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

LittleWashita HUC10 1113030208 Polygon in OH

Show More
Composite Resource Composite Resource

ABSTRACT:

Milburnie Lake - Neuse River HUC10 Watershed 0302020107 at Durham-Raleigh, NC

Show More
Composite Resource Composite Resource
croton_NY_domain_polygon
Created: May 16, 2018, 10:47 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

croton_NY_domain_polygon

Show More
Web App Resource Web App Resource
CyberGIS (beta) JupyterHub
Created: Sept. 26, 2018, 5:53 p.m.
Authors: Dandong Yin

ABSTRACT:

Jupyter environment set in CyberGIS Center for interaction with HPC

Show More
Composite Resource Composite Resource

ABSTRACT:

The US National Water Model (NWM) is a mesoscale hydrologic model that provides streamflow forecasts and other valuable hydrologic information for the continental United States. Since its release at the National Water Center (NWC) in 2016, the NWM has garnered broad attention and great interest across the hydrology science community. Several projects are underway with the goal of delivering this advanced modeling technique and its data to researchers and end users in the community.
As one of the flagship projects, the CUAHSI HydroShare project is working toward providing a complete solution for storing, managing and sharing NWM data. So far, it has set up the largest open-access data archive for the NWM outputs and has developed several different open-source tools and web applications assisting users with data access.
However, there is a growing demand in the hydrologic sciences community for the capability to run a local instance of NWM at regional watersheds to support research applications such as cross-model comparison, historical data analysis and etc.
In this paper, we present ongoing work to develop web applications on top of HydroShare for collecting NWM input data to support model execution at smaller scale watersheds.

Show More
Composite Resource Composite Resource
Model Instance Resource Model Instance Resource
WRF-Hydro Test Case at Croton New York
Created: March 7, 2019, 9:58 p.m.
Authors: david gochis

ABSTRACT:

This example test case includes a small region (15km by 16km) encompassing the West Branch of the Croton River, NY, USA (USGS stream gage 0137462010) during hurricane Irene, 2011-08-26 to 2011-09-02. The simulation begins with a restart from a spinup period from 2010-10-01 to 2011-08-26. There are 3 basic routing configurations included in the test case, National Water Model (NWM), Gridded, and NCAR Reach.

Show More
Composite Resource Composite Resource
czo community sql files
Created: April 9, 2019, 4:23 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

czo community sql files

Show More
Composite Resource Composite Resource
watershed test
Created: May 6, 2019, 7:59 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

watershed test

Show More
Web App Resource Web App Resource
CyberGIS-Jupyter for Water (dev at hs07)
Created: Sept. 4, 2019, 3:30 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

Dev deploy for CyberGIS-Jupyter for Water

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?id=${HS_RES_ID}%2526subfolder=Downloads

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?start=${HS_FILE_PATH}%2526id=${HS_RES_ID}%2526subfolder=Downloads

Show More
Web App Resource Web App Resource

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resources contains 3 notebooks that walk you through running a SUMMA model on CyberGIS-Jupyter for Water platform.

summa_local.ipynb --- run a summa model on Jupyter notebook server (container) directly
summa_hpc.ipynb --- submit summa model as a job to XSEDE COMET High Performance Computing (HPC) cluster
summa_ensemble_hpc.ipynb --- submit an ensemble summa mode as a job to XSEDE COMET High Performance Computing (HPC) cluster

Show More
Composite Resource Composite Resource

ABSTRACT:

User may want to set up a custom Python environment (kernel) and run notebooks with it on the CyberGIS-Jupyter for Water platform. This resource has 2 notebooks to demonstrate walk you through the steps.

install_custom_python_environment.ipynb --- install a custom Python env with user-defined libraries using conda, and set it as a Jupyter Kernel.
recover_custom_python_environment.ipynb --- after a container rebuild, user needs to reactivate a previously installed custom Python env.

Show More
Composite Resource Composite Resource

ABSTRACT:

This Jupyter notebook illustrates the HAND workflow and its use in example flood emergency scenarios. The study area is Onion Creek (HUC10 code 1209020504).

This is also a demonstration of conducting geospatial anlysis with opensource toolkits (gdal) using an online Jupyter interface.

Environment required: CyberGIS-Jupyter for Water

Show More
Composite Resource Composite Resource
csvtest
Created: March 29, 2020, 4:01 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

asd

Show More
Composite Resource Composite Resource

ABSTRACT:

This example demonstrates how to use PostGIS capabilities in CyberGIS-Jupyter notebook environment. Modified from notebook by Weiye Chen (weiyec2@illinois.edu)

PostGIS is an extension to the PostgreSQL object-relational database system which allows GIS (Geographic Information Systems) objects to be stored in the database. PostGIS includes support for GiST-based R-Tree spatial indices, and functions for analysis and processing of GIS objects.

Resources for PostGIS:

Manual https://postgis.net/docs/
In this demo, we use PostGIS 3.0. Note that significant changes in APIs have been made to PostGIS compared to version 2.x. This demo assumes that you have basic knowledge of SQL.

Show More
Composite Resource Composite Resource

ABSTRACT:

These are examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 1_watershed_delineation folder is an example of a watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

Show More
Composite Resource Composite Resource

ABSTRACT:

!!! This is a fork from https://www.hydroshare.org/resource/5b964154ebf945848087bdc772cc921e/ with some minor modifications for CyberGIS-Jupyer for Water (CJW) platform !!!
----------------------------------------------------------------------------------------------------
The ability to test hypotheses about hydrology, geomorphology, and atmospheric processes is invaluable to research in the Earth and planetary sciences. To swiftly develop experiments using community resources is an extraordinary emerging opportunity to accelerate the rate of scientific advancement. Knowledge infrastructure is an intellectual framework to understand how people are creating, sharing, and distributing knowledge -- which has dramatically changed and is continually transformed by Internet technologies. We are actively designing a knowledge infrastructure system for earth surface investigations. In this paper, we illustrate how this infrastructure can be utilized to lower common barriers to reproducing modeling experiments. These barriers include: developing education and training materials for classroom use, publishing research that can be replicated by reviewers and readers, and advancing collaborative research by re-using earth surface models in new locations or in new applications. We outline six critical elements to this infrastructure, 1) design of workflows for ease of use by new users; 2) a community-supported collaborative web platform that supports publishing and privacy; 3) data storage that may be distributed to different locations; 4) a software environment; 5) a personalized cloud-based high performance computing (HPC) platform; and 6) a standardized modeling framework that is growing with open source contributions. Our methodology uses the following tools to meet the above functional requirements. Landlab is an open-source modeling toolkit for building, coupling, and exploring two-dimensional numerical models. The Consortium of Universities Allied for Hydrologic Science (CUAHSI) supports the development and maintenance of a JupyterHub server that provides the software environment for the system. Data storage and web access are provided by HydroShare, an online collaborative environment for sharing data and models. The knowledge infrastructure system accelerates knowledge development by providing a suite of modular and interoperable process components that can be combined to create an integrated model. Online collaboration functions provide multiple levels of sharing and privacy settings, open source license options, and DOI publishing, and cloud access to high-speed processing. This allows students, domain experts, collaborators, researcher, and sponsors to interactively execute and explore shared data and modeling resources. Our system is designed to support the user experiences on the continuum from fully developed modeling applications to prototyping new science tools. We have provided three computational narratives for readers to interact with hands-on, problem-based research demonstrations - these are publicly available Jupyter Notebooks available on HydroShare.

To interactively compute with these Notebooks, please see the ReadMe below.
To develop these Notebooks, go to Github: https://github.com/ChristinaB/pub_bandaragoda_etal_ems or https://zenodo.org/badge/latestdoi/187289993

Show More
Collection Resource Collection Resource

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2020 Q2)

Dear HydroShare Users,

We are pleased to announce a new quarterly release of CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes new capabilities to support the geoanalytics suite of GRASS for model pre/post-processing, PostGIS database, and Landlab Earth surface modelling toolkit along with several enhancements to job submission middleware, system security as well as service infrastructure. Please refer to the following list for details and examples.

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

GRASS GIS for model pre/post-processing:
Learn how to consolidate the features of the GRASS geoanalytics suite to support pre/post-processing for SUMMA and RHESSYs models in CJW.
Example notebooks: https://www.hydroshare.org/resource/4cbcfdd6e7f943e2969dd52e780bc52d/

Manage geospatial data with PostGIS:
PostGIS is an extension to the PostgreSQL object-relational database system which allows geospatial data to be efficiently stored while providing various advanced functions for in-situ data analysis and processing.
Example notebooks: https://www.hydroshare.org/resource/bb779d4cce564dd6afcf463c8910786f/

Security and service infrastructure enhancements
Trusted group: Starting from this release, all users are required to join the “CyberGIS-Jupyter for Water” trusted group at https://www.hydroshare.org/group/157 in order to access the CJW platform, which is a preventive measure to protect the shared computing resources from being abused by malicious users. A complete user profile page is highly recommended to expedite the approval process.
User metric submission to XSEDE: CJW, as a science gateway, is now sending unique user usage metrics to XSEDE to comply with its requirements.

Landlab for enabling collaborative numerical modeling in Earth sciences using knowledge infrastructure
Example notebooks: https://www.hydroshare.org/resource/370c288b61b84794b847ef85c4dd4ffb/
https://www.hydroshare.org/resource/6add6bee06bb4050bfe23e1081627614/

Job submission enhancements
Refactored the structure of the cyberGIS job submission system
Data-driven implementation for avoiding excessive data transmission between HydroShare and CJW
Add the specification of input parameters into a JSON file to improve the flexibility and generality of model management
Enable HPC-SUMMA object that can directly call SUMMA
Example notebooks: https://www.hydroshare.org/resource/4a4a22a69f92497ead81cc48700ba8f8/

Show More
Composite Resource Composite Resource

ABSTRACT:

This is an example of watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.

Show More
Composite Resource Composite Resource

ABSTRACT:

These is an examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

Show More
Composite Resource Composite Resource
sciunit
Created: May 14, 2020, 4:51 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

sciunit container

Show More
Model Program Resource Model Program Resource

ABSTRACT:

This HydroShare resource provides a Singularity image for Remote Approach 11: Using an HPC Cluster in the manuscript of "Comparing Approaches to Achieve Reproducible Computational Modeling for Hydrological and Environmental Systems" in Environmental Modeling and Software.

For more detailed information, please see this GitHub
https://github.com/DavidChoi76/Using_Singularity_in_Local_Approach_4_and_HPC_Cluster_Approach_11_for_the_reproducibility_of_SUMMA

Show More
Composite Resource Composite Resource

ABSTRACT:

This a reproducible demonstration of the landslide modeling results from eSurf paper: Strauch et al. (2018)

Show More
Composite Resource Composite Resource

ABSTRACT:

This example is to show the steps to run an ensemble SUMMA3.0 on HPC through the CyberGIS-Compute Service.

Show More
Composite Resource Composite Resource
NLDAS Forcing NetCDF using CAMELS datasets from 1980 to 2018
Created: Nov. 4, 2020, 8:57 p.m.
Authors: Naoki Mizukami · Wood, Andrew

ABSTRACT:

This resource was created using CAMELS (https://ral.ucar.edu/solutions/products/camels) `TIME SERIES NLDAS forced model output` from 1980 to 2018.
The original NLDAS (North American Land Data Assimilation System) hourly forcing data was created by NOAA by 0.125 x 0.125 degree grid.
Through creating CAMELS datasets, hourly forcing data were reaggregated to 671 basins in the USA.
In this study, we merged all CAMELS forcing data into one NetCDF file to take advantage of OPeNDAP (http://hyrax.hydroshare.org/opendap/hyrax/) in HydroShare.
Currently, using SUMMA CAMELS notebooks (https://www.hydroshare.org/resource/ac54c804641b40e2b33c746336a7517e/), we can extract forcing data to simulate SUMMA in the particular basins in 671 basins of CAMELS datasets.

Show More
Composite Resource Composite Resource
WRFHydro Test Case -- Croton River, NY
Created: Nov. 5, 2020, 4:58 a.m.
Authors:

ABSTRACT:

[You can run this model with the notebook at https://www.hydroshare.org/resource/8fe974c108ca4c6eaaf9b060779329b0/ in CyberGIS-Jupyter for Water platform]

WRFHydro Test Case -- Croton River, NY

#Overview This test case includes prepared geospatial data and input files for a
sample domain (region of interest) and prepared forcing data. This domain is a small region (15km x 16km) encompassing the West Branch
of the Croton River, NY, USA (USGS stream gage 0137462010) during hurricane
Irene, 2011-08-26 to 2011-09-02. The simulation begins with a restart from a
spinup period from 2010-10-01 to 2011-08-26. The forcing data
prepared for this test case is North American Land Data Assimilation System
(NLDAS) hourly data. There are 3 basic routing
configurations included in the test case, National Water Model (NWM), Gridded,
and NCAR Reach. See the WRF-Hydro V5 Technical Description located at
https://ral.ucar.edu/projects/wrf_hydro for a more detailed description of model
physics options, configurations, and input files.

Show More
Composite Resource Composite Resource

ABSTRACT:

The goal of this notebook is to show the steps to run an example National Water Model (WRFhydro) model on HPC resources through the CyberGIS-Compute Service. This notebook uses wrfhydropy, a Python wrapper for WRFHydro, in model preprocessing and postprocessing, and the resulting ready-to-run model is handed over to CyberGIS Computing Service for execution on a supported HPC resource (Virtual Roger/Keeling at UIUC or XSEDE COMET at SDSC). This example is adapted from the "ex_01_end_to_end.ipynb" notebook from wrfhydropy official github repo https://github.com/NCAR/wrf_hydro_py, and users are encouraged to refer to the tutorials there to get familair with wrfhydropy usages.

How to run:
1. Request to join the CyberGIS-Jupyter for Water group at https://www.hydroshare.org/group/157
2. Click the "Open with ..." button in the upper-right
3. Select "CyberGIS-Jupyter for Water"
4. Run through the notebook

Show More
Model Program Resource Model Program Resource
RHESSys East Coast version v7.2
Created: Dec. 19, 2020, 7:16 a.m.
Authors: Choi, Young-Don

ABSTRACT:

RHESSys East Coast version v7.2

Show More
Composite Resource Composite Resource

ABSTRACT:

These are example application notebooks to simulate SUMMA using CAMELS datasets.
There are three steps: (STEP-1) Create SUMMA input, (STEP-2) Execute SUMMA, (STEP-3) Visualize SUMMA output
Based on this example, users can change the HRU ID and simulation periods to analyze 671 basins in CAMELS datasets.

(STEP-1) A_1_camels_make_input.ipynb
- The first notebook creates SUMMA input using Camels dataset using `summa_camels_hydroshare.zip` in this resource and OpenDAP(https://www.hydroshare.org/resource/a28685d2dd584fe5885fc368cb76ff2a/).
(STEP-2) B_1_camels_pysumma_default_prob.ipynb, B_2_camels_pysumma_lhs_prob.ipynb, B_3_camels_pysumma_config_prob.ipynb, and
B_4_camels_pysumma_lhs_config_prob.ipynb
- These four notebooks execute SUMMA considering four different parameters and parameterization combinations
(STEP-3) C_1_camels_analyze_output_default_prob.ipynb, C_2_camels_analyze_output_lhs_prob.ipynb, C_3_camels_analyze_output_config_prob.ipynb,
C_4_camels_analyze_output_lhs_config_prob.ipynb
- The final four notebooks visualize SUMMA output of B-1, B-2, B-3, and B-4 notebooks.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms using the CyberGIS-Jupyter for Water (CJW) environment's supported High-Performance Computing (HPC) resource (XSEDE Comet) through CyberGIS-Compute Service, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021).

For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook utilizes the CJW environment's supported HPC resource (XSEDE Comet) through CyberGIS-Compute Service to executes SUMMA model. This notebook uses the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice. As this resource uses HPC, it enables a high-speed running of simulations which makes it suitable for larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month simulation period used in the paper) practical and much faster than when no HPC is used.

Show More
Web App Resource Web App Resource
CJW-k8s-test-js-169-80
Created: Feb. 13, 2021, 5:33 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

CJW K8s test
http://js-169-80.jetstream-cloud.org/

Show More
Composite Resource Composite Resource

ABSTRACT:

This notebook is created to support SUMMA general application workflows using CAMELS forcing, watershed attributes, and streamflow observation.
CAMELS datasets cover 671 basins across the USA, so users can apply SUMMA models in 671 basins.

Show More
Composite Resource Composite Resource
Run RHESSys model with CyberGIS-Compute Service on CJW
Created: March 1, 2021, 8:54 p.m.
Authors: Choi, Young-Don

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrological modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically-based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates parallel job submissions of RHESSys ensemble simulations from CyberGIS-Jupyer for water to HPC (XSEDE), visualizes RHESSys output, and evaluate RHESSys efficiency with simulation runoff and observation streamflow

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021). For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

Show More
Composite Resource Composite Resource

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS.

This resource contains basin attributes and parameters in NetCDF files.

Show More
Composite Resource Composite Resource

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies) is a large-sample hydrometeorological dataset that provides catchment attributes, forcings and GIS data for 671 small- to medium-sized basins across the CONUS (continental United States). HydroShare hosts a copy of CAMELS and exposes it through different public data access protocols (WMS, WFS and OPeNDAP) for easy visualization and subsetting of the dataset in community modeling research. This notebook demostrates how to set up SUMMA models with CAMELS dataset from HydroShare using various tools integrated in the CyberGIS-Jupyter for Water (CJW) environment and execution of ensemble model runs on a supported High-Performance Computing (HPC) resource (XSEDE Comet or UIUC Virtual Roger) through CyberGIS-Compute Service.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrologic modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates how to configure an ensemble RHESSys simulation with pyRHESSys, submit it to a supported HPC resource (XSEDE COMET or UIUC Virtual Roger) for execution through CyberGIS Computing Service, visualize model outputs with various tooks integrated in the CyberGIS-Jupyter for Water (CJW).

The model used here is based off of a pre-built RHESSys model for the Coweeta Subbasin 18 (0.124 𝑘𝑚2 ), a subbasins in Coweeta watershed (16 𝑘𝑚2 ), from the Coweeta Long Term Ecological Research (LTER) Program.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Web App Resource Web App Resource

ABSTRACT:

A newly deployed CyberGIS-Jupyter for Water (CJW) instance powered by Kubernetes (k8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology would significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user sign-up. We welcome all users to join the public testing and give us feedback. We anticipate the public testing on the new CJW would take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final “migration plan” is announced (TBD).

Please send comments and bug reports to help@cybergis.org

URL for direct access: https://go.illinois.edu/cjw-k8s

How to setup OpenWith for "Kubernetes-based CJW (user testing)"
1) Login HydroShare
2) Visit this resource landing page: https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
3) In the upper-right corner, click on the 3rd icon from the left (the one with 3x3 small squares), which should prompt "Add WebApp to Open With List"
4) Refresh the landing page of the resource that has notebooks you are interested in, and "Kubernetes-based CJW (user testing)" should show up in the OpenWith list now

Show More
Collection Resource Collection Resource

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Show More
Composite Resource Composite Resource

ABSTRACT:

How to Fix the Side Effect caused by New SSL Cert on HydroShare

Revisions:
March 18, 2021; Zhiyu/Drew Li; zhiyul@illinois.edu

Symptoms:
Jupyter Hub fails in OAuth handshaking with HydroShare
“HTTP 599: server certificate verification failed. CAfile: none CRLfile: none”
hs_restclient fails to authenticate
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.hydroshare.org', port=443): Max retries exceeded with url: /hsapi/userInfo/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

Cause:
HydroShare deployed a new SSL cert on March 17, 202. It is based on off a new CA, which is NOT included in the latest “ca-certificates” package (CA Bundle) on Ubuntu 18.04 and 20.04 as of this writing (other Linux distribution may also be affected).

Remedy:
Manually add this new CA into the CA Bundle on all clients that might need to talk to HydroShare.

Download the new CA cert:
Go to HydroShare keybase and download: star_hydroshare_org_124173627DigiCertCA.crt
Go to https://www.digicert.com/kb/digicert-root-certificates.htm, search for “GeoTrust TLS DV RSA Mixed SHA256 2020 CA-1” and download PEM format.

For Hub Dockerfile:

USER root
# get latest ca-bundle
RUN apt-get update && apt-get install -y ca-certificates
# load hydroshare new ca to image
COPY ./star_hydroshare_org_124173627DigiCertCA.crt /usr/local/share/ca-certificates/star_hydroshare_org_124173627DigiCertCA.crt
# update ca-bundle
RUN update-ca-certificates

For different conda envs in Dockerfile:

#Append new HydroShare CA to cacert.pem in Base conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/lib/python<VERSION>/site-packages/certifi/cacert.pem
# Append new HydroShare CA to user-created conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/envs/<ENV_NAME>/lib/python<VERSION>/site-packages/certifi/cacert.pem

References:
https://incognitjoe.github.io/adding-certs-to-requests.html
https://www.techrepublic.com/article/how-to-install-ca-certificates-in-ubuntu-server/

Show More
Collection Resource Collection Resource

ABSTRACT:

The overall goal of this collection is to use the basic strategy and architecture presented by Choi et al. (2021) to make components of a modern and complex hydrologic study (VBstudy; Van Beusekom et al., 2021) easier to reproduce.

In VBstudy, hydrological outputs from the SUMMA model for the 671 CAMELS catchments across the contiguous United States (CONUS) are investigated to understand their dependence on input forcing behavior across CONUS. VBstudy layes out a simple methodology that can be applied to understand the relative importance of seven model forcings (precipitation rate, air temperature, longwave radiation, specific humidity, shortwave radiation, wind speed, and air pressure).

Choi et al. (2021) integrated three components through seamless data transfers for a reproducible research: (1) online data and model repositories; (2) computational environments leveraging containerization and self-documented computational notebooks; and (3) Application Programming Interfaces (APIs) that provide programmatic control of complex computational models.

Therefore, in the current research, we integrated the following three components through seamless data transfers to make components of a modern and complex hydrologic study (VBstudy) easier to reproduce:
(1) HydroShare as online data and model repository;
(2) CyberGIS-Jupyter for Water for self-documented computational notebooks as computational environment (with and without HPC notebooks);
(3) pySUMMA as Application Programming Interfaces (APIs) that provide programmatic control of complex computational models.

This collection includes three resources:

1- First resource, provides the entire NLDAS forcing datasets used in the paper.

2- Second resource provides an end-to-end workflow of CAMELS basin modeling with SUMMA for the paper simulations configured for execution in connected JupyterHub compute platforms. This resource is well-suited for a smaller scale exploration: it explores one example CAMELS site and a period of 18-month simulation to only demonstrate the capabilities of the notebooks.

3- Third resource, however, uses HPC (High-Performance Computing) through CyberGIS Computing Service. The HPC enables a high-speed running of simulations which makes it suitable for running larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month simulation period used in the VBstudy) practical and much faster than the second resource.

Greater details can be found in each resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource is an example to demonstrate the vPICO presentations in EGU General Assembly 2021 (https://meetingorganizer.copernicus.org/EGU21/session/40092#vPICO_presentations).
- Session: EOS5.3 session - The evolving open-science landscape in geosciences: open data, software, publications, and community initiatives
- Title: An Approach for Open and Reproducible Hydrological Modeling using Sciunit and HydroShare

Using this notebook, you can test how to create an immutable and interoperable Sciunit Container for open and reproducible hydrological modeling.

You can start using "NB_01_An_Approach_for_Open_and_Reproducible_Hydrological_Modeling_using_Sciunit_and_HydroShare.ipynb" notebook in "CyberGIS-Jupyter for water" after clicking "Open with...". in Right-Above.

Show More
Composite Resource Composite Resource
(HS 3) Large Spatial Sample Datasets in Maryland
Created: April 10, 2021, 1:01 a.m.
Authors: Choi, Young-Don

ABSTRACT:

This HydroShare resource was created to share large spatial sample datasets in Maryland on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can check the uploaded LSS datasets on HydroShare-GeoServer and THREDDS using this HS resource id.

Then, through the RHESSys workflows, users can subset LSS datasets using OWSLib and xarray.

Show More
Composite Resource Composite Resource
(HS 4) Large Spatial Sample Datasets in Virginia
Created: April 25, 2021, 12:26 a.m.
Authors: Choi, Young-Don

ABSTRACT:

This HydroShare resource was created to share large spatial sample datasets in Virginia on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can check the uploaded LSS datasets on HydroShare-GeoServer and THREDDS using this HS resource id.

Then, through the RHESSys workflows, users can subset LSS datasets using OWSLib and xarray.

Show More
Composite Resource Composite Resource
(HS 2) Large Spatial Sample Datasets in North Carolina
Created: April 25, 2021, 12:27 a.m.
Authors: Choi, Young-Don

ABSTRACT:

This HydroShare resource was created to share large spatial sample datasets in North Carolina on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can check the uploaded LSS datasets on HydroShare-GeoServer and THREDDS using this HS resource id.

Then, through the RHESSys workflows, users can subset LSS datasets using OWSLib and xarray.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks created for the study "An Approach for Creating Immutable and Interoperable End-to-End Hydrological Modeling Computational Workflows" led by researcher Young-Don Choi submitted to the 2021 EarthCube Annual meeting, Notebook Sessions.

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file provided in this resource.

For the sake of completeness, the abstract for the study submitted to the EarthCube session is mentioned below:

"Reproducibility is a fundamental requirement to advance science. Creating reproducible hydrological models that include all required data, software, and workflows, however, is often burdensome and requires significant work. Computational hydrology is a rapidly advancing field with fast-evolving technologies to support increasingly complex computational hydrologic modeling. The growing model complexity in terms of variety of software and cyberinfrastructure capabilities makes achieving computational reproducibility extremely challenging. Through recent reproducibility research, there have been efforts to integrate three components: 1) (meta)data, 2) computational environments, and 3) workflows. However, each component is still separate, and researchers must interoperate between these three components. These separations make verifying end-to-end reproducibility challenging. Sciunit was developed to assist scientists, who are not programming experts, with encapsulating these three components into a container to enable reproducibility in an immutable form. However, there were still limitations to support interoperable computational environments and apply end-to-end solutions, which are an ultimate goal of reproducible hydrological modeling. Therefore, the objective of this research is to advance the existing Sciunit capabilities to not only support immutable, but also interoperable computational environments and apply an end-to-end modeling workflow using the Regional Hydro-Ecologic Simulation System (RHESSys) hydrologic model as an example. First, we create an end-to-end workflow for RHESSys using pyRHESSys on the CyberGIS-Jupyter for Water platform. Second, we encapsulate the aforementioned three components and create configurations that include lists of encapsulated dependencies using Sciunit. Third, we create two HydroShare resources, one for immutable reproducibility evaluation using Sciunit and the other for interoperable reproducibility evaluation using library configurations created by Sciunit. Finally, we evaluate the reproducibility of Sciunit in MyBinder, which is a different computational environment, using these two resources. This research presents a detailed example of a user-centric case study demonstrating the application of an open and interoperable containerization approach from a hydrologic modeler’s perspective."

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource was created to execute RHESSys workflows at Coweeta Subbasin18, North Carolina for the evaluation of the original datasets approach which manually collected spatial datasets to represent the traditional approach.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource was created to execute RHESSys workflows at Scotts Level Branch, Maryland for the evaluation of the original datasets approach which manually collected spatial datasets to represent the traditional approach.

Show More
Composite Resource Composite Resource
(HS 16) RHESSys Raw Input at Spout Run, Virginia
Created: May 7, 2021, 11:07 p.m.
Authors: Choi, Young-Don

ABSTRACT:

This HydroShare resource was created to execute RHESSys workflows at Spout Run, Virginia for the evaluation of the original datasets approach which manually collected spatial datasets to represent the traditional approach.

Show More
Composite Resource Composite Resource

ABSTRACT:

For the automated workflows, we create Jupyter notebooks for each state. In these workflows, GIS processing to merge, extract and project GeoTIFF data was the most important process. For this process, we used ArcPy which is a python package to perform geographic data analysis, data conversion, and data management in ArcGIS (Toms, 2015). After creating state-scale LSS datasets in GeoTIFF format, we convert GeoTIFF to NetCDF using xarray and rioxarray Python packages. Xarray is a Python package to work with multi-dimensional arrays and rioxarray is rasterio xarray extension. Rasterio is a Python library to read and write GeoTIFF and other raster formats. We used xarray to manipulate data type and add metadata in NetCDF file and rioxarray to save GeoTIFF to NetCDF format. Through these procedures, we created three composite HyddroShare resources to share state-scale LSS datasets.
Due to the limitation of ArcGIS Pro license which is a commercial GIS software, we developed this Jupyter notebook on Windows OS.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the GeoServer approach at Coweeta Subbasin18, NC

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the original (HydroShare) approach at Coweeta Subbasin18, NC

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the THREDDS approach at Coweeta Subbasin18, NC

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the original (HydroShare) approach at Scotts Level Branch, MD

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the GeoServer approach at Scotts Level Branch, MD

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the THREDDS approach at Scotts Level Branch, MD

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the original (HydroShare) approach at Spout Run, VA

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the THREDDS approach at Spout Run, VA

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks to evaluate data consistency in three different watersheds (different resolutions: 10, 30, and 60 m). In these procedures, we created 9 case studies using three different datasets (Original (HydroShare), GeoServer, and OPeNDAP) and three watersheds (Coweeta subbasin18, Scotts Level Branch, and Spout Run). For the evaluation of the original datasets approach which manually collected spatial datasets to represent the traditional approach, we created three HydroShare resources for each watershed. Then, we present the evaluation results of data consistency in three different watersheds (different resolutions: 10, 30, and 60 m) using Jupyter notebooks (HS-6 resource). For evaluation, we used regression plots for model output (RHESSys streamflow output).

Show More
Collection Resource Collection Resource

ABSTRACT:

Reproducibility is a fundamental requirement to advance science, and data management is the basic element for reproducibility. In hydrological modeling, there have been many efforts to improve the use of spatial data as model input; however, data sharing is file-level, the use of APIs are difficult, and data distribution service is fragile from fast-evolving technologies. Currently large datasets, GeoServer, and OPeNDAP are only used separately, limiting their benefits. The objective of this study is to create and share interoperable and reusable state scale large spatial datasets on GeoServer and OPeNDAP in HydroShare for open and reproducible seamless environmental modelling. We, first, present the procedures for creating and sharing large datasets. Then, we present application workflows with an example of the Regional Hydro-Ecologic Simulation System and evaluate the data consistency of large datasets. We apply three different scales of watershed in three different states to evaluate data consistency in modeling workflows.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021). For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms using the CyberGIS-Jupyter for Water (CJW) environment's supported High-Performance Computing (HPC) resource (XSEDE Comet) through CyberGIS-Compute Service, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021).

For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook utilizes the CJW environment's supported HPC resource (XSEDE Comet) through CyberGIS-Compute Service to executes SUMMA model. This notebook uses the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice. As this resource uses HPC, it enables a high-speed running of simulations which makes it suitable for larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month simulation period used in the paper) practical and much faster than when no HPC is used.

Show More
Composite Resource Composite Resource

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks for RHESSys modeling workflow using the GeoServer approach at Spout Run, VA

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file which is provided in this resource.

Show More
Composite Resource Composite Resource
Run WRFHydro Hands-on Training v5.2.x on CyberGIS-Jupyter for Water
Created: May 25, 2021, 8:30 p.m.
Authors: ·

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRFHydro Hands-on Training v5.2.x (Nov 2020). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook will retrieve the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and put them in a similar directory structure as the Docker-based local setup.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource
Run WRF&WRFHydro Coupled Training v5.1.2 on CyberGIS-Jupyter for Water
Created: May 26, 2021, 9:05 p.m.
Authors: ·

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRF&WRF-Hydro Coupled Testcase Online Lesson (v5.1.2). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, two new folders will be created (wrf-hydro-training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Collection Resource Collection Resource

ABSTRACT:

This is a collection that holds all the WRFHydro official training materials you can run on CyberGIS-Jupyter for Water without installing or downloading any software or data onto your local computer. This collection will expand as new training lessons be added.

Show More
Collection Resource Collection Resource

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Support for NCAR/UCAR WRFHydro Training Notebooks: Users are now able to set up the necessary environment to run WRFHydro hands-on training notebooks (https://ral.ucar.edu/projects/wrf_hydro/training-materials) through simple clicks on CJW. For details on how to run these training notebooks please review the following HydroShare resources: (a) WRFHydro Hands-on Training v5.2.x on CJW; and (b) WRF&WRFHydro Coupled Training v5.1.2 on CJW. This provides an alternative solution to the traditional Docker-based local setup (https://hub.docker.com/r/wrfhydro/training/) that makes it easy for users to complete this training as it does not require installation or downloading of any software or data onto the user’s local computer. Additionally, it enables users to access more powerful computing resources in a cloud-based Jupyter environment. All necessary materials required to complete the training are retrieved from official data sources managed by the NCAR/UCAR WRFHydro Development Team and accessible on CJW via a browser.

2) Transition from XSEDE Comet to Expanse: Comet HPC will be decommissioned on July 31, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW will be transferred to the Expanse HPC (https://portal.xsede.org/sdsc-expanse). As a result, the CyberGIS-Compute service will also drop the support on Comet and replace it with Expanse. Despite this change, we do not expect any action is required for the majority of users. The CyberGIS-Compute service and its SDK will redirect all jobs submitted to Comet to Expanse with a warning message showing up. All previously developed notebooks that use Comet will continue to run after this transition. Please contact us for solutions if you have models or notebooks that access Comet without going through the CyberGIS-Compute service.

3) Extended user testing of Kubernetes-based CJW instance: In the previous release, we announced a Kubernetes-based (Aka K8s: https://kubernetes.io/) CJW instance deployed at https://go.illinois.edu/cjw-k8s for user testing with a preliminary migration plan on deprecation of the current DockerSwarm-based CJW. Due to the growing complexity of K8s and more features being developed and added for a smooth user experience, we have decided to continue conducting extensive testing in this release. We encourage all users to join this testing process and would greatly appreciate your feedback. The current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan is implemented.

Please refer to the following HydroShare resources for details and examples:
Run WRFHydro Hands-on Training v5.2.x on CJW
https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/
Run WRF&WRFHydro Coupled Training v5.1.2 on CJW
https://www.hydroshare.org/resource/c2389a2f05564da08ab218e59bdf1e81/
User testing on Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s
Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/6a1ddb17155a4b27b885f442ad14e344/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

Show More
Web App Resource Web App Resource
CyberGIS-Jupyter for Water (Dev)
Created: Aug. 23, 2021, 6:08 p.m.
Authors: Li, Zhiyu/Drew

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

Show More
Collection Resource Collection Resource

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1)Community Commons for Easy Access to HydroShare Notebooks:
Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/. Interested in learning how CJW enables computational reproducibility of such notebooks? Read on!

2)Enhanced Reproducibility of CJW:
Starting from this release, CJW has implemented a new solution in kernel and software management such that each notebook is now associated with a specific versioned kernel. This approach significantly enhances reproducibility in the CJW environment as each notebook is always opened with the original kernel it is tied to. We encourage users to test their notebooks and reach out to us if you experience any problems.

3)Transferring Large Model Output Datasets:
CJW has introduced a new Globus-based (https://www.globus.org/) data-transfer capability to enhance the performance and stability of transferring a large number of model outputs from high-performance computing (HPC) resources back to CJW, which is a common scenario in running hydrologic models.

4)Seamless Transition from XSEDE Comet to Expanse:
Comet was decommissioned on July 15, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW have been transferred to the Expanse (https://portal.xsede.org/sdsc-expanse). The CyberGIS-Compute service and its SDK now automatically redirect all jobs submitted to Comet to Expanse while prompting users with a simple warning message.

Please refer to the following HydroShare resources for details and examples:
Community Commons for Easy Access to HydroShare Notebooks
https://cybergisxhub.cigi.illinois.edu/hydroshare/
Enhanced Reproducibility of CJW
https://www.hydroshare.org/resource/df8244042e5445edb106d93e5a491d29/
Transferring Large Model Output Datasets
https://www.hydroshare.org/resource/5aa3d27ec5ef4b1b8dfa621ea284af14/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/f3315ec8c1df4f4ab5d2274220de0351/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

Show More
Composite Resource Composite Resource

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource

ABSTRACT:

Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/.

Show More
Collection Resource Collection Resource

ABSTRACT:

This resource provides a quick way to discover and navigate through different example notebooks developed in the CyberGIS-Jupyter for Water (CJW) webapp.
The collection is being updated periodically as new resources will be added and old ones gets deprecated.

How to launch a notebook with CJW:
0) Request to join the CyberGIS-Jupyter for Water Group on HydroShare (one-time effort): Click on the “Ask to join” button in the lower left corner, and a request will normally get approved very quick if your user profile is complete and up-to-date.
1) Click on a resource below to open up the landing page of selected resource;
2) Click on the OpenWith button in the upper-right corner and select "CyberGIS-Jupyter for Water";
3) Follow the instructions in the notebook;

Show More