Zhiyu/Drew Li

University of Illinois at Urbana-Champaign

 Recent Activity

ABSTRACT:

How to Fix the Side Effect caused by New SSL Cert on HydroShare

Revisions:
March 18, 2021; Zhiyu/Drew Li; zhiyul@illinois.edu

Symptoms:
Jupyter Hub fails in OAuth handshaking with HydroShare
“HTTP 599: server certificate verification failed. CAfile: none CRLfile: none”
hs_restclient fails to authenticate
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.hydroshare.org', port=443): Max retries exceeded with url: /hsapi/userInfo/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

Cause:
HydroShare deployed a new SSL cert on March 17, 202. It is based on off a new CA, which is NOT included in the latest “ca-certificates” package (CA Bundle) on Ubuntu 18.04 and 20.04 as of this writing (other Linux distribution may also be affected).

Remedy:
Manually add this new CA into the CA Bundle on all clients that might need to talk to HydroShare.

Download the new CA cert:
Go to HydroShare keybase and download: star_hydroshare_org_124173627DigiCertCA.crt
Go to https://www.digicert.com/kb/digicert-root-certificates.htm, search for “GeoTrust TLS DV RSA Mixed SHA256 2020 CA-1” and download PEM format.

For Hub Dockerfile:

USER root
# get latest ca-bundle
RUN apt-get update && apt-get install -y ca-certificates
# load hydroshare new ca to image
COPY ./star_hydroshare_org_124173627DigiCertCA.crt /usr/local/share/ca-certificates/star_hydroshare_org_124173627DigiCertCA.crt
# update ca-bundle
RUN update-ca-certificates

For different conda envs in Dockerfile:

#Append new HydroShare CA to cacert.pem in Base conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/lib/python<VERSION>/site-packages/certifi/cacert.pem
# Append new HydroShare CA to user-created conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/envs/<ENV_NAME>/lib/python<VERSION>/site-packages/certifi/cacert.pem

References:
https://incognitjoe.github.io/adding-certs-to-requests.html
https://www.techrepublic.com/article/how-to-install-ca-certificates-in-ubuntu-server/

Show More

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Show More

ABSTRACT:

A newly deployed CyberGIS-Jupyter for Water (CJW) instance powered by Kubernetes (k8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology would significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user sign-up. We welcome all users to join the public testing and give us feedback. We anticipate the public testing on the new CJW would take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final “migration plan” is announced (TBD).

Please send comments and bug reports to help@cybergis.org

URL for direct access: https://go.illinois.edu/cjw-k8s

How to setup OpenWith for "Kubernetes-based CJW (user testing)"
1) Login HydroShare
2) Visit this resource landing page: https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
3) In the upper-right corner, click on the 3rd icon from the left (the one with 3x3 small squares), which should prompt "Add WebApp to Open With List"
4) Refresh the landing page of the resource that has notebooks you are interested in, and "Kubernetes-based CJW (user testing)" should show up in the OpenWith list now

Show More

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrologic modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates how to configure an ensemble RHESSys simulation with pyRHESSys, submit it to a supported HPC resource (XSEDE COMET or UIUC Virtual Roger) for execution through CyberGIS Computing Service, visualize model outputs with various tooks integrated in the CyberGIS-Jupyter for Water (CJW).

The model used here is based off of a pre-built RHESSys model for the Coweeta Subbasin 18 (0.124 𝑘𝑚2 ), a subbasins in Coweeta watershed (16 𝑘𝑚2 ), from the Coweeta Long Term Ecological Research (LTER) Program.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies) is a large-sample hydrometeorological dataset that provides catchment attributes, forcings and GIS data for 671 small- to medium-sized basins across the CONUS (continental United States). HydroShare hosts a copy of CAMELS and exposes it through different public data access protocols (WMS, WFS and OPeNDAP) for easy visualization and subsetting of the dataset in community modeling research. This notebook demostrates how to set up SUMMA models with CAMELS dataset from HydroShare using various tools integrated in the CyberGIS-Jupyter for Water (CJW) environment and execution of ensemble model runs on a supported High-Performance Computing (HPC) resource (XSEDE Comet or UIUC Virtual Roger) through CyberGIS-Compute Service.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More

 Contact

Resources
All 0
Collection 0
Composite Resource 0
Generic 0
Geographic Feature 0
Geographic Raster 0
HIS Referenced Time Series 0
Model Instance 0
Model Program 0
MODFLOW Model Instance Resource 0
Multidimensional (NetCDF) 0
Script Resource 0
SWAT Model Instance 0
Time Series 0
Web App 0
Composite Resource Composite Resource
US States
Created: Nov. 3, 2015, 5:43 p.m.
Authors: Drew Li

ABSTRACT:

US States polygon shapefiles

Show More
Composite Resource Composite Resource
NHD Flowlines Colorado Region
Created: Dec. 11, 2015, 8:07 p.m.
Authors: Drew Li

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

Show More
Composite Resource Composite Resource
NHD Flowlines Colorado Region
Created: Aug. 11, 2016, 4:50 p.m.
Authors: Drew Li

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

Show More
Composite Resource Composite Resource
gdal-2.1.2-binary
Created: Nov. 17, 2016, 7:22 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This gdal 2.1.2 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains a sqlite/spatialite geodatabase for assisting the workflow of subsetting NWM Ver1.1 netcdf.
1) grid cell polygon for land and forcing files
2) stream polyline (huc 8, 10, 12)
3) reservoir polygon

Download all the 5 split zip files into one folder and unzip the first one (nwm.zip.001) using 7z

Show More
Composite Resource Composite Resource

ABSTRACT:

Subset National Water Model (NWM) Ver1.1 20170327 results for Bear River-Frontal Great Salt Lake HUC12 watershed (comid: 160102040504).
Python library used to prepare this data: https://pypi.python.org/pypi/subset_nwm_netcdf/

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.3 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.
xy_land_NAD1983.tif is for querying gird cell indices of NWM forcing and land files
xy_terrain_NAD1983.tif is for querying gird cell indices of NWM terrain files

Show More
Composite Resource Composite Resource

ABSTRACT:

A subset of National Water Model (NWM) Ver1.1 20170404 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

Show More
Composite Resource Composite Resource
TwoMileCreek watershed at Tuscaloosa, Alabama
Created: April 12, 2017, 6:03 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

TwoMileCreek watershed at Tuscaloosa, Alabama

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.4-1.1.9 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.

Download and unzip on Ubuntu
sudo apt-get install wget unzip
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.001
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.002
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.003
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.004
cat nwm.zip.* > nwm.zip
rm nwm.zip.*
unzip nwm.zip
rm nwm.zip

Show More
Composite Resource Composite Resource

ABSTRACT:

A Subset of National Water Model (NWM) Ver1.1 20170419 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

Show More
Composite Resource Composite Resource
bash script to download NWM v1.1 outputs
Created: April 29, 2017, 3:59 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

bash script to download NWM v1.1 outputs

Show More
Composite Resource Composite Resource

ABSTRACT:

Hand drawn Utah state border in geojson featurecollection format Projection: WGS84 (EPSG: 4326). This resource was created to test NWM viewer app.

Show More
Composite Resource Composite Resource

ABSTRACT:

Hand drawn Alabama state border in geojson polygon format. This resource was created to test NWM viewer app.

Show More
Composite Resource Composite Resource

ABSTRACT:

NWM v1.1 forcing_analysis_assimilation files of TwoMileCreek watershed region for 20170419

Show More
Collection Resource Collection Resource
HydroShare App Demos
Created: June 7, 2017, 12:56 a.m.
Authors: Christina Bandaragoda

ABSTRACT:

This is a collection of step by step demonstrations on how to use HydroShare Apps.

Show More
Composite Resource Composite Resource
Demo: View multimedia from maps
Created: June 7, 2017, 12:59 a.m.
Authors: Jimmy Phuong · Christina Bandaragoda

ABSTRACT:

This is a step-by-step demonstration of how to Add Images, PDFs, and Videos to digital maps using the HydroShare GIS App using an example from this related HydroShare resource: Ames, D. (2016). Algae Growth in Utah Lake Time-lapse, HydroShare, http://www.hydroshare.org/resource/4c8ecb05a72647339df0df6e9a87718f

Show More
Composite Resource Composite Resource

ABSTRACT:

This is a step-by-step demonstration of how to view and download forecasts from any stream in the National Hydrography Dataset with the National Water Model App.

Show More
Composite Resource Composite Resource
stream_sld
Created: June 20, 2017, 11:48 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

stream_sld

Show More
Composite Resource Composite Resource
Thunder Creek Watershed Boundary
Created: July 25, 2017, 9:07 p.m.
Authors: Christina Bandaragoda

ABSTRACT:

Thunder Creek, Skagit River Basin, State of Washington, USA.

Show More
Web App Resource Web App Resource
National Water Model Forecast Viewer App
Created: Dec. 7, 2017, 3:33 p.m.
Authors: · michael souffront · Zhiyu (Drew) Li · Jim Nelson · Dan Ames

ABSTRACT:

The NWM Viewer app has 2 main features provided in Home Mode and Subset Mode respectively:
Home Mode: Retrieve and View NWM Time Series for a single stream reach, reservoir or grid cell.
Subset Mode: Subset NWM outputs (NetCDF files) with a watershed polygon to get 'shrunken' NetCDFs that only contain data for a specific area.

Show More
Web App Resource Web App Resource
Data Rods Explorer App
Created: Dec. 7, 2017, 3:38 p.m.
Authors: Gonzalo E. Espinoza · David Arctur ·

ABSTRACT:

The Data Rods Explorer (DRE) is a web client app that enables users to browse several NASA-hosted data sets. The interface enables visualization and download of NASA observation retrievals and land surface model (LSM) outputs by variable, space and time. The key variables are precipitation, wind, temperature, surface downward radiation flux, heat flux, humidity, soil moisture, groundwater, runoff, and evapotranspiration. These variables describe the main components of the water cycle over land masses.

Show More
Composite Resource Composite Resource
Rocky River HUC10 0303000305 at Raleigh NC
Created: Dec. 19, 2017, 5:05 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

Rocky River HUC10 0303000305 at Raleigh NC

Show More
Composite Resource Composite Resource
GDAL2.2.3-ubuntu14-x64-binary
Created: Dec. 29, 2017, 5:57 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This gdal 2.2.3 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python

GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal223build
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=gnu++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++11 support: yes

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

Show More
Composite Resource Composite Resource
GDAL2.3-dev-bb4c395-ubuntu16-x64-binary
Created: Dec. 30, 2017, 12:50 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This GDAL2.3-dev-bb4c395 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python
GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal_23_dev_bb4c395_binary
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=c++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++14 support: no

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
crypto/openssl support: yes
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
JPEG-Lossless/CharLS: no
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

Show More
Composite Resource Composite Resource
NWM Viewer App GIS data
Created: Jan. 9, 2018, 6:28 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

This resource contains GIS data for National Water Model Viewer App in ArcGIS geodatabase format:
channels : line--2.7 million NHD+ stream reaches
reservoirs: point--1260 reservoirs locations
usgs_gauge: point--NHD+ USGS gauges locations
grid_land: polygon--grid cell polygons for land and forcing outputs

nwm_app_data.mxd : ArcMap 10 project file

Show More
Composite Resource Composite Resource
LittleWashita HUC10 1113030208 Polygon in OH
Created: Jan. 18, 2018, 12:06 a.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

LittleWashita HUC10 1113030208 Polygon in OH

Show More
Composite Resource Composite Resource

ABSTRACT:

Milburnie Lake - Neuse River HUC10 Watershed 0302020107 at Durham-Raleigh, NC

Show More
Composite Resource Composite Resource
croton_NY_domain_polygon
Created: May 16, 2018, 10:47 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

croton_NY_domain_polygon

Show More
Web App Resource Web App Resource
CyberGIS (beta) JupyterHub
Created: Sept. 26, 2018, 5:53 p.m.
Authors: Dandong Yin

ABSTRACT:

Jupyter environment set in CyberGIS Center for interaction with HPC

Show More
Composite Resource Composite Resource

ABSTRACT:

The US National Water Model (NWM) is a mesoscale hydrologic model that provides streamflow forecasts and other valuable hydrologic information for the continental United States. Since its release at the National Water Center (NWC) in 2016, the NWM has garnered broad attention and great interest across the hydrology science community. Several projects are underway with the goal of delivering this advanced modeling technique and its data to researchers and end users in the community.
As one of the flagship projects, the CUAHSI HydroShare project is working toward providing a complete solution for storing, managing and sharing NWM data. So far, it has set up the largest open-access data archive for the NWM outputs and has developed several different open-source tools and web applications assisting users with data access.
However, there is a growing demand in the hydrologic sciences community for the capability to run a local instance of NWM at regional watersheds to support research applications such as cross-model comparison, historical data analysis and etc.
In this paper, we present ongoing work to develop web applications on top of HydroShare for collecting NWM input data to support model execution at smaller scale watersheds.

Show More
Composite Resource Composite Resource
Composite Resource Composite Resource
czo community sql files
Created: April 9, 2019, 4:23 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

czo community sql files

Show More
Composite Resource Composite Resource
watershed test
Created: May 6, 2019, 7:59 p.m.
Authors: Zhiyu (Drew) Li

ABSTRACT:

watershed test

Show More
Web App Resource Web App Resource
CyberGIS-Jupyter for Water (dev at hs07)
Created: Sept. 4, 2019, 3:30 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

Dev deploy for CyberGIS-Jupyter for Water

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?id=${HS_RES_ID}%2526subfolder=Downloads

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?start=${HS_FILE_PATH}%2526id=${HS_RES_ID}%2526subfolder=Downloads

Show More
Web App Resource Web App Resource
CyberGIS-Jupyter for Water
Created: Oct. 21, 2019, 1:19 p.m.
Authors: Li, Zhiyu (Drew) · Lu, Fangzheng · Padmanabhan, Anand · Wang, Shaowen

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

Show More
Composite Resource Composite Resource

ABSTRACT:

This resources contains 3 notebooks that walk you through running a SUMMA model on CyberGIS-Jupyter for Water platform.

summa_local.ipynb --- run a summa model on Jupyter notebook server (container) directly
summa_hpc.ipynb --- submit summa model as a job to XSEDE COMET High Performance Computing (HPC) cluster
summa_ensemble_hpc.ipynb --- submit an ensemble summa mode as a job to XSEDE COMET High Performance Computing (HPC) cluster

Show More
Composite Resource Composite Resource

ABSTRACT:

User may want to set up a custom Python environment (kernel) and run notebooks with it on the CyberGIS-Jupyter for Water platform. This resource has 2 notebooks to demonstrate walk you through the steps.

install_custom_python_environment.ipynb --- install a custom Python env with user-defined libraries using conda, and set it as a Jupyter Kernel.
recover_custom_python_environment.ipynb --- after a container rebuild, user needs to reactivate a previously installed custom Python env.

Show More
Composite Resource Composite Resource
HAND notebook -- CyberGIS-Jupyter for Water
Created: Dec. 6, 2019, 1:33 a.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

HAND notebook

Show More
Composite Resource Composite Resource
csvtest
Created: March 29, 2020, 4:01 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

asd

Show More
Composite Resource Composite Resource

ABSTRACT:

This example demonstrates how to use PostGIS capabilities in CyberGIS-Jupyter notebook environment. Modified from notebook by Weiye Chen (weiyec2@illinois.edu)

PostGIS is an extension to the PostgreSQL object-relational database system which allows GIS (Geographic Information Systems) objects to be stored in the database. PostGIS includes support for GiST-based R-Tree spatial indices, and functions for analysis and processing of GIS objects.

Resources for PostGIS:

Manual https://postgis.net/docs/
In this demo, we use PostGIS 3.0. Note that significant changes in APIs have been made to PostGIS compared to version 2.x. This demo assumes that you have basic knowledge of SQL.

Show More
Composite Resource Composite Resource

ABSTRACT:

These are examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 1_watershed_delineation folder is an example of a watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

Show More
Composite Resource Composite Resource

ABSTRACT:

!!! This is a fork from https://www.hydroshare.org/resource/5b964154ebf945848087bdc772cc921e/ with some minor modifications for CyberGIS-Jupyer for Water (CJW) platform !!!
----------------------------------------------------------------------------------------------------
The ability to test hypotheses about hydrology, geomorphology, and atmospheric processes is invaluable to research in the Earth and planetary sciences. To swiftly develop experiments using community resources is an extraordinary emerging opportunity to accelerate the rate of scientific advancement. Knowledge infrastructure is an intellectual framework to understand how people are creating, sharing, and distributing knowledge -- which has dramatically changed and is continually transformed by Internet technologies. We are actively designing a knowledge infrastructure system for earth surface investigations. In this paper, we illustrate how this infrastructure can be utilized to lower common barriers to reproducing modeling experiments. These barriers include: developing education and training materials for classroom use, publishing research that can be replicated by reviewers and readers, and advancing collaborative research by re-using earth surface models in new locations or in new applications. We outline six critical elements to this infrastructure, 1) design of workflows for ease of use by new users; 2) a community-supported collaborative web platform that supports publishing and privacy; 3) data storage that may be distributed to different locations; 4) a software environment; 5) a personalized cloud-based high performance computing (HPC) platform; and 6) a standardized modeling framework that is growing with open source contributions. Our methodology uses the following tools to meet the above functional requirements. Landlab is an open-source modeling toolkit for building, coupling, and exploring two-dimensional numerical models. The Consortium of Universities Allied for Hydrologic Science (CUAHSI) supports the development and maintenance of a JupyterHub server that provides the software environment for the system. Data storage and web access are provided by HydroShare, an online collaborative environment for sharing data and models. The knowledge infrastructure system accelerates knowledge development by providing a suite of modular and interoperable process components that can be combined to create an integrated model. Online collaboration functions provide multiple levels of sharing and privacy settings, open source license options, and DOI publishing, and cloud access to high-speed processing. This allows students, domain experts, collaborators, researcher, and sponsors to interactively execute and explore shared data and modeling resources. Our system is designed to support the user experiences on the continuum from fully developed modeling applications to prototyping new science tools. We have provided three computational narratives for readers to interact with hands-on, problem-based research demonstrations - these are publicly available Jupyter Notebooks available on HydroShare.

To interactively compute with these Notebooks, please see the ReadMe below.
To develop these Notebooks, go to Github: https://github.com/ChristinaB/pub_bandaragoda_etal_ems or https://zenodo.org/badge/latestdoi/187289993

Show More
Collection Resource Collection Resource

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2020 Q2)

Dear HydroShare Users,

We are pleased to announce a new quarterly release of CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes new capabilities to support the geoanalytics suite of GRASS for model pre/post-processing, PostGIS database, and Landlab Earth surface modelling toolkit along with several enhancements to job submission middleware, system security as well as service infrastructure. Please refer to the following list for details and examples.

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

GRASS GIS for model pre/post-processing:
Learn how to consolidate the features of the GRASS geoanalytics suite to support pre/post-processing for SUMMA and RHESSYs models in CJW.
Example notebooks: https://www.hydroshare.org/resource/4cbcfdd6e7f943e2969dd52e780bc52d/

Manage geospatial data with PostGIS:
PostGIS is an extension to the PostgreSQL object-relational database system which allows geospatial data to be efficiently stored while providing various advanced functions for in-situ data analysis and processing.
Example notebooks: https://www.hydroshare.org/resource/bb779d4cce564dd6afcf463c8910786f/

Security and service infrastructure enhancements
Trusted group: Starting from this release, all users are required to join the “CyberGIS-Jupyter for Water” trusted group at https://www.hydroshare.org/group/157 in order to access the CJW platform, which is a preventive measure to protect the shared computing resources from being abused by malicious users. A complete user profile page is highly recommended to expedite the approval process.
User metric submission to XSEDE: CJW, as a science gateway, is now sending unique user usage metrics to XSEDE to comply with its requirements.

Landlab for enabling collaborative numerical modeling in Earth sciences using knowledge infrastructure
Example notebooks: https://www.hydroshare.org/resource/370c288b61b84794b847ef85c4dd4ffb/
https://www.hydroshare.org/resource/6add6bee06bb4050bfe23e1081627614/

Job submission enhancements
Refactored the structure of the cyberGIS job submission system
Data-driven implementation for avoiding excessive data transmission between HydroShare and CJW
Add the specification of input parameters into a JSON file to improve the flexibility and generality of model management
Enable HPC-SUMMA object that can directly call SUMMA
Example notebooks: https://www.hydroshare.org/resource/4a4a22a69f92497ead81cc48700ba8f8/

Show More
Composite Resource Composite Resource

ABSTRACT:

This is an example of watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.

Show More
Composite Resource Composite Resource

ABSTRACT:

These is an examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

Show More
Composite Resource Composite Resource
sciunit
Created: May 14, 2020, 4:51 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

sciunit container

Show More
Composite Resource Composite Resource

ABSTRACT:

This a reproducible demonstration of the landslide modeling results from eSurf paper: Strauch et al. (2018)

Show More
Composite Resource Composite Resource

ABSTRACT:

This example is to show the steps to run an ensemble SUMMA3.0 on HPC through the CyberGIS Computing Service.

Show More
Composite Resource Composite Resource
WRFHydro Test Case -- Croton River, NY
Created: Nov. 5, 2020, 4:58 a.m.
Authors:

ABSTRACT:

[You can run this model with the notebook at https://www.hydroshare.org/resource/8fe974c108ca4c6eaaf9b060779329b0/ in CyberGIS-Jupyter for Water platform]

WRFHydro Test Case -- Croton River, NY

#Overview This test case includes prepared geospatial data and input files for a
sample domain (region of interest) and prepared forcing data. This domain is a small region (15km x 16km) encompassing the West Branch
of the Croton River, NY, USA (USGS stream gage 0137462010) during hurricane
Irene, 2011-08-26 to 2011-09-02. The simulation begins with a restart from a
spinup period from 2010-10-01 to 2011-08-26. The forcing data
prepared for this test case is North American Land Data Assimilation System
(NLDAS) hourly data. There are 3 basic routing
configurations included in the test case, National Water Model (NWM), Gridded,
and NCAR Reach. See the WRF-Hydro V5 Technical Description located at
https://ral.ucar.edu/projects/wrf_hydro for a more detailed description of model
physics options, configurations, and input files.

Show More
Composite Resource Composite Resource

ABSTRACT:

The goal of this notebook is to show the steps to run an example National Water Model (WRFhydro) model on HPC resources through the CyberGIS-Compute Service. This notebook uses wrfhydropy, a Python wrapper for WRFHydro, in model preprocessing and postprocessing, and the resulting ready-to-run model is handed over to CyberGIS Computing Service for execution on a supported HPC resource (Virtual Roger/Keeling at UIUC or XSEDE COMET at SDSC). This example is adapted from the "ex_01_end_to_end.ipynb" notebook from wrfhydropy official github repo https://github.com/NCAR/wrf_hydro_py, and users are encouraged to refer to the tutorials there to get familair with wrfhydropy usages.

How to run:
1. Request to join the CyberGIS-Jupyter for Water group at https://www.hydroshare.org/group/157
2. Click the "Open with ..." button in the upper-right
3. Select "CyberGIS-Jupyter for Water"
4. Run through the notebook

Show More
Model Program Resource Model Program Resource
RHESSys East Coast version v7.2
Created: Dec. 19, 2020, 7:16 a.m.
Authors: Choi, Young-Don

ABSTRACT:

RHESSys East Coast version v7.2

Show More
Composite Resource Composite Resource

ABSTRACT:

These are example application notebooks to simulate SUMMA using CAMELS datasets.
There are three steps: (STEP-1) Create SUMMA input, (STEP-2) Execute SUMMA, (STEP-3) Visualize SUMMA output
Based on this example, users can change the HRU ID and simulation periods to analyze 671 basins in CAMELS datasets.

(STEP-1) A_1_camels_make_input.ipynb
- The first notebook creates SUMMA input using Camels dataset using `summa_camels_hydroshare.zip` in this resource and OpenDAP(https://www.hydroshare.org/resource/a28685d2dd584fe5885fc368cb76ff2a/).
(STEP-2) B_1_camels_pysumma_default_prob.ipynb, B_2_camels_pysumma_lhs_prob.ipynb, B_3_camels_pysumma_config_prob.ipynb, and
B_4_camels_pysumma_lhs_config_prob.ipynb
- These four notebooks execute SUMMA considering four different parameters and parameterization combinations
(STEP-3) C_1_camels_analyze_output_default_prob.ipynb, C_2_camels_analyze_output_lhs_prob.ipynb, C_3_camels_analyze_output_config_prob.ipynb,
C_4_camels_analyze_output_lhs_config_prob.ipynb
- The final four notebooks visualize SUMMA output of B-1, B-2, B-3, and B-4 notebooks.

Show More
Composite Resource Composite Resource

ABSTRACT:

The overall goal of this resource is to provide the hydrologic modelers with the datasets and an end-to-end workflow to explore the sensitivity of hydrologic model simulations to variability in the characteristics of meteorological forcings that is further described in the research paper, Van Beusekom et al. (2021). In this paper, hydrological outputs from the SUMMA model for the 671 CAMELS catchments across the contiguous United States (CONUS) are investigated to understand their dependence on input forcing behavior across CONUS. The paper lays out a simple methodology that can be applied to understand the relative importance of seven model forcings (precipitation rate, air temperature, longwave radiation, specific humidity, shortwave radiation, wind speed, and air pressure). This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the paper. For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the goal mentioned above for one example CAMELS site and a period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

Show More
Web App Resource Web App Resource
CJW-k8s-test-js-169-80
Created: Feb. 13, 2021, 5:33 p.m.
Authors: Li, Zhiyu (Drew)

ABSTRACT:

CJW K8s test
http://js-169-80.jetstream-cloud.org/

Show More
Composite Resource Composite Resource
Run RHESSys model with CyberGIS-Compute Service on CJW
Created: March 1, 2021, 8:54 p.m.
Authors: Choi, Young-Don

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrological modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically-based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates parallel job submissions of RHESSys ensemble simulations from CyberGIS-Jupyer for water to HPC (XSEDE), visualizes RHESSys output, and evaluate RHESSys efficiency with simulation runoff and observation streamflow

Show More
Composite Resource Composite Resource

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021). For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

Show More
Composite Resource Composite Resource

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS.

This resource contains basin attributes and parameters in NetCDF files.

Show More
Composite Resource Composite Resource

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies) is a large-sample hydrometeorological dataset that provides catchment attributes, forcings and GIS data for 671 small- to medium-sized basins across the CONUS (continental United States). HydroShare hosts a copy of CAMELS and exposes it through different public data access protocols (WMS, WFS and OPeNDAP) for easy visualization and subsetting of the dataset in community modeling research. This notebook demostrates how to set up SUMMA models with CAMELS dataset from HydroShare using various tools integrated in the CyberGIS-Jupyter for Water (CJW) environment and execution of ensemble model runs on a supported High-Performance Computing (HPC) resource (XSEDE Comet or UIUC Virtual Roger) through CyberGIS-Compute Service.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Composite Resource Composite Resource

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrologic modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates how to configure an ensemble RHESSys simulation with pyRHESSys, submit it to a supported HPC resource (XSEDE COMET or UIUC Virtual Roger) for execution through CyberGIS Computing Service, visualize model outputs with various tooks integrated in the CyberGIS-Jupyter for Water (CJW).

The model used here is based off of a pre-built RHESSys model for the Coweeta Subbasin 18 (0.124 𝑘𝑚2 ), a subbasins in Coweeta watershed (16 𝑘𝑚2 ), from the Coweeta Long Term Ecological Research (LTER) Program.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Show More
Web App Resource Web App Resource

ABSTRACT:

A newly deployed CyberGIS-Jupyter for Water (CJW) instance powered by Kubernetes (k8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology would significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user sign-up. We welcome all users to join the public testing and give us feedback. We anticipate the public testing on the new CJW would take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final “migration plan” is announced (TBD).

Please send comments and bug reports to help@cybergis.org

URL for direct access: https://go.illinois.edu/cjw-k8s

How to setup OpenWith for "Kubernetes-based CJW (user testing)"
1) Login HydroShare
2) Visit this resource landing page: https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
3) In the upper-right corner, click on the 3rd icon from the left (the one with 3x3 small squares), which should prompt "Add WebApp to Open With List"
4) Refresh the landing page of the resource that has notebooks you are interested in, and "Kubernetes-based CJW (user testing)" should show up in the OpenWith list now

Show More
Collection Resource Collection Resource

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Show More
Composite Resource Composite Resource

ABSTRACT:

How to Fix the Side Effect caused by New SSL Cert on HydroShare

Revisions:
March 18, 2021; Zhiyu/Drew Li; zhiyul@illinois.edu

Symptoms:
Jupyter Hub fails in OAuth handshaking with HydroShare
“HTTP 599: server certificate verification failed. CAfile: none CRLfile: none”
hs_restclient fails to authenticate
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.hydroshare.org', port=443): Max retries exceeded with url: /hsapi/userInfo/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

Cause:
HydroShare deployed a new SSL cert on March 17, 202. It is based on off a new CA, which is NOT included in the latest “ca-certificates” package (CA Bundle) on Ubuntu 18.04 and 20.04 as of this writing (other Linux distribution may also be affected).

Remedy:
Manually add this new CA into the CA Bundle on all clients that might need to talk to HydroShare.

Download the new CA cert:
Go to HydroShare keybase and download: star_hydroshare_org_124173627DigiCertCA.crt
Go to https://www.digicert.com/kb/digicert-root-certificates.htm, search for “GeoTrust TLS DV RSA Mixed SHA256 2020 CA-1” and download PEM format.

For Hub Dockerfile:

USER root
# get latest ca-bundle
RUN apt-get update && apt-get install -y ca-certificates
# load hydroshare new ca to image
COPY ./star_hydroshare_org_124173627DigiCertCA.crt /usr/local/share/ca-certificates/star_hydroshare_org_124173627DigiCertCA.crt
# update ca-bundle
RUN update-ca-certificates

For different conda envs in Dockerfile:

#Append new HydroShare CA to cacert.pem in Base conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/lib/python<VERSION>/site-packages/certifi/cacert.pem
# Append new HydroShare CA to user-created conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/envs/<ENV_NAME>/lib/python<VERSION>/site-packages/certifi/cacert.pem

References:
https://incognitjoe.github.io/adding-certs-to-requests.html
https://www.techrepublic.com/article/how-to-install-ca-certificates-in-ubuntu-server/

Show More