Transferring Large Model Output Datasets using Globus within CyberGIS-Compute


Authors:
Owners:
Resource type: Composite Resource
Storage: The size of this resource is 182.0 KB
Created: Aug 31, 2021 at 7:31 p.m.
Last updated: Sep 07, 2021 at 11:52 p.m.
Citation: See how to cite this resource
Sharing Status: Public
Views: 97
Downloads: 46
+1 Votes: Be the first one to 
 this.
Comments: No comments (yet)

Abstract

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

Subject Keywords

Deleting all keywords will set the resource sharing status to private.

Content

References

This resource belongs to the following collections:
Title Owners Sharing Status My Permission
CyberGIS-Jupyter for Water (CJW) 2021 Q3 Release Notes Zhiyu/Drew Li · Alexander Michels  Public &  Shareable Open Access
Collection of CyberGIS-Jupyter for Water (CJW) Example Notebooks Zhiyu/Drew Li  Public &  Shareable Open Access

How to Cite

Li, Z., Z. Xiao, A. Padmanabhan, S. Wang (2021). Transferring Large Model Output Datasets using Globus within CyberGIS-Compute, HydroShare, http://www.hydroshare.org/resource/5aa3d27ec5ef4b1b8dfa621ea284af14

This resource is shared under the Creative Commons Attribution CC BY.

 http://creativecommons.org/licenses/by/4.0/
CC-BY

Comments

There are currently no comments

New Comment

required