{
"cells": [
{
"cell_type": "markdown",
"id": "statutory-retro",
"metadata": {},
"source": [
"# Setting Up Run-time Environment for WRFHydro Hands-on Training v5.2.x on CyberGIS-Jupyter for Water"
]
},
{
"cell_type": "markdown",
"id": "presidential-cherry",
"metadata": {},
"source": [
"The [HydroShare](https://www.hydroshare.org/) project is pleased to bring you this notebook that can set up a run-time environment on the [CyberGIS-Jupyter for Water](http://go.illinois.edu/cybergis-jupyter-water) (CJW) platform for [WRFHydro Hands-on Training v5.2.x (Nov 2020)](https://ral.ucar.edu/projects/wrf_hydro/training-materials). In contrast to the [Docker-based local setup](https://hub.docker.com/r/wrfhydro/training/), this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser ([Google Chrome](https://www.google.com/chrome/) recommended).\n",
"\n",
"This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the [NCAR/UCAR WRFHydro Development Team](https://ral.ucar.edu/projects/wrf_hydro/team), and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, three new folders will be created (wrf-hydro-training, GIS_Training, and WRF_WPS) alongside. The training notebooks are stored in **wrf-hydro-training --> lessons** as shown below.\n",
"\n",
"![alt text](statics/folder_structure.png)\n",
"\n",
"\n",
"The following table lists all the data (and sources) used by this notebook:\n",
"\n",
"| Repo/File | Link | Commit/Tag/SHA-1|\n",
"| ------------- |:-------------:| -----:|\n",
"| wrf_hydro_nwm_public | https://github.com/NCAR/wrf_hydro_nwm_public | v5.2.0-rc1 |\n",
"| wrf_hydro_training | https://github.com/NCAR/wrf_hydro_training | 57a05ca277a1618185fb5e1646757124d3b3c49e |\n",
"| WrfHydroForcing | https://github.com/NCAR/WrfHydroForcing | f4ddcd760ac78245b9ed1e2a6190b02167b774de |\n",
"| wrf_hydro_model_tools | https://github.com/NCAR/wrf_hydro_model_tools | b70bbf455623aa1ee344a548ea5f2605e661a6f4 |\n",
"| GIS_Training | https://github.com/mcasali/GIS_Training | 97972aa1c50427d04475bfe83eb04a5dbcff23ab |\n",
"| croton_NY_training_example_v5.2.tar.gz | https://drive.google.com/uc?id=1_Ivc02C1WWkZcuBaSl9YePrqQE5osiCp | 9e3b13cb7825caf5aa8ce5e9e5267b5144538e33 |\n",
"| nldas_mfe_forcing.tar.gz | https://drive.google.com/uc?id=10Q-0eVakrVmFwZ27ftDDtsSHsg0YBQAT | 3f7f52fee7876b797ce8dc128fbc0a169480051b |\n",
"| geog_conus.tar.gz | https://drive.google.com/uc?id=1X71fdaSEJ5GWyNY2MDIy9cC6E7A0kihl | 5f7cbbc05a4665667ff32de7fb2393fd79ec5c30 |\n",
"\n",
"**To run this notebook: Cell (top menu) --> Run All**\n",
"\n",
"Note: You are supposed to run this notebook only once. Each run will set up a fresh new environment, and any pre-existing training setup (codes, notebooks and model data) and your edits will be removed.\n",
"\n",
"**After running this notebook successfully, the lesson notebooks will have been downloaded into the workspace and you can start the training at [Lesson 1](wrf-hydro-training/lessons/Lesson-1-compile.ipynb), or view all notebooks [here](wrf-hydro-training/lessons).** **Until this is done these links are inactive.** \n",
"\n",
"**Troubleshooting:**\n",
"If any error message occurred in this notebook, it might be caused by some previously executed training notebooks still running and referencing model data or files on disk . Try one of the two actions below, and run this notebook again.\n",
"\n",
"- Stop all other training notebooks: Go to each notebook --> Kernel (top menu) --> Shutdown; \n",
"- Restart Jupyter server: Control Panel (upper-righer corner) --> Stop My Server --> Start My Server;\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "serious-corporation",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"rm -rf ./wrf-hydro-training 2>/dev/null \n",
"rm -rf ./WRF_WPS 2>/dev/null \n",
"rm -rf ./GIS_Training 2>/dev/null \n",
"rm -rf ~/wrf-hydro-training 2>/dev/null \n",
"rm -rf ~/WRF_WPS 2>/dev/null \n",
"rm -rf ~/GIS_Training 2>/dev/null \n",
"rm -rf /home/docker/wrf-hydro-training 2>/dev/null \n",
"rm -rf /home/docker/WRF_WPS 2>/dev/null \n",
"rm -rf /home/docker/GIS_Training 2>/dev/null "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "mounted-frame",
"metadata": {},
"outputs": [],
"source": [
"! ls $(pwd)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unknown-bennett",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"mkdir -p ./wrf-hydro-training\n",
"mkdir -p ./WRF_WPS\n",
"if [ -d \"/WRF_WPS/WPS\" ] \n",
"then\n",
" ln -sf /WRF_WPS/WPS $(pwd)/WRF_WPS/WPS\n",
"else\n",
" ln -sf $EBROOTWPS/WPS-$EBVERSIONWPS $(pwd)/WRF_WPS/WPS\n",
" echo \"Using WPS from easybuild....\"\n",
"fi\n",
"\n",
"ln -sf $(pwd)/wrf-hydro-training ~/wrf-hydro-training\n",
"ln -sf $(pwd)/GIS_Training ~/GIS_Training\n",
"ln -sf $(pwd)/wrf-hydro-training /home/docker/wrf-hydro-training\n",
"ln -sf $(pwd)/GIS_Training /home/docker/GIS_Training\n",
"ln -sf $(pwd)/WRF_WPS ~/WRF_WPS\n",
"ln -sf $(pwd)/WRF_WPS /home/docker/WRF_WPS"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "addressed-qualification",
"metadata": {},
"outputs": [],
"source": [
"# WRFHYDRO version on https://github.com/NCAR/wrf_hydro_nwm_public\n",
"wrfhydro_version=\"v5.2.0-rc1\"\n",
"# wrf_hydro_training branch or commit on https://github.com/NCAR/wrf_hydro_training\n",
"training_version=\"57a05ca277a1618185fb5e1646757124d3b3c49e\"\n",
"# forcing on https://github.com/NCAR/WrfHydroForcing\n",
"forcing_version = \"f4ddcd760ac78245b9ed1e2a6190b02167b774de\"\n",
"# model tools version on https://github.com/NCAR/wrf_hydro_model_tools\n",
"model_tools_version = \"b70bbf455623aa1ee344a548ea5f2605e661a6f4\"\n",
"# gis training version on https://github.com/mcasali/GIS_Training\n",
"gis_version = \"97972aa1c50427d04475bfe83eb04a5dbcff23ab\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "worse-trauma",
"metadata": {},
"outputs": [],
"source": [
"!wget https://github.com/NCAR/wrf_hydro_nwm_public/archive/{wrfhydro_version}.tar.gz\n",
"!tar -xzvf {wrfhydro_version}.tar.gz\n",
"!rm {wrfhydro_version}.tar.gz\n",
"!mv ./wrf_hydro_nwm_public* ./wrf-hydro-training/wrf_hydro_nwm_public"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "multiple-waters",
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/NCAR/wrf_hydro_training\n",
"!cd ./wrf_hydro_training && git checkout {training_version}\n",
"!mv ./wrf_hydro_training/lessons/training ./wrf-hydro-training/lessons\n",
"!rm -rf ./wrf_hydro_training"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "institutional-stage",
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"!{sys.executable} -m pip install nbformat"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "running-mechanism",
"metadata": {},
"outputs": [],
"source": [
"import nbformat\n",
"import glob\n",
"\n",
"# Set notebooks to the WRFHydro kernel on CJW\n",
"for fp in glob.glob(\"./wrf-hydro-training/lessons/*.ipynb\"):\n",
" nb = nbformat.read(fp, as_version=4)\n",
" if nb.metadata.kernelspec.display_name == \"Python 3\" and nb.metadata.kernelspec.name == \"python3\":\n",
" print(\"changing kernal for {}\".format(fp))\n",
" nb.metadata.kernelspec.display_name = \"WRFHydro-2021-09\"\n",
" nb.metadata.kernelspec.name = \"wrfhydro-2021-09\"\n",
" nbformat.write(nb, fp, version=nbformat.NO_CONVERT)\n",
" elif nb.metadata.kernelspec.display_name == \"Bash\" and nb.metadata.kernelspec.name == \"bash\":\n",
" print(\"changing kernal for {}\".format(fp))\n",
" nb.metadata.kernelspec.display_name = \"Bash-2021-09\"\n",
" nb.metadata.kernelspec.name = \"bash-2021-09\"\n",
" nbformat.write(nb, fp, version=nbformat.NO_CONVERT)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "liable-fossil",
"metadata": {},
"outputs": [],
"source": [
"!pip install gdown"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "comic-finance",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"gdown https://drive.google.com/uc?id=1_Ivc02C1WWkZcuBaSl9YePrqQE5osiCp\n",
"tar -xzvf croton*.tar.gz\n",
"rm croton*.tar.gz\n",
"mv ./example_case ./wrf-hydro-training/example_case"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "vocational-abuse",
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/NCAR/WrfHydroForcing.git\n",
"!cd ./WrfHydroForcing && git checkout {forcing_version}\n",
"!mv ./WrfHydroForcing ./wrf-hydro-training/WrfHydroForcing"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "capable-horse",
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/NCAR/wrf_hydro_model_tools.git\n",
"!cd wrf_hydro_model_tools && git checkout {model_tools_version} \n",
"!mv ./wrf_hydro_model_tools ./wrf-hydro-training/wrf_hydro_model_tools"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "peripheral-swimming",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"gdown https://drive.google.com/uc?id=10Q-0eVakrVmFwZ27ftDDtsSHsg0YBQAT\n",
"mkdir -p ./wrf-hydro-training/regridding\n",
"mv nldas*.tar.gz ./wrf-hydro-training/regridding/nldas_mfe_forcing.tar.gz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "difficult-veteran",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"gdown https://drive.google.com/uc?id=1X71fdaSEJ5GWyNY2MDIy9cC6E7A0kihl \n",
"tar -xzvf geog_conus.tar.gz\n",
"rm geog_conus.tar.gz\n",
"mv ./geog_conus ./WRF_WPS/geog_conus"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "seasonal-guyana",
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/mcasali/GIS_Training\n",
"!cd GIS_Training && git checkout {gis_version}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eastern-taylor",
"metadata": {},
"outputs": [],
"source": [
"print(\"Done\")"
]
},
{
"cell_type": "markdown",
"id": "strategic-carry",
"metadata": {},
"source": [
"# Next - Start from [Lesson 1](wrf-hydro-training/lessons/Lesson-1-compile.ipynb) or view [All Lessons]( wrf-hydro-training/lessons)\n",
"\n",
"**If there was any error occurred in this notebook, you can try to re-run it by: Kernel (top menu) --> Restart & Run All. If the same error persisits, you can try restarting the jupyter environment: Control Panel (upper-right corner) --> Stop My Server --> Start My Server, and then run this notebook again.**\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "WRFHydro-2021-09",
"language": "python",
"name": "wrfhydro-2021-09"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}