CAMELS dataset in NetCDF/Feather formats


Authors:
Owners: This resource does not have an owner who is an active HydroShare user. Contact CUAHSI (help@cuahsi.org) for information on this resource.
Resource type: Composite Resource
Storage: The size of this resource is 131.9 MB
Created: Mar 01, 2022 at 10:57 p.m.
Last updated: Mar 02, 2022 at 4:58 a.m.
DOI: 10.4211/hs.658c359b8c83494aac0f58145b1b04e6
Citation: See how to cite this resource
Content types: Multidimensional Content 
Sharing Status: Published
Views: 174
Downloads: 24
+1 Votes: Be the first one to 
 this.
Comments: No comments (yet)

Abstract

The CAMELS datasets are not provided in an ideal format and takes a bit of data processing to convert them to useful and convenient forms for geospatial analyses. So, I decided to use the beloved netcdf and feather formats to make the dataset more accessible while taking care of some small annoyances! Three data sources are available from the CAMELS dataset:
1. Observed Flow: Streamflow observations for all 671 stations.
2. Basin Geometries: Polygons representing basins' boundaries for all 671 stations.
3. Basin Attributes: 60 Basin-level attributes for all 671 stations.

Two files are available:
1. camels_attributes_v2.0.feather: Includes basin geometries and 60 basin-level attributes that are available in CAMELS.
2. camels_attrs_v2_streamflow_v1p2.nc: Includes observed flows for all 671 stations, as well as the 60 basin-level attributes. It has two dimensions (station_id and time) and 60 data variables.

Additionally, some small annoyances in the original dataset are taken care of:
1. Station names didn't have a consistent format and there were some missing commas and extra periods! Now, the names have a consistent format (title) and there is comma before the states.
2. Station IDs and HUC 02 are strings with leading zeros if needed.

The code that was used to generate the dataset can be found at https://github.com/cheginit/camels_netcdf.

Subject Keywords

Resource Level Coverage

Spatial

Coordinate System/Geographic Projection:
WGS 84 EPSG:4326
Coordinate Units:
Decimal degrees
Place/Area Name:
CONUS
North Latitude
49.2603°
East Longitude
-66.6002°
South Latitude
26.5782°
West Longitude
-124.6081°

Temporal

Start Date:
End Date:

Content

README.md

CAMELS dataset in NetCDF/Feather formats

Motivation

The CAMELS datasets are not provided in an ideal format and takes a bit of data processing to convert them to useful and convenient forms for geospatial analyses. So, I decided to use the beloved netcdf and feather formats to make the dataset more accessible while taking care of some small annoyances!

Usage

First make sure that h5netcdf, geopandas, xarray, and requests Python packages are installed, then load the files directly like so:

```python import io

import geopandas as gpd import requests import xarray as xr

r = requests.get("https://media.githubusercontent.com/media/cheginit/camels_netcdf/main/camels_attributes_v2.0.feather") attrs = gpd.read_feather(io.BytesIO(r.content))

r = requests.get("https://media.githubusercontent.com/media/cheginit/camels_netcdf/main/camels_attrs_v2_streamflow_v1p2.nc") qobs = xr.open_dataset(io.BytesIO(r.content), engine="h5netcdf") ```

Methodology

This repo contains the code that I used to generate the datasets. Three data sources are available from the CAMELS dataset:

The camel_netcdf.py generates two files:

  • camels_attributes_v2.0.feather: Includes basin geometries and 60 basin-level attributes that are available in CAMELS.
  • camels_attrs_v2_streamflow_v1p2.nc: Includes observed flows for all 671 stations, as well as the 60 basin-level attributes. It has two dimensions (station_id and time) and 60 data variables.

Additionally, the script takes care of some small annoyances:

  • Station names didn't have a consistent format and there were some missing commas and extra periods! Now, the names have a consistent format (title) and there is comma before the states.
  • Station IDs and HUC 02 are strings with leading zeros if needed.

You can recreate the generated files locally using mambaforge (or conda) like so:

bash git clone https://github.com/cheginit/camels_netcdf cd camels_netcdf mamba env create -f environment.yml conda activate camels chmod +x ./camels_netcdf.py ./camels_netcdf.py

Example Plots

Snow fraction using camels_attributes_v2.0.feather: camels_snow_fraction

The dataset camels_attrs_v2_streamflow_v1p2.nc: dataset

Streamflow observations for USGS-01013500: qobs_01013500

Contributions

Contributions are welcome! Please feel free to open an issue/PR if you have any suggestions that can improve the database.

Data Services

The following web services are available for data contained in this resource. Geospatial Feature and Raster data are made available via Open Geospatial Consortium Web Services. The provided links can be copied and pasted into GIS software to access these data. Multidimensional NetCDF data are made available via a THREDDS Data Server using remote data access protocols such as OPeNDAP. Other data services may be made available in the future to support additional data types.

How to Cite

Chegini, T. (2022). CAMELS dataset in NetCDF/Feather formats, HydroShare, https://doi.org/10.4211/hs.658c359b8c83494aac0f58145b1b04e6

This resource is shared under the Creative Commons Attribution CC BY.

http://creativecommons.org/licenses/by/4.0/
CC-BY

Comments

There are currently no comments

New Comment

required