Checking for non-preferred file/folder path names (may take a long time depending on the number of files/folders) ...
This resource contains some files/folders that have non-preferred characters in their name. Show non-conforming files/folders.
This resource contains content types with files that need to be updated to match with metadata changes. Show content type files that need updating.
Authors: |
|
|
---|---|---|
Owners: |
|
This resource does not have an owner who is an active HydroShare user. Contact CUAHSI (help@cuahsi.org) for information on this resource. |
Type: | Resource | |
Storage: | The size of this resource is 131.9 MB | |
Created: | Mar 01, 2022 at 10:57 p.m. | |
Last updated: | Mar 02, 2022 at 4:58 a.m. | |
DOI: | 10.4211/hs.658c359b8c83494aac0f58145b1b04e6 | |
Citation: | See how to cite this resource | |
Content types: | Multidimensional Content |
Sharing Status: | Published |
---|---|
Views: | 1593 |
Downloads: | 799 |
+1 Votes: | Be the first one to this. |
Comments: | No comments (yet) |
Abstract
The CAMELS datasets are not provided in an ideal format and takes a bit of data processing to convert them to useful and convenient forms for geospatial analyses. So, I decided to use the beloved netcdf and feather formats to make the dataset more accessible while taking care of some small annoyances! Three data sources are available from the CAMELS dataset:
1. Observed Flow: Streamflow observations for all 671 stations.
2. Basin Geometries: Polygons representing basins' boundaries for all 671 stations.
3. Basin Attributes: 60 Basin-level attributes for all 671 stations.
Two files are available:
1. camels_attributes_v2.0.feather: Includes basin geometries and 60 basin-level attributes that are available in CAMELS.
2. camels_attrs_v2_streamflow_v1p2.nc: Includes observed flows for all 671 stations, as well as the 60 basin-level attributes. It has two dimensions (station_id and time) and 60 data variables.
Additionally, some small annoyances in the original dataset are taken care of:
1. Station names didn't have a consistent format and there were some missing commas and extra periods! Now, the names have a consistent format (title) and there is comma before the states.
2. Station IDs and HUC 02 are strings with leading zeros if needed.
The code that was used to generate the dataset can be found at https://github.com/cheginit/camels_netcdf.
Subject Keywords
Coverage
Spatial
Temporal
Start Date: | |
---|---|
End Date: |
Content
README.md
CAMELS dataset in NetCDF/Feather formats
Motivation
The CAMELS datasets
are not provided in an ideal format and takes
a bit of data processing to convert them to useful and convenient
forms for geospatial analyses. So, I decided to use the beloved netcdf
and feather
formats to make the dataset more accessible while taking care of some
small annoyances!
Usage
First make sure that h5netcdf
, geopandas
, xarray
, and requests
Python packages are installed, then load the files directly like so:
```python import io
import geopandas as gpd import requests import xarray as xr
r = requests.get("https://media.githubusercontent.com/media/cheginit/camels_netcdf/main/camels_attributes_v2.0.feather") attrs = gpd.read_feather(io.BytesIO(r.content))
r = requests.get("https://media.githubusercontent.com/media/cheginit/camels_netcdf/main/camels_attrs_v2_streamflow_v1p2.nc") qobs = xr.open_dataset(io.BytesIO(r.content), engine="h5netcdf") ```
Methodology
This repo contains the code that I used to generate the datasets. Three data sources are available from the CAMELS dataset:
- Observed Flow: Streamflow observations for all 671 stations.
- Basin Geometries: Polygons representing basins' boundaries for all 671 stations.
- Basin Attributes: 60 Basin-level attributes for all 671 stations.
The camel_netcdf.py
generates two files:
camels_attributes_v2.0.feather
: Includes basin geometries and 60 basin-level attributes that are available in CAMELS.camels_attrs_v2_streamflow_v1p2.nc
: Includes observed flows for all 671 stations, as well as the 60 basin-level attributes. It has two dimensions (station_id
andtime
) and 60 data variables.
Additionally, the script takes care of some small annoyances:
- Station names didn't have a consistent format and there were some missing
commas and extra periods! Now, the names have a consistent format (
title
) and there is comma before the states. - Station IDs and HUC 02 are strings with leading zeros if needed.
You can recreate the generated files locally using
mambaforge
(or conda
) like so:
bash
git clone https://github.com/cheginit/camels_netcdf
cd camels_netcdf
mamba env create -f environment.yml
conda activate camels
chmod +x ./camels_netcdf.py
./camels_netcdf.py
Example Plots
Snow fraction using camels_attributes_v2.0.feather
:
The dataset camels_attrs_v2_streamflow_v1p2.nc
:
Streamflow observations for USGS-01013500:
Contributions
Contributions are welcome! Please feel free to open an issue/PR if you have any suggestions that can improve the database.
Data Services
How to Cite
This resource is shared under the Creative Commons Attribution CC BY.
http://creativecommons.org/licenses/by/4.0/![CC-BY](https://storage.googleapis.com/hydroshare-prod-static-media/static/img/cc-badges/CC-BY.04d377d96855.png)
Comments
There are currently no comments
New Comment