Checking for non-preferred file/folder path names (may take a long time depending on the number of files/folders) ...

New Zealand Hydrological Society Data Workshop 2024: A Python Package for Automating Aquatic Data QA/QC


Authors:
Owners: This resource does not have an owner who is an active HydroShare user. Contact CUAHSI (help@cuahsi.org) for information on this resource.
Type: Resource
Storage: The size of this resource is 159.6 MB
Created: Apr 08, 2024 at 10:59 p.m.
Last updated: Apr 09, 2024 at 2:20 a.m.
Citation: See how to cite this resource
Sharing Status: Public
Views: 327
Downloads: 109
+1 Votes: Be the first one to 
 this.
Comments: No comments (yet)

Abstract

This resource was created for the 2024 New Zealand Hydrological Society Data Workshop in Queenstown, NZ. This resource contains Jupyter Notebooks with examples for conducting quality control post processing for in situ aquatic sensor data. The code uses the Python pyhydroqc package to detect anomalies. This resource consists of 3 example notebooks and associated data files. For more information, see the original resource from which this was derived: http://www.hydroshare.org/resource/451c4f9697654b1682d87ee619cd7924.

Notebooks:
1. Example 1: Import and plot data
2. Example 2: Perform rules-based quality control
3. Example 3: Perform model-based quality control (ARIMA)
4. Example 4: Model-based quality control (ARIMA) with user data

Data files:
Data files are available for 6 aquatic sites in the Logan River Observatory. Each file contains data for one site for a single year. Each file corresponds to a single year of data. The files are named according to monitoring site (FranklinBasin, TonyGrove, WaterLab, MainStreet, Mendon, BlackSmithFork) and year. The files were sourced by querying the Logan River Observatory relational database, and equivalent data could be obtained from the LRO website or on HydroShare. Additional information on sites, variables, and methods can be found on the LRO website (http://lrodata.usu.edu/tsa/) or HydroShare (https://www.hydroshare.org/search/?q=logan%20river%20observatory). Each file has the same structure indexed with a datetime column (mountain standard time) with three columns corresponding to each variable. Variable abbreviations and units are:
- temp: water temperature, degrees C
- cond: specific conductance, μS/cm
- ph: pH, standard units
- do: dissolved oxygen, mg/L
- turb: turbidity, NTU
- stage: stage height, cm

For each variable, there are 3 columns:
- Raw data value measured by the sensor (column header is the variable abbreviation).
- Technician quality controlled (corrected) value (column header is the variable abbreviation appended with '_cor').
- Technician labels/qualifiers (column header is the variable abbreviation appended with '_qual').

There is also a file "data.csv" for use with Example 4. If any user wants to bring their own data file, they should structure it similarly to this file with a single column of datetime values and a single column of numeric observations labeled "raw".

Subject Keywords

Content

Related Resources

The content of this resource is derived from Jones, A. S. (2022). Hydroinformatics Instruction Module Example Code: Sensor Data Quality Control with pyhydroqc, HydroShare, http://www.hydroshare.org/resource/451c4f9697654b1682d87ee619cd7924, accessed on: 04/08/2024

Credits

Funding Agencies

This resource was created using funding from the following sources:
Agency Name Award Title Award Number
National Science Foundation Collaborative Research: Elements: Advancing Data Science and Analytics for Water (DSAW) 1931297

Contributors

People or Organizations that contributed technically, materially, financially, or provided general support for the creation of the resource's content but are not considered authors.

Name Organization Address Phone Author Identifiers
Jeffery S. Horsburgh Utah State University Utah, US 4357972946 ORCID , ResearchGateID , GoogleScholarID
Camilo J. Bastidas Pacheco Utah State University UT, US 4357545722

How to Cite

Jones, A. S. (2024). New Zealand Hydrological Society Data Workshop 2024: A Python Package for Automating Aquatic Data QA/QC, HydroShare, http://www.hydroshare.org/resource/5e942e193e494f3fab89dc317d8084fa

This resource is shared under the Creative Commons Attribution CC BY.

http://creativecommons.org/licenses/by/4.0/
CC-BY

Comments

There are currently no comments

New Comment

required