Products included in this HydroShare resource:

• Code corresponding to the Data Analytics Layer described within the related manuscript.
• The anonymized quality controlled data collected and used for analyses described in the manuscript.
• Python and R code used to create the figures and tables included in the System Test section of the manuscript.

Folders are organized as follows:

1. DataAnalyticsLayer contains:

   • QC_data.csv: the quality controlled data collected for Case Study 1 described in the manuscript.
   • sites.csv: the characteristics of the site included in the study.
   • Training_Dataset.csv: manually labeled events at the property described in the manuscript, used to train the machine learning algorithm implemented.
   • InfluxDB_Loading.ipynb: A Jupyter Notebook with information required to set up InfluxDB, create an InfluxDB database with characteristics described in the manuscript, and upload QC_data.csv into this database.
   • settings.json: Information required to connect to the InfluxDB database created on the previous step.
   • da_functiuons.py: A simple application programming interface (API) for querying data from the InfluxDB database.
   • CIWS_Disaggregator.py: Code to filter, disaggreagate and classify the raw data into end uses of water.
   • data_analytics.ipynb: A Jupyter Notebook with code to analyze the raw data collected using the tools described on e and f.
     

2. SystemTest contains:

   • data_loader.log: log file form the Data Loading Service (DLS) testing.
   • transfer_manager.log: log file from the Data Transfer Manager (DTM) test between 6 and 96 dataloggers.
   • transfer_manager_480.log: log file from the DTM test with 480 dataloggers.
   • data_poster_nf.log: log file from the Data Posting Service (DPS) test, n represents the number of files for each test, as described in the manuscript.
   • DataPosting_test.R, TransferManager_test.R, and DataLoader_test.R: R files to analyze the test from the DPS, DTM and DLS respectively.

All personally identifiable information was removed from the files published here to protect the identities of the study participants.

The R code provided in this resource was developed using R version 4.0.3. The following R packages are required for running the provided scripts:

• lubridate - Version 1.7.8. Functions for working with dates/times.
• tidyverse - Version 1.3.0. A collection of R packages designed for data science.
• RColorBrewer - 1.1.2 Tool to manage colors with R.

The Python code in this resource was developed using Python 3.7.7. The following Python dependencies are required for running the provided scripts:

• os - Functions for interacting with the operating system.
• influx - Version 5.3.1. A client for interacting with InfluxDB.
• pandas - Version 0.24.2. A fast, powerful, flexible and easy to use open source data analysis and manipulation tool.
• matplotlib - Version 3.3.3. A comprehensive library for creating static, animated, and interactive visualizations in Python.
• seaborn - Version 0.11.1. A Python data visualization library based on matplotlib.
• numpy - Version 0.16.4. A package for scientific computing with Python.
• sklearn - Version 0.21.2 tools for predictive data analysis
• json a lightweight data interchange format inspired by JavaScript object literal syntax (although it is not a strict subset of JavaScript

Instructions for Reproducing Results

To reproduce the results included in the DataAnalyticsLayer folder:

1. Download the entire folder to a machine on which you have created a Python environment that satisfies the above dependencies.
2. Install InfluxDB on a machine using the instructions available at https://docs.influxdata.com/influxdb/v1.8/introduction/install/. We used InfluxDB version 1.8.4 for our testing.
3. Run the InfluxDB_Loading.ipynb Jupyter Notebook to create an InfluxDB database and load the data from QC_data.csv. All the configuration parameters for connecting to Influx are included in the settings.json file. Default parameters (hostname, port, username, password) for a new Influx installation are used, if these values have been previously modified, settings.json needs to be updated before running this script.
4. Run the data_analytics.ipynb Jupyter Notebook to reproduce the results presented in the Data Analytics Layer section for case study 1.

Note: there are more detailed instructions within each Jupyter notebook script.

To reproduce the results included in the SystemTest folder:

1. Download the entire folder. Leave the files together in the folder to ensure the paths to the files remain correct.
2. Execute any R Script using R https://cran.r-project.org/ or R-Studio https://rstudio.com/.