README file describing data and code in repository for:
	Manuscript: 'Bootstrap aggregation and cross-validation methods to reduce overfitting in reservoir control policy search'
	Authors: Brodeur, Z, Herman, J.D., and Steinschneider, S.

Current as of 3 May 2020

Main directory 'bagging_cross-val_policy-search' files:
'ensemble_optimize.py' Python code to optimize 30x policy trees for both historical and bootstrapped cases
'validate_pickbest.py' Python code to analyze scaled policy scores based on validation performance
'calibrate_pickbest.py' Python code to analyze scaled policy scores based on calibration performance
'simulate_totcost_metrics.py' Python code to produce summed/scaled total cost metrics
'simulate_metrics.py' Python code to produce separate water supply and flood overage metrics for primary figures
'single_runs.py' Python code to do individual runs of the Folsom model
'results_plot.r' R code to plot primary results and calculate significance values
'results_tot_plot.r' R code to plot addition summed/scaled cost results
'folsom_model_optimize.py' Folsom reservoir model configured for optimizing policies via 'ensemble_optimize.py'
'folsom_model_ensemble.py' Folsom reservoir model configured for output analysis for both ensemble mode and single policies

'data' repository
	'generate_paleo-bootstrap_training.R' R code to generate training period (1982-2016) paleo-bootstrap resampled datasets
	'generate_paleo-bootstrap_test.R' R code to generate test period (1923-1981) paleo-bootstrap resampled datasets
	'generate_paleo-bootstrap_test_review.R' same as 'generate..test.R' but with random resampling of paleo flows
	'folsom-daily-w2016.csv' Inflow, Outflow, Storage, Evap timeseries for Folsom Reservoir, 1922-2016
	'demand.txt' Average daily demand by day of water year for Folsom reservoir
	'Sacramento_paleo_inflow.csv' Paleo annual inflow data for American River at Folsom Reservoir, 900-2012 CE
	'plot-static-trees.py' Script to create graphical depictions of policy trees
	Sub-directory('resamp')
		'inflow_forecast_0.csv,... ""_29.csv' Paleo-bootstrapped inflow sequences based on training period, WY 1982-2016
	Sub-directory('resamp_test')
		'inflow_forecast_0.csv,... ""_29.csv' Paleo-bootstrapped inflow sequences based on test period, WY 1923-1981
	Sub-directory('resamp_test1','resamp_test1','resamp_test1')
		'inflow_forecast_0.csv,... ""_29.csv' same as resamp test, but with resampled paleo annual flows in 1, 2, and 3 year blocks
	Sub-directory('figs')
		Output directory for 'plot-static-trees' script
		

'output' repository
	'results_2381.csv' Results csv file for primary figure plots via 'results_plot.r' for testing period
	'results_8216.csv' "" for training period
	'results_2381_rs1,rs2,rs3.csv' same as 'results_2381.csv' but with resampled paleo annual flows in 1, 2, and 3 year blocks
	'results_tcost_2381.csv' Results csv file for summed/scaled cost boxplots via 'results_tot_plot.r' for testing period
	'results_tcost_8216.csv' "" for training period
	Sub-directory('p1_ens')
		'snapshots-forecast-p1-gw0.0TAF-seed-0.pkl,...""-29.pkl' Policy trees 0-29 all trained on historical 1982-2016 data
	Sub-directory('p1_ens_resamp')
		'snapshots-forecast-p1-gw0.0TAF-seed-0.pkl,...""-29.pkl' Policy trees 0-29 each trained on bootstrapped 1982-2016 data

'pdf' repository
	Plots of paleo-bootstrap resampling diagnostics as described in R resampling code

'ptreeopt' repository
	Python modules to run tree-based evolutionary optimization algorithm

'graphvis'
	Python package for plotting graphical models (tree structures)

Steps to recreate results:
1) Generate bootstrap samples via 'generate_paleo..R' scripts
2) Optimize policies to both historical and bootstrapped training data via 'ensemble_optimize.py' script
3) Find best policies via calibration and validation procedures via 'calibrate_pickbest.py' and 'validate_pickbest.py' scripts
4) Run 'simulate_metrics.py' with manual selections of best policies from step 3
5) Plot results via 'results_plot.r' script