# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# 1. System Profile

# ---
# ## Problem 2: Water Quality Comparison (20 points)
#
# ### Background
#
# Country Club Bayou, formerly <NAME>, is located in southeast Houston.
# The bayou drains from east to west connecting to Brays Bayou. The upper portion of the
# bayou is conveyed in a concrete channel that was initially placed in the early 1900’s. The
# lower portion of the bayou from the Hughes Street railroad bridge to the confluence with
# Braes Bayou is open, unlined channel.
#
# Pollution of the bayou has been problematic for at least a dozen years. Currently
# suspected high nutrient loading somewhere in the covered portion of the bayou
# contributes to observed low dissolved oxygen values, a septic `odor`, and septic (black)
# color. The out-fall from the covered portion of the bayou to the open portion is just
# upstream of the Hughes Street Bridge. Samples collected at the bridge by the City of
# Houston Health Department confirm these historical observations.
#
# At times the water at the out-fall just upstream of the Hughes Street Bridge has not meet
# state water quality standards for unclassified waters. Unclassified waters are waters
# which are not specifically listed in Appendices A or D of §307.10 of Title:30, Part 1,
# Chapter 307 of the Texas Administrative Code. 
#
# The table lists some of the relevant standards. 
# <table>
# <tr><td>Parameter</td><td> Value</td><td> Remarks</td></tr>
#     <tr><td>Dissolved Oxygen</td><td> 2.0 mg/L - 24 hr. average        </td><td>  </td></tr>
#     <tr><td>                </td><td> 1.5 mg/L - absolute minimum      </td><td>  </td></tr>
#     <tr><td>                </td><td> 3.0 mg/L - proposed (circa 1998) </td><td>  </td></tr>
#     <tr><td>         Sulfate</td><td>  65 mg/L - proposed (circa 1998) </td><td>  </td></tr>
#     <tr><td>              pH</td><td>  6.5-9.0 - proposed (circa 1998) </td><td>  </td></tr>
#     <tr><td>  Fecal Coliform</td><td>                    200 cfu/100mL </td><td> Contact recreation </td></tr>
#     <tr><td>                </td><td>                   2000 cfu/100mL </td><td> Non-contact recreation </td></tr>
#     <tr><td>     Temperature</td><td> 4oF above ambient</td><td> Fall, Winter, Spring</td></tr>
#     <tr><td>                </td><td> 1.5oF above ambient</td><td> Summer</td></tr>
#     </table>
# These values are proposed for Segment 1014 (Buffalo Bayou above tidal) for contact
# recreation and limited aquatic life use. 
#
# See: [http://www.tnrcc.state.tx.us/water/quality/standards/revisions.html](http://www.tnrcc.state.tx.us/water/quality/standards/revisions.html)
#
# The figure below is a map of the area with storm and sanitary sewer systems drawn onto the map (the study pre-dates affordable GIS)
#
# ![figure 1](CCBayoyMap.png)
#
# The current land-use in the area ranges includes residential, light-industrial, and several
# manufacturing facilities. The covered portion of the storm sewer system is owned
# by the City of Houston, while the open portion appears to be privately owned except for
# the portion through Wortham Park (COH). There are no known discharge permits issued
# by any authority for discharge into Country Club Bayou.
#
# The figure below shows sample locations were water quality samples were collected
#
# ![figure 1](SampleLocations.png)
#
# For this example, the data from the Hughes Street location are examined.  
# The data are stored in the file named HUGHES_STREET.txt and look like:
#
#     MM	DD	YY	pH	T_degC	DO_ppm	NH3_ppm	Odor
#     11	1	1999	6.63	22.5	4.54	0.63	0
#     5	28	1998	7.3	23.2	1.7	0.54	0
#     6	3	1998	7.2	24.1	1.2	0.31	0
#     6	8	1998	7.1	24.8	1	3.96	0
#     6	15	1998	7.2	25	2.9	0.57	0
#     6	26	1998	7.3	27.2	3.1	1.29	0
#     6	29	1998	7.7	26.2	7.9	0.28	0
#     7	8	1998	6.5	26.2	0.28	0.21	0
#     7	20	1998	6.5	28	8.1	0.32	0
#     ...
#     7	15	1999	7.08	27.5	3.19	0.52	0
#     5	12	1998	7	21.6	1.41	0.66	1
#     5	21	1998	7.2	23.9	1.2	1.23	1
#     5	27	1998	6.9	23.6	1.1	0.36	1
#     6	11	1998	7	24.9	1.7	2.5	1
#     ...
#
# ## Problem Statement
#
# Determine if the mean value of **pH**,**T_degC**,**DO_ppm**, and **NH3_ppm** are significantly different when the data are separated using the Odor/No Odor code.

# The data are located at [http://54.243.252.9/engr-1330-webroot/4-Databases/HUGHES_STREET.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/HUGHES_STREET.txt)

# <hr/><hr/> 
#
# **Step 1:** 
#
# <hr/>
#
# Read the "HUGHES_STREET.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe. <br>

# +
# downloadload datafile
# read dataframe and describe contents
# -

# <hr/><hr/> 
#
# **Step 2:** 
#
# <hr/>
#
# Split into two databases, No Odor and Odor <br>

# +
# 
# -

# <hr/><hr/> 
#
# **Step 3:** 
#
# <hr/>
#
# Describe the resulting data series <br>

# +
# No Odor
# DO, Temp, NH3, pH - mean, variance 
# -

# <hr/><hr/> 
#
# **Step 4:** 
#
# <hr/>
#
# Perform appropriate hypothesis tests on the data to determine if the water quality data are different in Odor/No Odor cases.<br>

# +
#
# -

# ## References
# <NAME>., <NAME>., <NAME>., <NAME>., <NAME>. 2000. 
# ``Investigation and Demonstration of Intervention Strategies to Improve Water Quality on Country Club Bayou.'' Final Report to Houston Wastewater Program, Texas Natural Resources Conservation Commission, City of Houston, and the Environmental Institute of Houston 
#
# [https://172.16.17.32/documents/about-me/MyWebPapers/project_reports/ccbayou_report/](https://172.16.17.32/documents/about-me/MyWebPapers/project_reports/ccbayou_report/)

# ---
#
# ## Problem 3: Treaty Terms for the Colorado River (20 points)
#
# ### Background
#
# The Colorado river originates in the Rocky mountains and has contributing tributaries
# and/or flows through the states of Wyoming, Colorado, Utah, Nevada, Arizona,
# and California. 
# Prior to dams the river used to actually run to the ocean and at one
# time in history actually had commerce by river boats. 
# In the late 1800’s and early 1900’s the river and its tributaries were dammed and diverted. 
# Around the 1920’sa legal document called the Colorado River Compact was developed. 
# A Compact is similar to a treaty, except the parties to the treaty are states (essentially the individual states have entered into the treaty with the United States and Mexico). 
# The Colorado River Compact divides the Colorado River into Upper and Lower Basins with the division being at Lee Ferry on the Colorado River one mile below the Paria River in Arizona.
# The Lower Basin states are Arizona, California, and Nevada, with small portions of New Mexico and Utah that are tributary to the Colorado River below Lee Ferry. The Upper Basin states are Colorado, New Mexico, Utah, and Wyoming, with a small portion of Arizona tributary to the Colorado River above Lee Ferry.
#
# Article III of the Compact apportions the waters of the Colorado River to the Upper
# and Lower Basins as follows:
# The Compact apportions the right to exclusive beneficial consumptive use of 7.5
# million acre-feet of water per year from the ”Colorado River System” in perpetuity
# to the Upper Basin and the Lower Basin.
# The Compact allows an additional 1.0 million acre-feet per year of increased beneficial
# consumptive use to the Lower Basin.
# It provides water for Mexico pursuant to treaty. Water must first come from any
# surplus over the waters allocated to the states in Article III(a) and (b). If that
# surplus is insufficient, then the burden of that deficiency shall be shared equally by
# the Upper and Lower Basins.
# The Compact provides that the Upper Basin states will not cause the flow of the
# river at Lee Ferry, Arizona to be depleted below an aggregate of 75 million acrefeet
# for any period of ten consecutive years beginning with the ratification of the
# Compact.
# It provides that the Upper Basin states will not withhold water and the states of the
# Lower Basin shall not require delivery of water which cannot reasonably be applied
# to domestic and agricultural uses.

# ## Problem Statement
#
# The file named [colorado.txt](http://192.168.127.12/engr-1330-webroot/4-Databases/colorado.txt) contains annual natural flows volumes in the upper
# Colorado river basin from 1906 to present. These flows are historical reconstruction
# at approximately the same location as the current USGS gage 09380000. The
# location is approximately above (upstream of) Lee Ferry, hence the data represent
# the flows out of the upper basin.
# The data are in two columns, the first is the water year, the second column is
# cumulative discharge for the year in units of acre-feet.
#
#     YEAR ANNUAL.VOLUME
#     1906 18550021
#     1907 21201694
#     1908 12218817
#     1909 22356301
#     1910 14650616
#     ...
# Prepare an analysis of the data to support or refute the provisions of the Compact.
# You can assume that the data from 1906 to 1927 (the original Compact was ratified
# in 1922, many additional Compacts have been signed since that time, all trying to reapportion
# water in the river basin witout violating the original agreement) were available to the
# authors of the Colorado River Compact, analyze this period of data and decide if the
# allocations to the lower basin are realistic (8.5 million acre-feet-per year) assuming
# the upper basin allocation is removed. Consumptive use means that the water will
# not return to the basin. The data are ’natural’ flows and the upper basin allocations
# are not removed in the data. 

# <hr/><hr/> 
#
# **Step 1:** 
#
# <hr/>
#
# Read the "colorado.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe. <br>

# +
# download the data file
# read into a dataframe
# plot
# describe the dataframe
# -

# <hr/><hr/> 
#
# **Step 2:** 
#
# <hr/>
#
# Evaluate if the mean value in the pre-compact period is substantially different from the post-compact period. <br>

# +
# split the data into pre-compact and post-compact
# describe the two subsets
# box-plot both
# test for normal
# hypothesis test for difference in means
# conclusion(s) 
# -

# <hr/><hr/> 
#
# **Step 3:** 
#
# <hr/>
#
# Estimate the probability based on the the 20-year period that the lower basin allocation will not be met if upper basin allocation is satisfied. Estimate the probability that the allocation will not be met using the entire data set if the upper basin allocation is satisfied.<br>

# +
# empirical probability approach on the 20 years pre-compact; full data
# -

# <hr/><hr/> 
#
# **Step 4:** 
#
# <hr/>
#
# Next assume that lower basin is satisfied first (this is the historical result of the
# original and subsequent treaties). Estimate the probability that the upper basin
# allocation will not be met. Determine an upper basin allocation that will be met
# 95% of the time. Again use the 20 year initial record, then the entire record.

# +
# empirical probability approach on the 20 years pre-compact; full data
# -

# <hr/><hr/> 
#
# **Step 5:** 
#
# <hr/>
#
# Was the initial 20 year period sufficiently different from the later data that the
# Compact authors made decisions based on unusually high or low flows?

# +
# make a decision and report
# -

# ## Problem 4: Solids in Rivers

# + [markdown] jupyter={"source_hidden": true}
# ## Problem 5: Watershed Response Metrics (20 points)
#
# ### Background 
# Rainfall-Runoff response prediction is a vital step in engineering design for mitigating flood-induced infrastructure failure.   One easy to measure characteristic of a watershed is its drainage area.  Harder to quantify are its characteristic response time, and its conversion (of precipitation into runoff) factor.
#
# ### Study Database
#
# The [watersheds.csv](http://54.243.252.9/engr-1330-webroot/4-Databases/watersheds.csv) dataset contains (measured)drainage area for 92 study watersheds in Texas (cite source), and the accompanying data:
#
# |Columns|Info.|
# |:---|:---|
# |STATION_ID |USGS HUC-8 Station ID code|
# |TDA |Total drainage area (sq. miles) |
# |RCOEF|Runoff Ratio (Runoff Depth/Precipitation Depth)|
# |TPEAK|Characteristic Time (minutes)|
# |FPEAK|Peaking factor (same as NRCS factor)|
# |QP_OBS|Observed peak discharge (measured)|
# |QP_MOD|Modeled peak discharge (modeled)| 
#
# ### Assess the data:
#
# Using the following steps, assess the utility of the response metric.
#
# -

# <hr/><hr/> 
#
# **Step 1:** 
#
# <hr/>
#
# Read the "watersheds.csv" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe. <br>

# +
# import packages
# read data file
# summarize contents + markdown cell as needed
# -

# <hr/><hr/> 
#
# **Step 2:** <hr/>
#
# Using your laboratory scripts as a guide, assess the utility of **TDA** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **TDA** on the x-axis) <br>

# +
# extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# # + markdown cell as needed
# -

# <hr/><hr/> 
#
# **Step 3:**
#
# <hr/>
#
# Using your laboratory scripts as a guide, assess the utility of **log(TDA)** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **log(TDA)** on the x-axis)
#
# Which mapping of **TDA**  (arithmetic or logarithmic) produces a more useful graph? 

# +
# extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# # + markdown cell as needed
# -

# <hr/><hr/> 
#
# **Step 4:**
#
# <hr/>
#
# Using your laboratory scripts as a guide, assess the performance of the underlying (not disclosed) model, using **QP_MOD** as a predictor of **QP_OBS** (e.g. perform a regression and produce plots, with **QP_MOD** on the x-axis).
#
# Include an equal value line (x=y) on the plot and make an assertion of model bias (does the model overpredict or underpredict the observations?)
# <br>

# +
# extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# # + markdown cell as needed
# -

#

#

# <div class="alert alert-block alert-danger">
#     <b><h1>Problem 6: Currency Authentication (20 points) </h1></b> 
# </div>
#
# ### Background 
# The U.S. Secret Santas examined 1372 images of banknotes, and classified them as genuine or forgery. Data were extracted from images using an industrial camera for print inspection; The final images have 400x 400 pixels. Due to the object lens and distance to the investigated object gray-scale pictures with a resolution of about 660 dpi were gained. A Wavelet Transform model was used to extract features from images.  The resulting data model has 4 predictor variables (variance of image, skewness, kurtosis, entropy). The classification variable is encoded as 0 (authentic) or 1 (forgery). 
#
# ### Study Database
#
# The [data_banknote_authentication.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/data_banknote_authentication.txt) dataset contains the data as 5 columns.
#
#
# ### Problem Statement
#
# Build a classification engine (logistic regression) to classify a banknote image into authentic or forgery using a 70/30 training/testing split.
#
# Three banknotes are presented for authentication.  Determine which banknote images with with the following parameters are authentic or counterfeit
#
# Banknote 1:
#
# |Quality|Value|
# |---|---|
# |Variance|2.25040|
# |Skewness|3.57570|
# |Kurtosis|0.35273|
# |Entropy|0.28360|
#
#
# Banknote 2: 
#
# |Quality|Value|
# |---|---|
# |Variance|-1.3971|
# |Skewness| 3.3191|
# |Kurtosis|-1.3927|
# |Entropy|-1.9948|
#
#
# Banknote 3: 
#
# |Quality|Value|
# |---|---|
# |Variance|0.93574|
# |Skewness|8.8865|
# |Kurtosis|-1.68301|
# |Entropy|-1.65999|
#
#

# <hr/><hr/> 
#
# **Step 1:** 
#
# <hr/>
#
# Read the "data_banknote_authentication.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe. <br>

# +
# import packages
# read data file
# summarize contents + markdown cell as needed
# -

# <hr/><hr/> 
#
# **Step 2:** <hr/>
#
# make a pairplot to visually examine if classification is feasible and identify outliers <br>

# +
# sns.pairplot ...
# comments
# -

# <hr/><hr/> 
#
# **Step 3:**
#
# <hr/>
#
# Build training and testing sets (use sklearn as guidance)

# +
# inp_df = ...
# out_df = ...
# scaler = ...
# inp_df = ...
# X_train, X_test, y_train, y_test = ...
# -

# <hr/><hr/> 
#
# **Step 4:**
#
# <hr/>
#
# Fit the training set; examine accuracy from testing set<br>

# +
# from sklearn.linear_model import LogisticRegression
# clf = ...
# clf.fit(...)
# ...
# -

# <hr/><hr/> 
#
# **Step 5:**
#
# <hr/>
#
# Build the classification engine, determine classification of value(s) in problem statement<br>

# +
# 
# banknote 1 fake or real?
# banknote 2 fake or real?
# banknote 3 fake or real?
# -

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# %matplotlib inline
import seaborn as sns
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

df = pd.read_csv('data_banknote_authentication.txt')

df.describe()
#df.columns =['Variance', 'Skewness', 'Kurtosis', 'Entropy','Class']

df.head()

sns.pairplot(df, hue='Class', height=2.5)

inp_df = df.drop(df.columns[[4]], axis=1)
out_df = df.drop(df.columns[[0,1,2,3]], axis=1)
#
scaler = StandardScaler()
inp_df = scaler.fit_transform(inp_df)
#
X_train, X_test, y_train, y_test = train_test_split(inp_df, out_df, test_size=0.2, random_state=42)

X_tr_arr = X_train
X_ts_arr = X_test
y_tr_arr = y_train.to_numpy()  
y_ts_arr = y_test.to_numpy()

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_tr_arr, y_tr_arr)
print (clf.intercept_, clf.coef_)
pred = clf.predict(X_ts_arr)
print ('Accuracy from sk-learn: {0}'.format(clf.score(X_ts_arr, y_ts_arr)))


def mymodel(x1,x2,x3,x4,w1,w2,w3,w4,b):
    z = w1*x1 + w2*x2 +w3*x3 +w4*x4 +b #linear model to produce estimator to send to logistic fn
    yhat = sigmoid_activation(z)
    return(yhat)



df_fake = df[df["Class"]==1]
df_fake.head(30)

df_real = df[df["Class"]==0]
df_real.head(30)

sepal_l = 0.93584
sepal_w =  8.8855
petal_l = -1.68310
petal_w = -1.65990
myguess = mymodel(sepal_l,sepal_w,petal_l,petal_w,clf.coef_[0,0],clf.coef_[0,1],clf.coef_[0,2],clf.coef_[0,3],clf.intercept_[0])
print(myguess)

0.93584 	8.8855 	-1.68310 	-1.65990

StarcoderdataJupyterScriptsDedupFiltered