<jupyter_start><jupyter_text>The Anscombe's Quartet Dataset  Submitted by: <NAME> G00364748 ------------------------------------------------------------------------------------------------------  Assignment: Fundamentals of Data Analysis  Submission Date: 9th November 2018 ------------------------------------------------------------------------------------------------------  Table of Contents Introduction Statement of Work Technology Used Aim of Investigation Findings References ------------------------------------------------------------------------------------------------------  Introduction This repository has been created as apart of my ongoing studies in Data Analytics.My intention with this repository is to investigate the Anscombe's Quartet Dataset using the available functions within Jupyter. This dataset was constructed to highlight the importance of using graphs in statistical analysis ------------------------------------------------------------------------------------------------------  Statement of Work This is a statement to confirm that the work submitted within this repository is my own work. Any information sourced online or 3rd party has been referenced appropriately. I can therefore confirm that my submitted work is in line with the Quality Assurance Framework of GMIT. ------------------------------------------------------------------------------------------------------  Technology Used * Anaconda software package* Python Programming Language* Command Prompt* Jupyter Notebook* GitHub.com* LearnOnline (GMIT) ------------------------------------------------------------------------------------------------------  Aim of Investigation It is my intention to:1. Explain the background to the dataset1. Plot any interesting aspects of the dataset1. Calculate the descriptive statistics of the variables in the dataset1. Explain why the dataset is interesting based on the previously obtained info ------------------------------------------------------------------------------------------------------  Findings  1. The background to the datasetIn 1973, a statistician called <NAME> constructed the dataset quartet. His philosophy of the dataset was to "demonstrate both the importance of graphing data before analyzing it and the effect of outliers on statistical properties."[1] His development questioned the opinion another staticians that "numerical calculations are exact, but graphs are rough."![anscombe](dataset.gif)The breakdown of the information contained within the datasets: There are 8 variables; representing four pairings of an outcome and a predictor.All sets have 11 observations:the same mean of x (9) and y (7.5),the same fitted regression line (y = 3 + 0.5 x),the same regression and residual sum of squares and therefore the same multiple R-squared of 0.67. But they represent very different situations, as you will see by clicking on each dataset[2]This dataset has proven to me that statistics do not always tell the entire story when it comes to results given by data. And visualising the data allows an unbiased insight into the analysing of the data.This was deemed an issue by Anscombe and for this reason that he went on to create the quartet of datasets to visually show this flaw. [3]A 'real-life' example to highlight Anscombe's work is the following study carried out in 2012.The study involved the comparison of starting wages for law graduates based on "The National Association of Law Placement (NALP)" report from 2012.In summary, the statistical data showed the starting average wage was 80,798. When the data was graphed however, it showed two peaks in the data between 35,000USD - 75000USD at again at 160000USD."A much more accurate statement would be that most law graduates make around 50,000USD on average, and those who go to one of the top law schools make 160,000USD on average."[4]In 2017, a man by the name of <NAME>, inspired by Anscombe's work, tweeted an image showing how his data looked when graphed. It turned out to be in the shape of a dinosaur.While sharing his work, he urges people to *"never trust summary statistics alone; always visualize your data"*[5]Cairo caused a ripple effect from his work as at the end of 2017 - start of 2018 another publication involving the Datasaurus was created but this time it included an additional 12 plots with similar statistics but very different graphs when plotted.*"These 13 datasets (the Datasaurus, plus 12 others) each have the same summary statistics (x/y mean, x/y standard deviation, and Pearson's correlation) to two decimal places, while being drastically different in appearance."*[6][6] 2. Plot any interesting aspects of the dataset  The below graph shows the entire data of the quartetBased on the summary properties, there is not huge difference in them, showing the close relationship when it comes to the black and white data.Should you solely rely on the below output rather than visual graphs, it would be easy to assume these were more closely related than they actually are.Once viewing the graphs it is easy to see just how different the outputs are based on the shapes they produce on graphs.<jupyter_code># http://nbviewer.jupyter.org/github/psychemedia/ou-tm351/blob/master/notebooks-RFC/Anscombe's%20Quartet%20%5Bopen%5D.ipynb [7]
import pandas as pd
aq=pd.read_csv('anscombe.csv',header=[0,1],index_col=[0])
aq
#Created 3 side-by-side plots to view the min, max & mean outputs visually 
plt.subplot(1,3,1)
l = aq.min()
plt.hist(l)
plt.title("Min")
plt.xlabel("x axis")
plt.ylabel("y axis")

plt.subplot(1,3,2)
y = aq.max()
plt.hist(y)
plt.title("Max")
plt.xlabel("x axis")
plt.ylabel("y axis")

plt.subplot(1,3,3)
z = aq.mean()
plt.hist(z)
plt.title("Mean")
plt.xlabel("x axis")
plt.ylabel("y axis")
# Variance as a visual
plt.subplot(1,3,3)
h = aq.var()
plt.hist(h)
plt.title("Variance")
plt.xlabel("x axis")
plt.ylabel("y axis")

plt.show()<jupyter_output><empty_output><jupyter_text>3. Calculate the descriptive statistics of the variables in the dataset *"Descriptive statistics are used to describe the basic features of the data in a study. They provide simple summaries about the sample and the measures. Together with simple graphics analysis, they form the basis of virtually every quantitative analysis of data."* [8]The website that has this definition of Descriptive Statistics also wrote some interesting examples of what happen when you include solely statistics rather than a overview of all available content.The first example relates to American baseball and if you were to take the statistics of a batter. If you were to only review the times the batter hit a ball it leaves it open to a very critical review; how many they missed vs hit.*"The batting average doesn't tell you whether the batter is hitting home runs or singles. It doesn't tell whether she's been in a slump or on a streak."*[8]Another example relates to a student and their result for a term. Again the result is the final number, a single piece of data defining the student's overall ability in the course. However the other aspects of the data from this example would include:* The level of the course being taken* The difficulty of the subjects* If these subjects were in their field of expertise or other areas.<jupyter_code>aq.mean()<jupyter_output><empty_output><jupyter_text>In the above code, I requested the mean as an output and as we can see:* The average mean of the X axis = 8.95* The average mean of the X axis = 7.47<jupyter_code># The min of the quartet dataset
aq.min()
# The max of the quartet dataset
aq.max()<jupyter_output><empty_output><jupyter_text>Having requested the min, max and mean output it is clear to see the close relationship and almost identical data for both axis of each graph. Gives plenty of backing to the old phrase that seeing really is believing.<jupyter_code># The variances of the quartet dataset
aq.var()
# X Axis Min and Max dataset
data = np.array([['','1st','2nd','3rd','4th'],
                ['Min',4.0,4.0,4.0,8.0],
                ['Max',14.0,14.0,14.0,19.0]])
                
print(pd.DataFrame(data=data[1:,1:],
                  index=data[1:,0],
                  columns=data[0,1:]))<jupyter_output>1st   2nd   3rd   4th
Min   4.0   4.0   4.0   8.0
Max  14.0  14.0  14.0  19.0
StarcoderdataJupyterStructuredCleanDedup