# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/Higgins2718/DS-Unit-1-Sprint-1-Dealing-With-Data/blob/master/module2-loadingdata/LS_DS_112_Loading_Data.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="-c0vWATuQ_Dn" colab_type="text"
# # Lambda School Data Science - Loading Data
#
# Data comes in many shapes and sizes - we'll start by loading tabular data, usually in csv format.
#
# Data set sources:
#
# - https://archive.ics.uci.edu/ml/datasets.html
# - https://github.com/awesomedata/awesome-public-datasets
# - https://registry.opendata.aws/ (beyond scope for now, but good to be aware of)
#
# Let's start with an example - [data about flags](https://archive.ics.uci.edu/ml/datasets/Flags).

# + [markdown] id="wxxBTeHUYs5a" colab_type="text"
# ## Lecture example - flag data

# + id="nc-iamjyRWwe" colab_type="code" outputId="b8350169-6f0c-4405-84a4-2880111b3f98" colab={"base_uri": "https://localhost:8080/", "height": 3406}
# Step 1 - find the actual file to download

# From navigating the page, clicking "Data Folder"
flag_data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data'

# You can "shell out" in a notebook for more powerful tools
# https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html

# Funny extension, but on inspection looks like a csv
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data

# Extensions are just a norm! You have to inspect to be sure what something is

# + id="UKfOq1tlUvbZ" colab_type="code" colab={}
# Step 2 - load the data

# How to deal with a csv? 🐼
import pandas as pd
flag_data = pd.read_csv(flag_data_url)

# + id="exKPtcJyUyCX" colab_type="code" outputId="d036d2da-f5ae-464e-eadd-65ed66579bbb" colab={"base_uri": "https://localhost:8080/", "height": 273}
# Step 3 - verify we've got *something*
flag_data.head()

# + id="rNmkv2g8VfAm" colab_type="code" outputId="de9415b2-a508-4766-cd1f-ac500cea02f9" colab={"base_uri": "https://localhost:8080/", "height": 555}
# Step 4 - Looks a bit odd - verify that it is what we want
flag_data.count()

# + id="iqPEwx3aWBDR" colab_type="code" outputId="82130cf7-e71a-46ba-a8e7-664ee4700348" colab={"base_uri": "https://localhost:8080/", "height": 106}
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data | wc

# + id="5R1d1Ka2WHAY" colab_type="code" outputId="e55f74d5-0e19-4473-d22f-a101b0f7fbd3" colab={"base_uri": "https://localhost:8080/", "height": 4813}
# So we have 193 observations with funny names, file has 194 rows
# Looks like the file has no header row, but read_csv assumes it does
help(pd.read_csv)

# + id="o-thnccIWTvc" colab_type="code" outputId="57799072-a851-4d4f-fd37-e24f6997492b" colab={"base_uri": "https://localhost:8080/", "height": 273}
# Alright, we can pass header=None to fix this
flag_data = pd.read_csv(flag_data_url, header=None)
flag_data.head()

# + id="iG9ZOkSMWZ6D" colab_type="code" outputId="d8b6f16a-6964-41f4-b438-9ddbff9eb08a" colab={"base_uri": "https://localhost:8080/", "height": 555}
flag_data.count()

# + id="gMcxnWbkWla1" colab_type="code" outputId="52fc01aa-00bd-4420-8539-0fe7dba9e263" colab={"base_uri": "https://localhost:8080/", "height": 555}
flag_data.isna().sum()

# + [markdown] id="AihdUkaDT8We" colab_type="text"
# ### Yes, but what does it *mean*?
#
# This data is fairly nice - it was "donated" and is already "clean" (no missing values). But there are no variable names - so we have to look at the codebook (also from the site).
#
# ```
# 1. name: Name of the country concerned
# 2. landmass: 1=N.America, 2=S.America, 3=Europe, 4=Africa, 4=Asia, 6=Oceania
# 3. zone: Geographic quadrant, based on Greenwich and the Equator; 1=NE, 2=SE, 3=SW, 4=NW
# 4. area: in thousands of square km
# 5. population: in round millions
# 6. language: 1=English, 2=Spanish, 3=French, 4=German, 5=Slavic, 6=Other Indo-European, 7=Chinese, 8=Arabic, 9=Japanese/Turkish/Finnish/Magyar, 10=Others
# 7. religion: 0=Catholic, 1=Other Christian, 2=Muslim, 3=Buddhist, 4=Hindu, 5=Ethnic, 6=Marxist, 7=Others
# 8. bars: Number of vertical bars in the flag
# 9. stripes: Number of horizontal stripes in the flag
# 10. colours: Number of different colours in the flag
# 11. red: 0 if red absent, 1 if red present in the flag
# 12. green: same for green
# 13. blue: same for blue
# 14. gold: same for gold (also yellow)
# 15. white: same for white
# 16. black: same for black
# 17. orange: same for orange (also brown)
# 18. mainhue: predominant colour in the flag (tie-breaks decided by taking the topmost hue, if that fails then the most central hue, and if that fails the leftmost hue)
# 19. circles: Number of circles in the flag
# 20. crosses: Number of (upright) crosses
# 21. saltires: Number of diagonal crosses
# 22. quarters: Number of quartered sections
# 23. sunstars: Number of sun or star symbols
# 24. crescent: 1 if a crescent moon symbol present, else 0
# 25. triangle: 1 if any triangles present, 0 otherwise
# 26. icon: 1 if an inanimate image present (e.g., a boat), otherwise 0
# 27. animate: 1 if an animate image (e.g., an eagle, a tree, a human hand) present, 0 otherwise
# 28. text: 1 if any letters or writing on the flag (e.g., a motto or slogan), 0 otherwise
# 29. topleft: colour in the top-left corner (moving right to decide tie-breaks)
# 30. botright: Colour in the bottom-left corner (moving left to decide tie-breaks)
# ```
#
# Exercise - read the help for `read_csv` and figure out how to load the data with the above variable names. One pitfall to note - with `header=None` pandas generated variable names starting from 0, but the above list starts from 1...

# + [markdown] id="nPbUK_cLY15U" colab_type="text"
# ## Your assignment - pick a dataset and do something like the above
#
# This is purposely open-ended - you can pick any data set you wish. It is highly advised you pick a dataset from UCI or a similar "clean" source.
#
# If you get that done and want to try more challenging or exotic things, go for it! Use documentation as illustrated above, and follow the 20-minute rule (that is - ask for help if you're stuck).
#
# If you have loaded a few traditional datasets, see the following section for suggested stretch goals.

# + id="NJdISe69ZT7E" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 5417} outputId="56d96e75-9e20-42d9-ff4a-633f456b4ef1"
# TODO your work here!
# And note you should write comments, descriptions, and add new
# code and text blocks as needed


# URL for dataset, horse-colic
colic_data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/horse-colic/horse-colic.data'


# Download dataset
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/horse-colic/horse-colic.data

# + id="w6iAz7JCGzgD" colab_type="code" colab={}
import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

colic_data = pd.read_csv(colic_data_url, header=None)

# + id="R10pr_vqG1dY" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="1aff08f9-2806-4a3d-a545-1ebd029e2070"
colic_data.head()

# + id="SqRZdDtiG4x1" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="fc2136eb-f141-4cc2-c689-3fff3a2d0030"
# Exploring dataset some more, looking at shape
colic_data.shape

# + id="6MLui9oTG7Qi" colab_type="code" colab={"resources": {"http://localhost:8080/nbextensions/google.colab/files.js": {"data": "<KEY> "ok": true, "headers": [["content-type", "application/javascript"]], "status": 200, "status_text": ""}}, "base_uri": "https://localhost:8080/", "height": 72} outputId="9e3a0540-6051-4d57-92d1-65c8b616c4b8"
from google.colab import files
uploaded = files.upload()

# + id="Wtyz-_P0HBJf" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 275} outputId="b2aa7507-8e9e-4508-8b83-18df291e8cc2"
df = pd.read_csv('horse-colic.data', header=None, names=['surgery?','Age',
                                                         'Hospital Number',
                                                         'rectal temperature',
                                                         'pulse',
                                                         'respiratory rate',
                                                   'temperature of extremities',
                                                         'peripheral pulse',
                                                         'mucous membranes',
                                                        'capillary refill time',
                                                         'pain',
                                                         'peristalsis',
                                                         'abdominal distension',
                                                         'nasogastric tube',
                                                         'nasogastric reflux',
                                                        'nasogastric reflux PH',
                                                   'rectal examination - feces',
                                                         'abdomen',
                                                         'packed cell volume',
                                                         'total protein',
                                                  'abdominocentesis appearance',
                                                  'abdomcentesis total protein',
                                                         'outcome',
                                                         'surgical lesion?',
                                                         'type of lesion ',
                                                         'type of lesion 2',
                                                         'type of lesion 3', 
                                                         'cp_data'], sep=" ",
                                                          na_values=["?"])
# Below line of code was used for debugging purposes; I had some difficulty with the CSV format.
#df = pd.read_csv('horse-colic.data', sep=" ")
df.head()


# + id="N01mfK3_HEPv" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 539} outputId="bbfe0cdf-ce45-4277-e09f-b82bd53c51b8"
# Lots of null values for nasogastric tube/reflux/reflux PH, rectal examination, abdomen, abdominocentesis appearance, and abdomcentesis total protein
df.isnull().sum()

# + [markdown] id="55ktlNrCHIoh" colab_type="text"
# As I commented in the above code cell, certain columns are missing copious amounts of data. It would be impractical to do anything other than remove all NaNs.

# + id="_7b4fp2sHJet" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 293} outputId="93046905-0fbd-4ba1-e1c7-0e9d1bebb573"
df = df[~(df.isnull().any(axis=1))]
print(df.shape)
df.head()

# + id="rYY9007wHN4r" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="ea865b3b-2325-4b3c-cf94-eea9a7e9d572"
# Testing to make sure all NaNs have been removed
df.isnull().any().sum()

# + [markdown] id="MZCxTwKuReV9" colab_type="text"
# ## Stretch Goals - Other types and sources of data
#
# Not all data comes in a nice single file - for example, image classification involves handling lots of image files. You still will probably want labels for them, so you may have tabular data in addition to the image blobs - and the images may be reduced in resolution and even fit in a regular csv as a bunch of numbers.
#
# If you're interested in natural language processing and analyzing text, that is another example where, while it can be put in a csv, you may end up loading much larger raw data and generating features that can then be thought of in a more standard tabular fashion.
#
# Overall you will in the course of learning data science deal with loading data in a variety of ways. Another common way to get data is from a database - most modern applications are backed by one or more databases, which you can query to get data to analyze. We'll cover this more in our data engineering unit.
#
# How does data get in the database? Most applications generate logs - text files with lots and lots of records of each use of the application. Databases are often populated based on these files, but in some situations you may directly analyze log files. The usual way to do this is with command line (Unix) tools - command lines are intimidating, so don't expect to learn them all at once, but depending on your interests it can be useful to practice.
#
# One last major source of data is APIs: https://github.com/toddmotto/public-apis
#
# API stands for Application Programming Interface, and while originally meant e.g. the way an application interfaced with the GUI or other aspects of an operating system, now it largely refers to online services that let you query and retrieve data. You can essentially think of most of them as "somebody else's database" - you have (usually limited) access.
#
# *Stretch goal* - research one of the above extended forms of data/data loading. See if you can get a basic example working in a notebook. Image, text, or (public) APIs are probably more tractable - databases are interesting, but there aren't many publicly accessible and they require a great deal of setup.

StarcoderdataJupyterScriptsDedupFiltered