<jupyter_start><jupyter_text>4. Working with Data in Python![](images/logo.png)Welcome to the third lab session!This lab includes some exercises about the material covered in session 4 concerning files and pandas. > **NOTE**: If your are running this lab on your own machine (i.e. not on our JupyterHub server), you need to make sure Pandas has already been installed. To check whether you have installed Pandas try running the cell below:<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>If nothing much happens then you're good to go. However, if you get an `ImportError`, this means pandas has not been installed. To install pandas, open your command prompt and type ```pip install pandas```and hit enter. Once the install has finished, you should be good to go. Come back and retry running the cell above. With that done, let's jump in! Exercise 1: Reading in `.txt` files a) `with` statementsIf you change tab to the Jupyter file explorer, you'll notice a folder named `dreams`. If you take a look inside, you'll see 5 files called `dream1.txt`, `dream2.txt` ... , `dream5.txt`. These are short write-ups of dreams people have had, taken from the website [DreamBank](http://www.dreambank.net/), which has a collection of over 20,000 dreams recorded from many different people. Each file is made up of 3 lines. The first gives the name of the dreamer, the second gives a date, and the third has the dream content. Below is some code that reads in `dream1.txt`. In the cell below that, rewrite the code so that it does the exact same thing, but uses the `with` statement.<jupyter_code>text_file = open('dreams/dream1.txt')

contents = text_file.read()

print(contents)

text_file.close()
with open('dreams/dream1.txt') as text_file:
    
    contents = text_file.read()
    
    print(contents)<jupyter_output>Name: Alta
Date: 19/01/1986
At work (?) was on the elevator with an older woman who wasn't sure what floor she wanted so she pushed extra buttons - I'd already gone up when I meant to go down to 2, and wasn't thrilled at having to stop so much. So I got off (she was surprised I took exception) and found myself outside, sort of, and now I have to get in a car that goes somewhere down some streets like in Oak town - the car was like an old '50s sort of thing with fins and they were coming off and needed to be stuck back down, took a while - arrived at a place that was an "elevator stop", nice old tree-lined street, the door didn't face the street - it was ivy-covered stone, very nice. Inside it was cave-like and open, but there was the elevator door. There was somebody there as well as 2 cats, and I made some comment about them having kittens (like it would be a good idea) and the guy said they were both males or something and I felt the same sort of cuteness about the exchange you usual[...]<jupyter_text>b) `readlines`Copy and paste your code from above. Replace `.read()` with `.readlines()`. This will create a list, where each element is a new line.Remember, sometimes it may look like text is going over multiple lines when really it's on just one. Using list indexing, select the 3rd element from this list (remember, list indexing starts from 0) so that you extract only the text, not the name or date. Print this text.<jupyter_code>with open('dreams/dream1.txt') as text_file:
    
    contents = text_file.readlines()[2]
    
print(contents)<jupyter_output>At work (?) was on the elevator with an older woman who wasn't sure what floor she wanted so she pushed extra buttons - I'd already gone up when I meant to go down to 2, and wasn't thrilled at having to stop so much. So I got off (she was surprised I took exception) and found myself outside, sort of, and now I have to get in a car that goes somewhere down some streets like in Oak town - the car was like an old '50s sort of thing with fins and they were coming off and needed to be stuck back down, took a while - arrived at a place that was an "elevator stop", nice old tree-lined street, the door didn't face the street - it was ivy-covered stone, very nice. Inside it was cave-like and open, but there was the elevator door. There was somebody there as well as 2 cats, and I made some comment about them having kittens (like it would be a good idea) and the guy said they were both males or something and I felt the same sort of cuteness about the exchange you usually do talking about kitties.[...]<jupyter_text>c) Extracting the text from multiple filesComplete the code below to print the text from all five files.<jupyter_code>os.listdir('dreams')
import os

for file_name in os.listdir('dreams'):
     
    path = 'dreams/' + file_name
    
    with open(path) as text_file:
        
        print(text_file.readlines()[2])<jupyter_output>A tall many-storied building stands near a road. A construction crew is looking for a structural problem. I go into the building to the basement and I see a twisted place in a girder, a long beam. I come out and tell the foreman the building is about to collapse. We start running away and hiding in the bushes. I am next to the road and realize I must stop traffic because the building is going to fall on the road. I run onto the road yelling at the pedestrians to stop! Some ignore me and break past me, walking into the dangerous area. I see the building and actually try to visualize it falling because it isn't and I've got all these people stopped and they are annoyed at me. It finally collapses far away from the road. A person glares at me for hindering them. I shrug. Better safe than sorry.
At work (?) was on the elevator with an older woman who wasn't sure what floor she wanted so she pushed extra buttons - I'd already gone up when I meant to go down to 2, and wasn't thrilled at havi[...]<jupyter_text>Exercise 2: Writing filesBelow you will see a function being imported from a file called `random_sentence.py`. (This is another use of `import` - to bring in functions written in external files). Try running the function `generate_random_sentence()` a few times to see what it does.<jupyter_code>from random_sentence import generate_random_sentence

random_sentence = generate_random_sentence()

print(random_sentence)<jupyter_output>The splendid pug really loved the industrious duck<jupyter_text>a) Writing many lines 1By using the function `generate_random_sentence()`, complete the code below to write 100 random sentences to a file called `random_sentences1.txt`, using the `.write()` function. Check this has worked by opening the file in the Jupyter launcher.Remember to use the `\n` character to signify a new line.<jupyter_code>n_sentences = 100

with open('random_sentences1.txt', 'w') as my_file:
    
    for i in range(n_sentences):
        
        my_file.write(generate_random_sentence() + '\n')<jupyter_output><empty_output><jupyter_text>a) Writing many lines 2Do the same thing again, but this time use a list of sentences with the `.writelines()` function.<jupyter_code>n_sentences = 100

lines = []

for i in range(n_sentences):
    
    lines.append(generate_random_sentence() + '\n')
    
    
with open('random_sentences2.txt', 'w') as my_file:
    
    my_file.writelines(lines)<jupyter_output><empty_output><jupyter_text>Exercise 3: Pandas a) Creating a DataFrame from a list of listsUse the following data to create a pandas DataFrame. Store it in a variable called `df`.<jupyter_code>import pandas as pd


column_names = ['name', 'calories', 'protein', 'fat', 'sodium', 'fiber']

data = [['100% Bran', 70, 4, 1, 130, 10],
        ['100% Natural Bran', 120, 3, 5, 15, 2],
        ['All-Bran', 70,  4, 1, 260, 9],
        ['All-Bran with Extra Fiber', 50,  4, 0, 140, 14],
        ['Almond Delight', 110, 2, 2, 200, 1]]

df = pd.DataFrame(data, columns=column_names)
df<jupyter_output><empty_output><jupyter_text>b) Column informationUse the function `df.mean()` to find the mean of each column. Do the same for `.min()` and `.max()`.<jupyter_code>df.mean()
df.min()
df.max()<jupyter_output><empty_output><jupyter_text>c) Loading data from a csv fileBelow is some code that opens the full dataset from a file. Edit it so that the `'name'` column becomes the index column.<jupyter_code>df = pd.read_csv('cereal.csv', index_col='name')
df<jupyter_output><empty_output><jupyter_text>d) Getting row informationUse `.loc[]` to find out all information about `'Cheerios'`<jupyter_code>df.loc['Cheerios']<jupyter_output><empty_output><jupyter_text>e) Adding a healthy or unhealthy labelWe are now going to add a new column called `'healthy'` which will contain the values `True` or `False` for each cereal type. If the calories are greater than 120 or the sugar is greater than 10, we will mark it as unhealthy. Otherwise we will mark it as healthy. The code below creates a list where each element is `True` or `False`, which specifies the health status of the corresponding cereal. Add this as a new column to the dataframe.<jupyter_code>healthy = []

for cereal in df.index:
    
    row = df.loc[cereal]
    
    if row['calories'] > 120 or row['sugars'] > 10:
       
        healthy.append(False)
        
    else:
        
        healthy.append(True)
        
print(healthy)
df['healthy'] = healthy
df<jupyter_output><empty_output>