# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Boolean Selection More
#
# ## Boolean selection on a Series
# All of the examples thus far have taken place on DataFrames. Boolean selection on a Series happens almost identically. Since there is only one dimension of data, the queries you ask are usually going to be simpler. First, let's select a single column of data as a Series such as the temperature column.

import pandas as pd
bikes = pd.read_csv('../data/bikes.csv', parse_dates=['starttime', 'stoptime'])
temp = bikes['temperature']
temp.head(3)

# Let's select temperatures greater than 90. The procedure is the same as with DataFrames. Create a boolean Series and pass that Series to *just the bracketes*.

filt = temp > 90
temp[filt].head(3)

# Select temperature less than 0 or greater than 95. Multiple condition boolean Series also work the same.

filt1 = temp < 0
filt2 = temp > 95
filt = filt1 | filt2
temp[filt].head()

# ### Re-read data with `starttime` in the index
# The default index is not very helpful. Let's reread the data with `starttime` in the index. While, this column may not be unique it does provide us with useful labels for each row.

bikes = pd.read_csv('../data/bikes.csv', 
                    parse_dates=['starttime', 'stoptime'], 
                    index_col='starttime')
temp2 = bikes['temperature']
temp2.head()

# Let's select temperatures greater than 90. We expect to get a summer month and we do.

filt = temp2 > 90
temp2[filt].head(5)

# Select temperature less than 0 or greater than 95. We expect to get some winter months in the result and we do.

filt1 = temp2 < 0
filt2 = temp2 > 95
filt = filt1 | filt2
temp2[filt].head()

# ## The `between` method
# The `between` method returns a boolean Series by testing whether the current value is between two given values. For instance, if want to select the temperatures between 50 and 60 degrees we do the following:

filt = temp2.between(50, 60)
filt.head(3)

# By default, the `between` method is inclusive of the given values, so temperatures of exactly 50 or 60 would be found in the result. We pass this boolean Series to *just the brackets* to complete the selection.

temp2[filt].head(3)

# ## Simultaneous boolean selection of rows and column labels with `loc`
# The `loc` indexer was thoroughly covered in an earlier chapter and will now be brought up again to show how it can simultaneously selects rows with boolean selection and columns by labels.
#
# Remember that `loc` takes both a row selection and a column selection separated by a comma. Since the row selection comes first, you can pass it the same exact inputs that you do for *just the brackets* and get the same results. Let's run some of the previous examples of boolean selection with `loc`.

filt = bikes['tripduration'] > 1000
bikes.loc[filt].head(3)

filt = bikes['events'].isin(['rain', 'snow', 'tstorms', 'sleet'])
bikes.loc[filt].head(3)

# ### Separate row and column selection with a comma for `loc`
# The nice benefit of `loc` is that it allows us to simultaneously do boolean selection for the rows and select columns by label. Let's select rides during rain or snow and the columns `events` and `tripduration`.

filt = bikes['events'].isin(['rain', 'snow'])
cols = ['events', 'tripduration']
bikes.loc[filt, cols].head()

# Now let's find all female riders with trip duration greater than 5000 when it was cloudy. We'll only return the columns used during the boolean selection.

filt1 = bikes['gender'] == 'Female'
filt2 = bikes['tripduration'] > 5000
filt3 = bikes['events'] == 'cloudy'
filt = filt1 & filt2 & filt3
cols = ['gender', 'tripduration', 'events']
bikes.loc[filt, cols]

# ## Column to column comparisons
# So far, we have created filters by comparing each of our column values to a single scalar value. It is possible to do element-by-element comparisons by comparing two columns to one another. For instance, the total bike capacity at the each station at the start and end of the ride is stored in the `dpcapacity_start` and `dpcapacity_end` columns. If we wanted to test whether there were more capacity at the start of the ride vs the end, we would do the following:

filt = bikes['dpcapacity_start'] > bikes['dpcapacity_end']

# Let's use this filter with `loc` to return all the rows where the start capacity is greater than the end.

cols = ['dpcapacity_start', 'dpcapacity_end']
bikes.loc[filt, cols].head(3)

# ### Boolean selection with `iloc` does not work
# The pandas developers decided not to allow boolean selection with `iloc`.

bikes.iloc[filt]

# ## Finding Missing Values with `isna`
# The `isna` method called from either a DataFrame or a Series returns `True` for every value that is missing and `False` for any other value. Let's see this in action by calling `isna` on the start capacity column.

bikes['dpcapacity_start'].isna().head(3)

# ### Filtering for missing values
#
# We can now use this boolean Series to select all the rows where the capacity start column is missing. Verify that those values are indeed missing. 

filt = bikes['dpcapacity_start'].isna()
bikes[filt].head(3)

# ### `isnull` is an alias for `isna`
# There is an identical method named `isnull` that you will see in other tutorials. It is an **alias** of `isna` meaning it does the exact same thing but has a different name. Either one is suitable to use, but I prefer `isna` because of the similarity to **NaN**, the representation of missing values. There are also other methods such as `dropna` and `fillna` that use the 'na' in their method names.

# ## Exercises
#
# ### Exercise 1
# <span  style="color:green; font-size:16px">Select the wind speed column a a Series and assign it to a variable. Are there any negative wind speeds?</span>



# ### Exercise 2
# <span  style="color:green; font-size:16px">Select all wind speed values between 12 and 16.</span>



# ### Exercise 3
# <span  style="color:green; font-size:16px">Select the `events` and `gender` columns for all trip durations longer than 1,000 seconds.</span>



# ### Exercise 4
# <span  style="color:green; font-size:16px">Read in the movie dataset with the title as the index. We will use this DataFrame for the rest of the problems. Select all the movies such that the Facebook likes for actor 2 are greater than those for actor 1.</span>



# ### Exercise 5
# <span  style="color:green; font-size:16px">Select the year, content rating, and IMDB score columns for movies from the year 2016 with IMDB score less than 4.</span>



# ### Exercise 6
# <span  style="color:green; font-size:16px">Select all the movies that are missing values for content rating.</span>



# ### Exercise 7
# <span  style="color:green; font-size:16px">Select all the movies that are missing values for both the gross and budget columns. Return just those columns to verify that those values are indeed missing.</span>



# ### Exercise 8
# <span  style="color:green; font-size:16px">Write a function `find_missing` that has three parameters, `df`, `col1` and `col2` where `df` is a DataFrame and `col1` and `col2` are column names. This function should return all the rows of the DataFrame where `col1` and `col2` are missing. Only return the two columns as well. Answer problem 7 with this function.</span>



StarcoderdataJupyterScriptsDedupFiltered