<jupyter_start><jupyter_text>Collection of snippets for Pandas<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Any `pd.Series` can also be a `pd.DataFrame`<jupyter_code>my_list = [1, 2, 3]
index = ["This", "That", "What!"]
pd.DataFrame(my_list, index=index)
pd.DataFrame(my_list, columns=['a_number'], index=index)<jupyter_output><empty_output><jupyter_text>Creating dataframes using dictionaries<jupyter_code>my_dict = {'this': 1, 'that': 2, 'what!': 3}<jupyter_output><empty_output><jupyter_text>We know that each row is a `pd.Series`<jupyter_code>pd.Series(my_dict)
pd.DataFrame(pd.Series(my_dict))<jupyter_output><empty_output><jupyter_text>Q: What would be the output of this?<jupyter_code>pd.DataFrame([{'a': 1, 'b': 2}, {'b': 3, 'c': 4}])<jupyter_output><empty_output><jupyter_text>Before moving on, I also want to talk about numpy vectorization again!<jupyter_code>np.array([1, 3, 4, 5, 5])
my_arr = np.array([1, 3, 4, 5, 5])
my_arr == 5<jupyter_output><empty_output><jupyter_text>Want to get the slice using filter<jupyter_code>my_arr[my_arr == 5]<jupyter_output><empty_output><jupyter_text>You can do it similarly using pandas!<jupyter_code>weather_df.head()<jupyter_output><empty_output><jupyter_text>Q: You need to return rows that has 'Weather == Kya Chal raha hai'  `df.query`>DataFrame.query: Query the columns of a frame with a boolean expression.<jupyter_code># Using loc or iloc
# NOTE: slice is `:5:2`, not 6 !
# df.loc[row number/slice, 'Column name/s']

## iloc == numpy slicing! Damn! 
## df.iloc[row number/slice, column number/slice]<jupyter_output><empty_output><jupyter_text>You can do a lot of things with the indices as well: `df.sort_index`<jupyter_code>weather_df2.sort_index(ascending=False).head()<jupyter_output><empty_output><jupyter_text>If you have sort values by index, then shouldn't there be sort_by_value?<jupyter_code>weather_df2.sort_values(by=['Temp (C)', 'Dew Point Temp (C)'], ascending=False)<jupyter_output><empty_output><jupyter_text>Add a few rows as lists<jupyter_code>date_time = ['2012-13-01', '2013-01-01', '2013-01-02', '2012-01-02']<jupyter_output><empty_output><jupyter_text>Letting few of them to be NaNs<jupyter_code>temp = [np.nan, np.nan, 30, 12]
dew_pt_temp = [-2, np.nan, np.nan, -1]
relative_humidity = [np.nan, np.nan, np.nan, np.nan]
wind_speed = [np.nan, np.nan, np.nan, 50]
visibility = [np.nan, 10.0, np.nan, 12.1]
stn_pressure = [104.1, np.nan, 101.2, 101.24]
weather = ['Snow', None, None, 'Fog']<jupyter_output><empty_output><jupyter_text>Create a new dataframe<jupyter_code>pd.DataFrame([date_time, temp, dew_pt_temp, relative_humidity, wind_speed, visibility, stn_pressure, weather])
weather_df3 = pd.DataFrame([date_time, temp, dew_pt_temp, relative_humidity, wind_speed, visibility, stn_pressure, weather]).T
weather_df3.columns.tolist()
weather_df3.columns = ['Date/Time', 'Temp (C)', 'Dew Point Temp (C)', 'Rel Hum (%)', 'Wind Spd (km/h)', 'Visibility (km)', 'Stn Press (kPa)', 'Weather']
weather_df3.columns.tolist()<jupyter_output><empty_output><jupyter_text>Display the dataframe<jupyter_code>weather_df3.head()<jupyter_output><empty_output><jupyter_text>Append the `weather_df2` and `weather_df3` by rows<jupyter_code>weather_df_appended_1 = weather_df.append(weather_df3)
weather_df_appended_1.reset_index(drop=True, inplace=True)
weather_df_appended_1.tail(10)
weather_df_appended2 = weather_df.append(weather_df3, ignore_index=True)<jupyter_output><empty_output><jupyter_text>Dealing with missing values<jupyter_code>weather_df.tail()<jupyter_output><empty_output><jupyter_text>How do you filter out the NaNs* Check the NaNs: `df.isna()`* Check the Null values: `df.isnull()` How do you treat those nulls or NaNs:* Drop the rows entirely!: `df.dropna()`* Fill the values with something: `df.fillna()`  We want count of the values: `.sum()`<jupyter_code>weather_df_appended2.isnull().sum()<jupyter_output><empty_output><jupyter_text>What if we just want to check if there's null data or not?: `.any()`<jupyter_code>weather_df_appended2.isnull().any()<jupyter_output><empty_output><jupyter_text>Finding the count of nan and empty strings (yes, empty string "" or " " is not handled separately, occurs mostly in character and categorical variable list)  Remember the numpy aggregate operations?  using df.mean to do fill operations<jupyter_code>weather_df_appended2['Temp (C)'].mean()
weather_df_appended2['Temp (C)'].fillna(value=weather_df_appended2['Temp (C)'].mean()).tail()<jupyter_output><empty_output><jupyter_text>Working with string, more examples!<jupyter_code>monte = pd.Series(['<NAME>', '<NAME>', '<NAME>',
                   '<NAME>', '<NAME>', '<NAME>'])<jupyter_output><empty_output><jupyter_text>There are generic methods: `.str.` But there's one essential thing: using regexes```bashMethod	    Descriptionmatch()	    Call re.match() on each element, returning a boolean.extract()	Call re.match() on each element, returning matched groups as strings.findall()	Call re.findall() on each elementreplace()	Replace occurrences of pattern with some other stringcontains()	Call re.search() on each element, returning a booleancount()	    Count occurrences of patternsplit()	    Equivalent to str.split(), but accepts regexpsrsplit()	Equivalent to str.rsplit(), but accepts regexps```  Using DateTime<jupyter_code>from datetime import datetime
datetime.now()<jupyter_output><empty_output><jupyter_text>`strftime` formatting in python [One stop source](http://strftime.org/)<jupyter_code>from dateutil import parser
date = parser.parse("4th of July, 2015")
date
date.strftime("%d")
dates = pd.to_datetime([datetime(2015, 7, 3), '4th of July, 2015',
                       '2015-Jul-6', '07-07-2015', '20150708', '2010/11/12', '2010.11.12'])
dates<jupyter_output><empty_output><jupyter_text>Formatting is baked in `pd.to_datetime`<jupyter_code>pd.to_datetime('2010/11/12', format='%Y/%m/%d')<jupyter_output><empty_output><jupyter_text>We talking about this yesterday!<jupyter_code>pd.date_range(start, periods=1000, freq='M')<jupyter_output><empty_output><jupyter_text>Requested snippet: What if there's a date: '13-13-2018'<jupyter_code>dates = pd.to_datetime([datetime(2015, 7, 3), '4th of July, 2015',
                       '2015-Jul-6', '07-07-2015', '20150708'])
dates<jupyter_output><empty_output><jupyter_text>Working with Categorical Variables<jupyter_code>weather_df_appended2['Weather'].head()
#pd.get_dummies(weather_df, columns=['Weather'])<jupyter_output><empty_output><jupyter_text>Few other tricks:  * Reading directly from a simple html webpage<jupyter_code>tables = pd.read_html("http://www.basketball-reference.com/leagues/NBA_2016_games.html")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup