<jupyter_start><jupyter_text>Visualizing Google Forms Data with Seaborn This is the second part of an article from [Practical Business Python](htp://pbpython.com) describing how to retrieve and analyze data from a Google Form.Please review [part 1](http://pbpython.com/pandas-google-forms-part1.html) for the details of how to set up authentication and get the data into the pandaqs dataframe.The full article corresponding to this notebook is [here](http://pbpython.com/pandas-google-forms-part2.html)  Setup Bring in our standard imports as well as the authentication libraries we will need to get access to our form.<jupyter_code>import gspread
from oauth2client.client import SignedJwtAssertionCredentials
import pandas as pd
import json<jupyter_output><empty_output><jupyter_text>Import Ipython display as well as graphing libraries. For this article, we will be using [seaborn](http://stanford.edu/~mwaskom/software/seaborn/index.html).<jupyter_code>from IPython.display import display
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Setup authentication process to pull in the survey data stored in the Google Sheet.<jupyter_code>SCOPE = ["https://spreadsheets.google.com/feeds"]
SECRETS_FILE = "Pbpython-key.json"
SPREADSHEET = "PBPython User Survey (Responses)"
# Based on docs here - http://gspread.readthedocs.org/en/latest/oauth2.html
# Load in the secret JSON key (must be a service account)
json_key = json.load(open(SECRETS_FILE))
# Authenticate using the signed key
credentials = SignedJwtAssertionCredentials(json_key['client_email'],
                                            json_key['private_key'], SCOPE)<jupyter_output><empty_output><jupyter_text>Now open up the file and read all data in a DataFrame<jupyter_code>gc = gspread.authorize(credentials)
# Open up the workbook based on the spreadsheet name
workbook = gc.open(SPREADSHEET)
# Get the first sheet
sheet = workbook.sheet1
# Extract all data into a dataframe
results = pd.DataFrame(sheet.get_all_records())
results.head()<jupyter_output><empty_output><jupyter_text>We need to do some cleanup to make the data easier to analyze.<jupyter_code># Do some minor cleanups on the data
# Rename the columns to make it easier to manipulate
# The data comes in through a dictionary so we can not assume order stays the
# same so must name each column
column_names = {'Timestamp': 'timestamp',
                'What version of python would you like to see used for the examples on the site?': 'version',
                'How useful is the content on practical business python?': 'useful',
                'What suggestions do you have for future content?': 'suggestions',
                'How frequently do you use the following tools? [Python]': 'freq-py',
                'How frequently do you use the following tools? [SQL]': 'freq-sql',
                'How frequently do you use the following tools? [R]': 'freq-r',
                'How frequently do you use the following tools? [Javascript]': 'freq-js',
                'How frequently do you use the following tools? [VBA]': 'freq-vba',
                'How frequently do you use the following tools? [Ruby]': 'freq-ruby',
                'Which OS do you use most frequently?': 'os',
                'Which python distribution do you primarily use?': 'distro',
                'How would you like to be notified about new articles on this site?': 'notify'
                }
results.rename(columns=column_names, inplace=True)
results.timestamp = pd.to_datetime(results.timestamp)
results.head()<jupyter_output><empty_output><jupyter_text>There are a small number of free form comments. Let's strip those out and remove them from the results.<jupyter_code>suggestions = results[results.suggestions.str.len() > 0]["suggestions"]<jupyter_output><empty_output><jupyter_text>Since there are only a small number of comments, just print them out.However, if we had more comments and wanted to do more analysis we certainly good.<jupyter_code>for index, row in suggestions.iteritems():
    display(row)<jupyter_output><empty_output><jupyter_text>Drop the suggestions. We won't use them any more.<jupyter_code>results.drop("suggestions", axis=1, inplace=True)
results.head()<jupyter_output><empty_output><jupyter_text>Explore the data For Numeric columns, start with describe to see what we have<jupyter_code>results.describe()<jupyter_output><empty_output><jupyter_text>Because we only have 1, 2, 3 as options the numeric results aren't telling us that much. I am going to convert the number to more useful descriptions.<jupyter_code>results['useful'] = results['useful'].map({1: '1-low', 2: '2-medium', 3: '3-high'})
results.head()<jupyter_output><empty_output><jupyter_text>Value counts give us an easy distribution view into the raw numbers<jupyter_code>results["version"].value_counts()<jupyter_output><empty_output><jupyter_text>Use normalize to see it by percentage.<jupyter_code>results.os.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>While the numbers are useful, wouldn't it be nicer to visually show the results?Seaborn's [factorplot](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.factorplot.html) is helpful for showing this kind of categorical data.Because factorplot is so powerful, I'll build up step by step to show how it can be used for complex data analysis.First, look at number of users by OS.<jupyter_code>sns.factorplot("os", data=results, palette="BuPu")<jupyter_output><empty_output><jupyter_text>It is easy to order the results using x_order<jupyter_code>sns.factorplot("os", x_order=["Linux", "Windows", "Mac"], data=results, palette="BuPu")<jupyter_output><empty_output><jupyter_text>Do a similar plot on python version<jupyter_code>sns.factorplot("version", data=results, palette="BuPu")<jupyter_output><empty_output><jupyter_text>This is useful but wouldn't it be better to compare with OS and preferred python version? This is where factorplot starts to show more versatility. The key component is to use hue to automatically slice the data by python version (in this case).<jupyter_code>sns.factorplot("os", hue="version", x_order=["Linux", "Windows", "Mac"], data=results, palette="Paired")<jupyter_output><empty_output><jupyter_text>Because seaborn knows how to work with dataframes, we just need to pass in the column names for the various arguments and it will do the analysis and presentation.How about if we try to see if there is any relationship between how useful the site is and OS/Python choice? We can add the useful column into the plot using col.<jupyter_code>sns.factorplot("version", hue="os", data=results, col="useful", palette="Paired")<jupyter_output><empty_output><jupyter_text>If we can add a column, we can also add a row and seaborn takes care of the rest.In looking at the data, we have two different versions of winpython so clean that up first.<jupyter_code>results['distro'] = results['distro'].str.replace('WinPython', 'winpython')
results.head()<jupyter_output><empty_output><jupyter_text>We can also look at the distros. Since there is some overlap with the distros and os, let's only look at a subset of distros. For instance, someone using winpython is not going to be using it on a Mac.<jupyter_code>results['distro'].value_counts()<jupyter_output><empty_output><jupyter_text>The most meaningful data would be looking at the Anaconda and Official python.org binaries. Let's filter all of our data only on these two values.<jupyter_code>results_distro = results[results["distro"].isin(["Anaconda", "Official python.org binaries"])]
results_distro.head()<jupyter_output><empty_output><jupyter_text>Now do our factorplot with multiple columns and rows using row and col.<jupyter_code>sns.factorplot("version", hue="os", data=results_distro, col="useful", row="distro", margin_titles=True, sharex=False)<jupyter_output><empty_output><jupyter_text>Responses over time We know that we have 55 results now. It would be interesting to see how those results came in over time. Using this method, we can very simply look at this by any time period we want.The seaborn's [timeseries](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.tsplot.html) supports this type of analysis and much more.For ease of calculating responses over time, add a count colum for each response.<jupyter_code>results["count"] = 1
results.head()<jupyter_output><empty_output><jupyter_text>To get totals over time, set our index to the timestamp<jupyter_code>total_results = results.set_index('timestamp')
total_results.head()<jupyter_output><empty_output><jupyter_text>Use pandas TimeGrouper to summarize the data by day and do a cumulative sum. We could easily do this for any time period too.<jupyter_code>running_results = total_results.groupby(pd.TimeGrouper('D'))["count"].count().cumsum()
running_results<jupyter_output><empty_output><jupyter_text>To label the x-axis we need to define our time range<jupyter_code>step = pd.Series(range(0,len(running_results)), name="Days")
sns.tsplot(running_results, value="Total Responses", time=step, color="husl")<jupyter_output><empty_output><jupyter_text>Heatmaps and Clustermaps The final section of data to analyze is the frequency with which readers are using different technology. I am going to use a [heatmap](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.heatmap.htmlseaborn.heatmap) to look for any interesting insights. Let's look at the data again.<jupyter_code>results.head()
results["freq-py"].value_counts()<jupyter_output><empty_output><jupyter_text>What we need to do is construct a single DataFrame with all the value_counts for the specific technology.First we will create a list containing each value count.<jupyter_code>all_counts = []
for tech in ["freq-py", "freq-sql", "freq-r", "freq-ruby", "freq-js", "freq-vba"]:
    all_counts.append(results[tech].value_counts())
display(all_counts)<jupyter_output><empty_output><jupyter_text>Now, concat the lists along axis=1.Fill in any nan values with 0 too.<jupyter_code>tech_usage = pd.concat(all_counts, keys=["Python", "SQL", "R", "Ruby", "javascript", "VBA"], axis=1)
tech_usage = tech_usage.fillna(0)
tech_usage<jupyter_output><empty_output><jupyter_text>We have a nice table but there are a few problems.First, we have one column with blank values that we don't want.Secondly, we would like to order from Daily -> Never. Use reindex to accomplish both tasks.<jupyter_code>tech_usage = tech_usage.reindex(["Daily", "A couple times a week", "Once a month", "Infrequently", "Never"])
tech_usage<jupyter_output><empty_output><jupyter_text>Now that the data is in the correct table format, we can create a heatmap.<jupyter_code>sns.heatmap(tech_usage, annot=True)<jupyter_output><empty_output><jupyter_text>So, what does this tell us?Not surprisingly, most people use python very frequently.Additionally, it looks like very few survey takers are using Ruby or VBA. A variation of the heatmap is the [clustermap](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.clustermap.htmlseaborn.clustermap). The main feature it does is that it tries to reorganize the data to more easily see relationships/clusters.<jupyter_code>sns.clustermap(tech_usage, annot=True)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup