# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Data Analytics Project from Scratch:
# _(In this project i've tried to showcase my approach and startegy for selecting and solving a data analytics problem and since this is only for the purpose of recalling and refreshing my data analytics skills that's why i might use lot of comments and key notes throughout the project)_

# ## Idea Creation:
#

# In this section of the notebook I'll go through the planning and idea creation for this new data analytics project and the next sections of the project will be data collection, data cleaning and EDA.
#
# For data analytics project ideation, I generally have two approaches. <br> <ul>I can either look first at what data is available and then do a project on what I find.</ul>  or<ul> I can start with the problem that I want to solve and then go out and collect or find the relevant data. </ul> <br>
# For the first style one can basically go through kaggle data sets or Google data sets and find something that's interesting or trending. One can generally just download these data sets and can skip the data collection phase.<br> <br>The second way of finding a project is finding our own data for a problem that we're interested in is a more interesting approach in general as well as also fascinatig for employers. <br>For the intents and purposes of this project I'd like to go in and collect my own data. I want to come up with my own problem to solve and I also want to scrape my own data.

# Let's go through a kind of brainstorming process for data science project. I always prefer doing project that I'm really interested in and which is more informative in learning point of view and in terms of the subject matter or that a potential audience is interested in like a pandemic is going on and I could try and understand that better, but right now I don't really want to do it on the pandemic because i already have done a SQL project on COVID-19 data to recall my SQL skills and now in order to bring diversity in my portfolio i would like to pick a project from some differet domain. 

# So I come up with an idea which is relevant to me (as a job seeker) in a way that the data science positions and roles are different across the country and that's generally reflected in salary range too. So maybe I could try and predict the salary of data science positions based on the lets say job title or like some info about the company and geography. I hope That's something that probably shouldn't be too hard to find the relevant data and that would be meaningful because if I could probably build a cool project or a product later on where I plug in that information and it shows up maybe a range of salaries so working on this project may not only enhance my analytics skills but may also help me to negotiate my salary in interviews.<br> So this was my whiteboard chicken scratch fro ideation and I've now got a project idea so in implementing this project idea I next have to figure out what data I want to collect I know that there's probably some salary data and some company data on LinkedIn and Glassdoor so those are probably the places I am gonna look first to try and collect data. So, In my next section I'll look in how to actually scrape that data and I'll try and implement it

# ## Data Science Skillset and Salary overview:

# As I had decided to do a project related to the data science field where I want to look at different skillsets and try to predict the salary of a position based on some of the factors associated with the job. I'll be looking for this data on LinkedIn and glassdoor.com

# let's take a quick look at these two websites and see what data is available there so I've pulled up some job descriptions for data science roles as you can see up here on LinkedIn and on Glassdoor

# <div style="text-align: left"> glassdoor</div>  | linkedin
# - | - 
# ![alt](glassdoor.gif) | ![alt](linkedin.gif)

# So on linkedin It can be seen that some of the information is pretty readily accessible. If we want to see where it's located for some scraping tools we can inspect the page and see that it's located in the span. If we take a look at Glassdoor it seems like they use the similar web architecture like linkedin but they do have some estimated salary data which is what we're going for. <br>Its very important to know that it's generally not a great practice to do predictions or use a dependent variable that is already an estimate but right now this is probably gonna be the best we can get here.<br> So right now let's try and  make our own Glassdoor prediction engine using some of the information that we have here. Let's go through and figure out how we're gonna scrape some of this data.

# ## Web Scraping

# <b>There are generally two approaches to scraping data online.</b> <ul>One can either go through and use beautiful soup which takes all of the HTML on a page and organizes it and lets you pull some of the elements</ul> <ul>or</ul> <ul>One can use selenium which is basically a bot that goes through and clicks on elements of the page and copies them into a data frame.<br> As we see both these websites use JavaScript so probably I want to go with the selenium approach. At first I'm gonna Google glassdoor scraper if someone has already written a selenium code for scraping glassdoor data then my job will be a lot more easier.<br> Here I find a glassdoor scraper in selenium that you can do in 10 minutes by [<NAME>ya](https://towardsdatascience.com/selenium-tutorial-scraping-glassdoor-com-in-10-minutes-3d0915c6d905) and this looks like a lot of the data that I really want right here and as we know there's no problem in data science when we use someone else's code we just want to make sure that they're credited for their work. Usually one can fork their code on github but i'll be copying the python code and will be adding the original link on top of the code as its more convenient for me. 

# I have downloaded the file and renamed it as glassdoor_scraper.py 

"""

original author: Kenarapfaik
url: https://github.com/arapfaik/scraping-glassdoor-selenium
"""
from selenium.common.exceptions import NoSuchElementException, ElementClickInterceptedException
from selenium import webdriver
import time
import pandas as pd
import pandas as pd
import glassdoor_scraper as gs
path = 'C:/Users/toqee/chromdriver'
df1 = gs.get_jobs('datascientist', 2, False, path, 2)
df1.to_csv('glassdoor_jobs.csv')

# Now that I have got this code in my notebook, There are some necessary changes such as checking the correct version of chrome driver and the path inside the code. Also I have to adjust the sleep time according to my own internet speed otherwise the browser will close too early and scraped data could be empty. This code already included "bypassing login screen" option as well as "accept cookies" options which made my work a lot more easier.

# So with the help of glassdoor_scraper by Kenarapfaik i was able to scrape 1000 job postings from glassdoor.com. With each job, I got the following:
#
# <ul><b>Job title <br>
# Salary Estimate <br>
# Job Description<br>
# Rating<br>
# Company<br>
# Location<br>
# Company Headquarters<br>
# Company Size<br>
# Company Founded Date<br>
# Type of Ownership<br>
# Industry<br>
# Sector<br>
# Revenue<br>
# Competitors<br> </ul>

# ## Data Cleaning

df=pd.read_csv('glassdoor_jobs.csv')
df.head()

# Here we go, this is the data that was obtained and it can be seen that there's a lot of cleaning required to process this data for EDA. Let's take a look at head and tail of the data and point out what kind of cleaning can be done to make this data more meaningful.
#
# <ul><b>
# Removing null values and rows without salary<br>
#     Parsing numeric data out of salary<br>
#     Making an employer column and providing salary and hourly wages<br>
#     Parsing rating out of company column<br>
#     Making a new column for company state<br>
#     Adding a column for if the job was at the company’s headquarters<br>
#     Transforming the founded date into age of company<br>
#     Making columns for if different skills were listed in the job description like Python, R, Excel, AWS, Spark<br>
#     Making Column for simplified job title and Seniority<br>
#     Creating Column for description length<br></b>

# ### Parsing Salary

# +
df['hourly'] = df['Salary Estimate'].apply(lambda x: 1 if 'per hour' in x.lower() else 0)
df['employer_provided'] = df['Salary Estimate'].apply(lambda x: 1 if 'employer provided salary:' in x.lower() else 0)

df = df[df['Salary Estimate'] != '-1']
salary = df['Salary Estimate'].apply(lambda x: x.split('(')[0])
minus_Kd = salary.apply(lambda x: x.replace('K','').replace('$',''))

min_hr = minus_Kd.apply(lambda x: x.lower().replace('per hour','').replace('employer provided salary:',''))

df['min_salary'] = min_hr.apply(lambda x: int(x.split('-')[0]))
df['max_salary'] = min_hr.apply(lambda x: int(x.split('-')[1]))
df['avg_salary'] = (df.min_salary+df.max_salary)/2
# -

# ### Company Name text Only

df['company_txt'] = df.apply(lambda x: x['Company Name'] if x['Rating'] <0 else x['Company Name'][:-3], axis = 1)

# ### State field 

df['job_state'] = df['Location'].apply(lambda x: x.split(',')[1])
df.job_state.value_counts()
df['same_state'] = df.apply(lambda x: 1 if x.Location == x.Headquarters else 0, axis = 1)

# ### age of company 
#

df['age'] = df.Founded.apply(lambda x: x if x <1 else 2020 - x)


# ### Parsing of job description (python, etc.)
#

# +
#python
df['python_yn'] = df['Job Description'].apply(lambda x: 1 if 'python' in x.lower() else 0)
 
#r studio 
df['R_yn'] = df['Job Description'].apply(lambda x: 1 if 'r studio' in x.lower() or 'r-studio' in x.lower() else 0)
df.R_yn.value_counts()

#spark 
df['spark'] = df['Job Description'].apply(lambda x: 1 if 'spark' in x.lower() else 0)
df.spark.value_counts()

#aws 
df['aws'] = df['Job Description'].apply(lambda x: 1 if 'aws' in x.lower() else 0)
df.aws.value_counts()

#excel
df['excel'] = df['Job Description'].apply(lambda x: 1 if 'excel' in x.lower() else 0)
df.excel.value_counts()

# +
df.columns

df_out = df.drop(['Unnamed: 0'], axis =1)

df_out.to_csv('salary_data_cleaned.csv',index = False)

df_out.head()
# -

# So this is the cleaned data which is ready for EDA and model building

# ## Exploratory Data Analysis

# EDA is the step where we really start understanding our data and where we start analyzing trends and begin to find insights. First, I'll be doing some additional data cleaning. I'll clean up the job titles, and do some feature engineering. Next, I'll look at histograms and box plots of the continuous data fields. This step is important if we want normal data to do a regression analysis during model building. Then I'll look at the categorical data. I want to see what companies, states, industries, and sectors these jobs are offered in. After that, I'll make some pivot tables to better understand how average salary is different across our categorical categories. Finally, I'll make a word cloud to visualize some of the most common words found in the descriptions. 

import matplotlib.pyplot as plt #matplotlib and seaborn was not imported yet
import seaborn as sns

df_out.columns


# +
def title_simplifier(title):
    if 'data scientist' in title.lower():
        return 'data scientist'
    elif 'data engineer' in title.lower():
        return 'data engineer'
    elif 'analyst' in title.lower():
        return 'analyst'
    elif 'machine learning' in title.lower():
        return 'mle'
    elif 'manager' in title.lower():
        return 'manager'
    elif 'director' in title.lower():
        return 'director'
    else:
        return 'na'
    
def seniority(title):
    if 'sr' in title.lower() or 'senior' in title.lower() or 'sr' in title.lower() or 'lead' in title.lower() or 'principal' in title.lower():
            return 'senior'
    elif 'jr' in title.lower() or 'jr.' in title.lower():
        return 'jr'
    else:
        return 'na'
		
## Job title and seniority 
		
##  Fix state Los Angeles 

##  Job description length 

##  Competitor count

## hourly wage to annual 

#remove new line from job title


# -

df_out['job_simp'] = df_out['Job Title'].apply(title_simplifier)

df_out.job_simp.value_counts()

df_out['seniority'] = df_out['Job Title'].apply(seniority)
df_out.seniority.value_counts()

# Fix state Los Angeles 
df_out['job_state']= df_out.job_state.apply(lambda x: x.strip() if x.strip().lower() != 'los angeles' else 'CA')
df_out.job_state.value_counts()


#  Job description length 
df_out['desc_len'] = df_out['Job Description'].apply(lambda x: len(x))
df_out['desc_len']

#Competitor count
df_out['num_comp'] = df_out['Competitors'].apply(lambda x: len(x.split(',')) if x != '-1' else 0)

df_out['Competitors']

# +
#hourly wage to annual 

df_out['min_salary'] = df_out.apply(lambda x: x.min_salary*2 if x.hourly ==1 else x.min_salary, axis =1)
df_out['max_salary'] = df_out.apply(lambda x: x.max_salary*2 if x.hourly ==1 else x.max_salary, axis =1)
# -

df_out[df_out.hourly ==1][['hourly','min_salary','max_salary']]

df_out.describe()

df_out.Rating.hist()

df_out.desc_len.hist()

df_out.avg_salary.hist()

df_out.age.hist()

df_out.boxplot(column = ['age','avg_salary','Rating'])

df_out.boxplot(column = 'Rating')

df_out[['age','avg_salary','Rating','desc_len']].corr()

cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(df_out[['age','avg_salary','Rating','num_comp']].corr(),vmax=.3, center=0, cmap=cmap,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})

df_out.columns

pd.pivot_table(df_out, index = 'job_simp', values = 'avg_salary')

pd.pivot_table(df_out, index = ['job_simp','seniority'], values = 'avg_salary')


pd.pivot_table(df_out, index = ['job_state','job_simp'], values = 'avg_salary').sort_values('job_state', ascending = False)

pd.options.display.max_rows
pd.set_option('display.max_rows', None)

pd.pivot_table(df_out, index = ['job_state','job_simp'], values = 'avg_salary', aggfunc = 'count').sort_values('job_state', ascending = False)

pd.pivot_table(df_out[df_out.job_simp == 'data scientist'], index = 'job_state', values = 'avg_salary').sort_values('avg_salary', ascending = False)

from wordcloud import WordCloud, ImageColorGenerator, STOPWORDS
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# +
#nltk.download()

# +
words = " ".join(df_out['Job Description'])

def punctuation_stop(text):
    """remove punctuation and stop words"""
    filtered = []
    stop_words = set(stopwords.words('english'))
    word_tokens = word_tokenize(text)
    for w in word_tokens:
        if w not in stop_words and w.isalpha():
            filtered.append(w.lower())
    return filtered


words_filtered = punctuation_stop(words)

text = " ".join([ele for ele in words_filtered])

wc= WordCloud(background_color="white", random_state=1,stopwords=STOPWORDS, max_words = 2000, width =800, height = 1500)
wc.generate(text)

plt.figure(figsize=[10,10])
plt.imshow(wc,interpolation="bilinear")
plt.axis('off')
plt.show()
# -

# ## Acknowledgments
#
# * [<NAME>](https://www.youtube.com/channel/UCiT9RITQ9PW6BhXK0y2jaeg) - Data Science Tutorial and Projects.
# * [Kaggle](https://www.kaggle.com/) - Courses.
# * [Glassdoor](https://www.glassdoor.co.uk/index.htm) - Job data.
# * [DS tools](https://data-flair.training/blogs/data-science-tools/) - Essential Data Science Ingredients

#

StarcoderdataJupyterScriptsDedupFiltered