# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 2
#     language: python
#     name: python2
# ---

# # Project 4: Web Scrapping Job Postings
#
# ## Background of the project
# You're working as a data scientist for a contracting firm that's rapidly expanding. Now that they have their most valuable employee (you!), they need to leverage data to win more contracts. Your firm offers technology and scientific solutions and wants to be competitive in the hiring market. Your principal has two main objectives:
#
# 1. Determine the industry factors that are most important in predicting the salary amounts for these data.
# 2. Determine the factors that distinguish job categories and titles from each other. For example, can required skills accurately predict job title?
#
# ## Introduction
# The hiring agency, which is currently expanding, wants to remain competitive in the job market so as to be able to win more contracts. This is only made possible by making use of current data and technological and scientific solutions. In accomplishing the goals of getting more contracts, we need to build models that can help determine the salary and the job categories and titles.
#
# ## Materials and Methods
#
# ### Dataset
# The dataset has to be obained from the jobs website - myfuturecareer.sg - which have information of salary range, seniority, job category along with text of the job description. As the job website is managed by the government, there exist sepecifications of the features that are required for the job post in the website. This includes requiring the publication of salary range and having the salary range to be limited to 1.5 to 2 times the minimum salary offered for the job.
# The code which is used to scrape the website and generate the final data set is in the appendix as program listing.
#
# ### Modelling
# The target variable are variable derived from running the models using the selected feature from the dataset. The main features are the job description which is a combination of both responsibility and requirements. The main feature is vectorized using CountVectors and only the token with description of number of years (of experience) and tokens with more than 2 words (that have at least three characters and exclude numbers) are selected, as they are considered significant with respect to the target variable. Two classification models are build for each target variable - using Random Forest and Naive Bayes. 
#
# ### Models to predict salary
# A salary category - the target variable - is given to each median salary (determined by the average of minimum salary and maximum salary) in order to build a classification model. Only jobs with salary category of between 1 and 4 (median salary of between \$2000 and \$10000) are included as the dataset for building the model. 
#
# In addition, to satisfy the requirement of having to predict the salary incorrectly with a lower salary instead of a higher salary, a new model will be built based on the two models built for salary category prediction. The target variable to be used will be salary categories based on the minimum salary while the feature to be used for prediction is job description.
#
# ### Models to predict job category
# The models built will predict the seniority of the position. However, given the variance in the description of seniority in each job post, a seniority class (Junior, Middle & Senior) will be given only to seniority description that do not have a mixture of seniority (such as having all three seniority classes in a single job post).

# +
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix, classification_report

from sklearn.externals.six import StringIO  
from IPython.display import Image  
from sklearn.tree import export_graphviz
import pydotplus
# -

# ## Pre-modelling - Feature Engineering and Analysis

# Load the scrapped data from the csv file.
csv_jobs = './jobs_set.csv'
df_jobs = pd.read_csv(csv_jobs)


# ### Feature engineering of salary
# Based on salary, the salary category is assigned in blocks of \$2000 from \$2000 to \$10000. Salary of less than \$2000 are category 0 while salary of more than or equal to \$10000 are category 5.

# Assign salary category for each salary amount
def get_salary_category(salary):
    if (salary < 2000):
        return 0
    elif (salary < 10000):
        return int((salary - 2000)/2000) + 1
    else:
        return 5


df_jobs['median_salary'] = (df_jobs['min_salary'] + df_jobs['max_salary'])/2
df_jobs['salary_category'] = df_jobs['median_salary'].map(get_salary_category)
df_jobs['min_salary_category'] = df_jobs['min_salary'].map(get_salary_category)

df_jobs['description'] = df_jobs['responsibilities'].str.cat(df_jobs['requirements'], sep='\n')

df_jobs.head()

# ### Initial Data Analysis of Seniority and Job Category
#
# Before carrying out any feature engineering involving seniority, I seek to understand the statistics of seniority and job category labels.

df_jobs['job_categories'].value_counts()

# Understand seniority category 
df_jobs['seniority'].value_counts()

# From the seniority labels, a small amount of job descriptions have a mix of junior, middle and senior positions. As such the seniority of the position cannot be determined and are excluded from the modelling to predict the seniority of the position.

# ### Feature engineering of seniority

# +
# Assign seniority class based on seniority provided there is no ambiguity.
list_junior = ['Junior Executive', 'Fresh/entry level', 'Executive, Junior Executive', 
               'Fresh/entry level, Executive', 'Fresh/entry level, Non-executive',
               'Fresh/entry level, Junior Executive', 'Fresh/entry level, Professional',
               'Executive, Non-executive, Junior Executive',
               'Fresh/entry level, Executive, Non-executive',
               'Fresh/entry level, Professional, Executive, Non-executive',
               'Manager, Fresh/entry level, Executive, Junior Executive',
              'Manager, Fresh/entry level, Executive, Non-executive', 
               'Fresh/entry level, Professional, Non-executive',
              'Manager, Fresh/entry level, Professional, Executive']
list_middle = ['Professional', 'Executive', 'Non-Executive', 'Manager', 'Middle Management', 'Professional, Executive',
               'Middle Management, Professional', 'Manager, Executive', 'Professional, Non-executive',
               'Executive, Non-executive', 'Middle Management, Executive', 'Senior Management, Manager',
               'Manager, Professional', 'Manager, Professional, Executive']
list_senior = ['Senior Executive', 'Executive, Senior Executive', 'Senior Management', 
               'Professional, Senior Executive', 'Manager, Professional, Senior Executive',
               'Manager, Executive, Senior Executive', 'Manager, Professional, Executive, Senior Executive',
              'Senior Management, Manager, Executive']
def get_seniority_class(seniority):
    seniority = seniority.strip()
    return "Junior" if seniority in list_junior else "Middle" if seniority in list_middle else "Senior" if seniority in list_senior else ""

df_jobs['seniority_class'] = df_jobs['seniority'].map(get_seniority_class)
# -

df_jobs[df_jobs['seniority'] == 'Senior Management'].head()

# Check the statistics of seniority classes.
df_jobs['seniority_class'].value_counts()

# Understand the salary range in each salary category, especially at salary category 0 and category 5.
plt.figure(figsize=(12, 10))
sns.boxplot(x='salary_category', y='median_salary', data=df_jobs)
plt.show()

# It can be seen that for salary category 5, the salary range is relatively high and outliers are existent. In addition, there are little quantative features that can be used to predict the salary using regression.

# Visualize the salary range in each category.
plt.figure(figsize=(12, 10))
sns.boxplot(x='salary_category', y='median_salary', data=df_jobs[df_jobs['salary_category'] != 5])
plt.show()

# Check the statistics of the salary category.
df_jobs['salary_category'].value_counts()

# The salary of the jobs in salary category of 1 to 4 (where the jobs provide median salary of between \$2000 and \$10000 per month) are distributed normally. There are too little samples from salary category 0 as compared to other salary categories.

# ## Factors that impact salary - Results

# +
# There are to few jobs with salary category of 0 and the median salary range for jobs in salary category of 5 is too
# wide. Therefore we will exclude it from the modelling in this section.

df_jobs_selected = df_jobs[(df_jobs['salary_category'] != 0) & (df_jobs['salary_category'] != 5)].copy()
# -

X1 = df_jobs_selected['description']
y1 = df_jobs_selected['salary_category']

# Split the dataset between training and test set.
X1_train, X1_test, y1_train, y1_test = train_test_split(X1, y1, test_size=0.25, random_state=45)

# Base accuracy values for test set
y1_test.value_counts()/len(y1_test)

# Preprocessing
# As the feature selected is text, it has to be vectorized before fitting the model.
# Significant tokens includes the number of years of experiences and words of more than three characters.
# Numbers itself does not have any meaning.
cV_1a = CountVectorizer(stop_words='english', strip_accents='unicode', analyzer='word', token_pattern='\d+ years|[A-Za-z]{3,}', ngram_range=(2, 5))

# +
# Modelling using Random Forest Classifier
rfC_1a = RandomForestClassifier(n_estimators=17, max_features=200)

pipeline_1a = Pipeline([
    ('vect', cV_1a),
    ('rf', rfC_1a)
])

pipeline_1a.fit(X1_train, y1_train)

# Initial validation of test set on the model using Random Forest.
pipeline_1a.score(X1_test, y1_test)
# -

# Understand the accuracy of the model using confusion matrix
cF1 = confusion_matrix(y1_test, pipeline_1a.predict(X1_test), labels=range(1, 5))
pd.DataFrame(cF1, index=range(1, 5), columns=['Predicted {}'.format(idx) for idx in range(1,5)])

print(classification_report(y1_test, pipeline_1a.predict(X1_test)))

# Understand the tokens that are used to build the trees.
list_features = [{'token': token, 'score': score} for (token, score) in zip(cV_1a.get_feature_names(), rfC_1a.feature_importances_) if score > 0]
df_features = pd.DataFrame(list_features)
df_features.sort_values(by = 'score', inplace=True)

df_features.head(20)

# +
# Modelling using Naive Bayes

nbC_1a = MultinomialNB(fit_prior=True)
pipeline_1a_nb = Pipeline([
    ('vect', cV_1a),
    ('nb', nbC_1a)
])
# -

# Carry out three-fold cross validation for scoring
cross_val_score(pipeline_1a_nb, X1_train, y1_train, cv=3)

pipeline_1a_nb.fit(X1_train, y1_train)

# Initial accuracy score on the testing set by model using Naive Bayes 
pipeline_1a_nb.score(X1_test, y1_test)

# Understand the accuracy of the model using confusion matrix
cF1_NB = confusion_matrix(y1_test, pipeline_1a_nb.predict(X1_test), labels=range(1, 5))
pd.DataFrame(cF1_NB, index=range(1,5), columns=['Predicted {}'.format(idx) for idx in range(1,5)])

print(classification_report(y1_test, pipeline_1a_nb.predict(X1_test)))

# Determine the top 10 most common tokens found for each salary category, in the order of 1 to 4.
list_token_score = [[{'token': token, 'score':score} for (token, score) in zip(cV_1a.get_feature_names(), list_coef) if score != 0] for list_coef in nbC_1a.coef_]
df_token_score = [pd.DataFrame(list_category_token_score) for list_category_token_score in list_token_score]
df_token_score_sorted = [df.sort_values(by='score', ascending=False, inplace=False) for df in df_token_score]

for category in range(0, 4):
    print(df_token_score_sorted[category].head(10))
    print('')

# Between the model using Random Forest and that using Naive Bayes, the model using Naive Bayes has a better recall for most of the salary category (except for category 2), and have a higher accuracy of prediction of salary category than that of Random Forest. However, the model using Naive Bayes have a greater tendacy of predicting higher salary category than that of Random Forest.
#
# I seek to understand the most significant tokens that are used in building the models by feature importance scores in Random Forest and coefficients in Naive Bayes. For Random Forest the tokens that are considered significant for decision trees have a mix of description of responsibilities, requirements of knowledge and number of years of experience. For Naive Bayes, "Machine Learning" is the most significant token for higher salary category while "Computer Science" is the most significant token for lower salary category. 

# ## Factors that impact job category - Results

# Exclude the job posts that do not have seniority class from the training and testing set, as they have varying 
# levels of seniority in the same job post.
df_jobs_selected_2 = df_jobs[df_jobs['seniority_class'] != ''].copy()

X2 = df_jobs_selected_2['description']
y2 = df_jobs_selected_2['seniority_class']

X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.25, random_state=45)

# Base accuracy on the testing set
y2_test.value_counts()/len(y2_test)

# Preprocessing
# As the feature selected is text, it has to be vectorized before fitting the model.
# Significant tokens includes the number of years of experiences and words of at least three characters.
# Numbers itself does not have any meaning.
cV_2a = CountVectorizer(stop_words='english', strip_accents='unicode', analyzer='word', token_pattern='\d+ years|[A-Za-z]{3,}', ngram_range=(2, 5))

# Modelling using Random Forest Classifier
rfC_2a = RandomForestClassifier(n_estimators=17, max_features=50)
pipeline_2a = Pipeline([
    ('vect', cV_2a),
    ('nb', rfC_2a)
])

# +
pipeline_2a.fit(X2_train, y2_train)

# Carry out initial validation using basic scoring
pipeline_2a.score(X2_test, y2_test)
# -

# Validate accuracy of the model using confusion matrix and classification report.
cF2a = confusion_matrix(y2_test, pipeline_2a.predict(X2_test), labels=["Junior", "Middle", "Senior"])
pd.DataFrame(cF2a, index=["Junior", "Middle", "Senior"], columns=['Predicted {}'.format(lbl) for lbl in ["Junior", "Middle", "Senior"]])

print(classification_report(y2_test, pipeline_2a.predict(X2_test)))

# Understand the tokens that are used to build the trees.
list_features_2 = [{'token': token, 'score': score} for (token, score) in zip(cV_2a.get_feature_names(), rfC_2a.feature_importances_) if score > 0]
df_features_2 = pd.DataFrame(list_features_2)
df_features_2.sort_values(by = 'score', inplace=True)

df_features_2.head(20)

# Modelling using Naive Bayes
nbC_2 = MultinomialNB()
pipeline_2_nb = Pipeline([
    ('vect', cV_2a),
    ('nb', nbC_2)
])

# Carry out three-fold cross validation
cross_val_score(pipeline_2_nb, X2_train, y2_train, cv=3)

# +
pipeline_2_nb.fit(X2_train, y2_train)

# Validate model using basic scoring on test set.
pipeline_2_nb.score(X2_test, y2_test)
# -

# Validate accuracy of the model using confusion matrix and classification report.
cF2b = confusion_matrix(y2_test, pipeline_2_nb.predict(X2_test), labels=["Junior", "Middle", "Senior"])
pd.DataFrame(cF2b, index=["Junior", "Middle", "Senior"], columns=['Predicted {}'.format(lbl) for lbl in ["Junior", "Middle", "Senior"]])

print(classification_report(y2_test, pipeline_2_nb.predict(X2_test)))

# Determine the top 10 most common tokens that can be found in each seniority class, in the order of ["Junior", "Middle", "Senior"]
list_token_score_2 = [[{'token': token, 'score':score} for (token, score) in zip(cV_2a.get_feature_names(), list_coef) if score != 0] for list_coef in nbC_2.coef_]
df_token_score_2 = [pd.DataFrame(list_category_token_score) for list_category_token_score in list_token_score_2]
df_token_score_sorted_2 = [df.sort_values(by='score', ascending=False, inplace=False) for df in df_token_score_2]

for category in range(0, 3):
    print(df_token_score_sorted[category].head(10))
    print('')

# None of the models have accuracy of more than 0.5 on the test set. The model using Naive Bayes has a higher recall across most classes of seniority than that of Random Forest Classifier. It is able to give a slightly more accurate prediction on jobs in Junior and Senior class than that of Random Forest.
#
# For interpretion of the models, large amount of tokens describing the scope of the job make up the most significant features in building model using Random Forest while when using Naive Bayes, the classification can be highly influenced by single word tokens. In the Senior class, the more significant tokens are "machine learning" and "communication skills" whereas "computer science" is relatively more significant in the Junior and Middle class.

# ## Model for incorrect prediction with lower salary category - Result
# In order to get incorrect prediction with lower salary category, I use the model that have greater tendency of predicting lower salary category and then tweaking it. In addition, the predictor will be based on minimum salary, so that the jobs can be classified at a lower salary category.

# +
# Only jobs with minimum salary category of between 0 and 4 (Minimum salary of between $2000 and $10000) are included.
df_jobs_selected_b = df_jobs[(df_jobs['min_salary_category'] != 0) & (df_jobs['min_salary_category'] != 5)].copy()

# Define X and y
Xb = df_jobs_selected_b['description']
yb = df_jobs_selected_b['min_salary_category']
# -

# Do a train test split
Xb_train, Xb_test, yb_train, yb_test = train_test_split(Xb, yb, test_size=0.25, random_state=45)

# +
# Preprocessing with count vectorizer
cV_b = CountVectorizer(stop_words='english', strip_accents='unicode', analyzer='word', token_pattern='\d+ years|[A-Za-z]{3,}', ngram_range=(1, 5))

# Building a model using Random Forest.
rfC_b = RandomForestClassifier(n_estimators=3, max_leaf_nodes=4, max_depth=4, max_features=100)

pipeline_b = Pipeline([
    ('vect', cV_b),
    ('rf', rfC_b)
])

pipeline_b.fit(X1_train, y1_train)
pipeline_b.score(X1_test, y1_test)
# -

# Verify accuracy of the model using confusion matrix.
cF_b = confusion_matrix(yb_test, pipeline_b.predict(Xb_test), labels=range(1, 5))
pd.DataFrame(cF_b, index=range(1, 5), columns=['Predicted {}'.format(idx) for idx in range(1,5)])

print(classification_report(yb_test, pipeline_b.predict(Xb_test)))

# +
from sklearn.metrics import roc_curve

def plot_roc(y, probs, threshmarkers=None):
    fpr, tpr, thresh = roc_curve(y, probs)

    plt.figure(figsize=(8,8))
    plt.plot(fpr, tpr, lw=2)
   
    plt.xlabel("False Positive Rate\n(1 - Specificity)")
    plt.ylabel("True Positive Rate\n(Sensitivity)")
    plt.xlim([-0.025, 1.025])
    plt.ylim([-0.025, 1.025])
    plt.xticks(np.linspace(0, 1, 21), rotation=45)
    plt.yticks(np.linspace(0, 1, 21))
    plt.show()


# -

# Plot the roc curve for this model.
predicted_proba = pipeline_b.predict_proba(Xb_test)
plot_roc(yb_test.map(lambda x: x == 2), predicted_proba[:, 1])

# The result indicates that the model has greater tendency of predicting a job with salary category 2. Whilst most of the jobs which are in the higher salary category are predicted to be of lower salary category, most of the jobs with salary category 1 will be wrongly predicted with higher salary category.
#
# In addition, tweaking the model will reduce the accuracy of the model as most of the jobs will be predicted at a lower salary category.

# ## Discussion
#
# ### Factors that impact salary
# The results indicates that using Naive Bayes is a better model to classify salary. Prediction of salary category is not significantly influenced by number of years of experience. In addition, given the lower accuracy rate for both models used to predict salary category, there is a concern that the models here may not be highly reliable for predicting salary.
#
# Experience is not included as a separate feature in building the model to predicting salary as they are written in the requirements and hence require extraction from the text. In addition, experience is the only quantitive feature in predicting the salary and it's not the only feature that can influence salary. Other factors include the scope of responsibilites, education level, level of responsibilities of the position and the demand and supply of talents in the industry, especially when the hiring market is the free market. More importantly, the salary amount also depends on the company's ability to offer the pay listed on the job description, which can be determined by the size and the finanical strength of the company. Unfortunately, it is impossible to determine the size and financial strength from the job description alone. 
#
# ### Factors that distingush job category
# The model built using Naive Bayes to classify seniority of the position reflects the differing job scope at different levels of seniority. For instance, jobs under "Senior" class may involve management as the word is relatively more common in the "Senior" class than in other classes, while the word "business" is relative less common in the "Junior" class, indicating that people in Junior job positions may not be as involved in business as compared to those in more Senior positions. The model built using Random Forest reflects a lot of the relationship between job scope and responsibilities and the seniority class. 
#
# However, what is not done here is to use job title as target variable and the job description as feature in building a new model. Although it can be done using topic modelling, there are huge variations in the job titles, especially in between companies - where different companies give different job titles for the same job scope and responsibilty. Also, there is no model to predict job category label as most of the labels are "Information Technology" especially when all of the jobs in this dataset are considered tech-jobs and not every description (responsibilities and requirements) contains information on the description of the line of business. Absence of important features will make the model relative inaccurate and unreliable.  

# ## Appendix: Web scrapping program listing
# This program listing details the python code that is used to scrape job information from the website and save them as a single dataset in one csv file.

# + active=""
# import os
# from __future__ import print_function
# from IPython.display import clear_output
# from selenium import webdriver
# import requests
# from bs4 import BeautifulSoup
# from time import sleep
# import pandas as pd
# import numpy as np
# import sys
# import string
#
# chromedriver = "./chromedriver/chromedriver"
# os.environ["webdriver.chrome.driver"] = chromedriver
#
# # Extractor to extract links to job description from job post.
# class JobPostListExtractor(object):
#     def __init__(self, html):
#         self._dom = BeautifulSoup(html, 'lxml')
#         self._base_url = "https://www.mycareersfuture.sg"
#         
#     def extract_list(self):
#         list_result = []
#         nodes = self._dom.select('div.card.relative')
#         nodes_with_salary = [node for node in nodes if len(node.select('span.salary_range span.light-silver')) == 0]
#         for node in nodes_with_salary:
#             list_result.append({
#                 "link": self._base_url + node.select("a.JobCard__card___22xP3")[0]['href'],
#                 "location": "" if len(node.select('p[name="location"]')) == 0 else node.select('p[name="location"]')[0].text
#             })
#         return list_result
#     
#     jobPostList_ = property(extract_list, None)
#
# # Extractor to extract information from job description.
# class JobDescriptionExtractor(object):
#     def __init__(self, html):
#         self._dom = BeautifulSoup(html, 'lxml')
#     
#     def extract(self):
#         salary_val = lambda s: s.replace('to', '').replace('$', '').replace(',', '')
#         salary =[int(salary_val(node.text)) for node in self._dom.select('span.salary_range span.dib')]
#         return {
#             "title": self._dom.select("h1#job_title")[0].text,
#             "company": self._dom.select("p#company")[0].text,
#             "seniority": "" if len(self._dom.select('p#seniority')) == 0 else self._dom.select('p#seniority')[0].text,
#             "job_categories": "" if len(self._dom.select('p#job-categories')) == 0 else self._dom.select('p#job-categories')[0].text,
#             "min_salary": salary[0],
#             "max_salary": salary[1],
#             "job_description": self._dom.select('div#job_description div#content')[0].text,
#             "requirements": self._dom.select('div#requirements div#content')[0].text
#         }
#
# # Define class that does the web scrapping from website 
# class JobPostWebClient(object):
#     def __init__(self):
#         self._driver = webdriver.Chrome(executable_path="./chromedriver/chromedriver")
#         self._base_url = "https://www.mycareersfuture.sg/"
#
#     def get_page_count(self, keyword):    
#         self._driver.get(self._base_url + "search?search={searchkey}".format(searchkey=keyword))
#         sleep(6)
#         domresultpage = BeautifulSoup(self._driver.page_source, 'lxml')
#         resultCount = domresultpage.select("span#search-results-message > div")[0].text.split(' ')[0].replace(',', '')
#         return np.ceil(int(resultCount)/20.0).astype(int)
#
#     def scrape_job_links_list(self, keyword, page=0):
#         url = self._base_url + "search?search={}&page={}".format(keyword, page)
#         self._driver.get(url)
#         sleep(6)
#         html = self._driver.page_source
#         return JobPostListExtractor(html).jobPostList_
#         
#     def scrape_job_description(self, url):
#         try:
#             self._driver.get(url)
#             sleep(6)
#             return JobDescriptionExtractor(self._driver.page_source).extract()
#         except:
#             return {}
#         
#     def close(self):
#         self._driver.close()

# + active=""
# # List the keywords that can be used to find the job postings.
# keywords = ['AI Engineer', 'Business Intelligence',
#                'Data Administrator', 'Data Analyst', 'Data Architect',
#               'Data Developer', 'Data Engineer', 'Data Manager',
#               'Data Scientist', 'Database Administrator', 'Machine Learning',
#               'Research Engineer', 'Research Scientist', 'Software Engineer']
# # Get list of job description links and save them as csv.
# client = JobPostWebClient()
# for keyword in keywords:
#     list_postings = []
#     print("Scrapping keyword: {}".format(keyword), file=sys.stderr)
#     lastPage = client.get_page_count(keyword)
#     print("Number of pages: {}".format(lastPage), file=sys.stderr)
#     for page in range(0, lastPage):
#         print("Scrapping page {}.".format(page), file=sys.stderr)
#         list_postings += client.scrape_job_links_list(keyword, page)
#     # Save list of links in csv
#     df_links = pd.DataFrame(list_postings)
#     df_links.to_csv('./links_{}.csv'.format(keyword.replace(' ', '_')))
#     clear_output()
# client.close()

# + active=""
# # List the keywords that can be used to find the job postings.
# jd_keywords = ['AI Engineer', 'Business Intelligence',
#                'Data Administrator', 'Data Analyst', 'Data Architect',
#               'Data Developer', 'Data Engineer', 'Data Manager',
#               'Data Scientist', 'Database Administrator', 'Machine Learning',
#               'Research Engineer', 'Research Scientist', 'Software Engineer']
#               
# # Get job description for each links to job description
# client = JobPostWebClient()
# for jdKeyword in jd_keywords:
#     df_links = pd.read_csv('./links_{}.csv'.format(jdKeyword.replace(" ", "_")))
#     list_jobDescription = []
#     for (index, row) in df_links.iterrows():
#         clear_output()
#         print("Scrapping link {}: {}".format(index, row['link']), file=sys.stderr)
#         jD = client.scrape_job_description(row['link'])
#         if len(jD.keys()) > 0:
#             list_jobDescription.append({
#                 'keyword': jdKeyword,
#                 'url': row['link'],
#                 'title': jD['title'],
#                 'company': jD['company'],
#                 'seniority': jD['seniority'],
#                 'job_categories': jD['job_categories'],
#                 'responsibilities': jD['job_description'],
#                 'requirements': jD['requirements'],
#                 'min_salary': jD['min_salary'],
#                 'max_salary': jD['max_salary']
#             })
#     clear_output()
#     df_jobDescription = pd.DataFrame(list_jobDescription)
#     df_jobDescription.to_csv('./jobs_{}.csv'.format(jdKeyword.replace(" ", "_")), encoding='utf-8')
# client.close()

# + active=""
# # Combine the csv files into one file.
# jd_csv_keywords = ['AI Engineer', 'Business Analyst', 'Business Intelligence',
#                'Data Administrator', 'Data Analyst', 'Data Architect',
#               'Data Developer', 'Data Engineer', 'Data Manager',
#               'Data Scientist', 'Database Administrator', 'Machine Learning',
#               'Research Engineer', 'Research Scientist', 'Software Engineer']
# jd_csv_files = ['./jobs_{}.csv'.format(keyword.replace(' ', '_')) for keyword in jd_csv_keywords]
# dfs_jobs = [pd.read_csv(csv_file) for csv_file in jd_csv_files]
#
# for df in dfs_jobs:
#     df.drop(columns=['Unnamed: 0'], inplace=True)
#
# df_combined.drop_duplicates(subset='url', inplace=True)
# df_combined.reset_index
#
# df_combined.to_csv('./jobs_set.csv', index=False)

StarcoderdataJupyterScriptsDedupFiltered