<jupyter_start><jupyter_text>** ** Step 1: DBLP PARSER Designing a DBLP parser for gathering data from AI conferences. (https://dblp.org/xml/)  Imports<jupyter_code>import requests
import pprint
from bs4 import BeautifulSoup
import pandas as pd
from pandas import DataFrame<jupyter_output><empty_output><jupyter_text>1.1 Parse authors names  Parsing URLIn this example the URL from the UAI 2019 conference is taken, but any conference URL from DBLP will work.<jupyter_code>URL = 'https://dblp.org/db/conf/uai/uai2019.html' 

page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')<jupyter_output><empty_output><jupyter_text>Extracting Title<jupyter_code>title = soup.title.string
title = title.replace("dblp: ", "")<jupyter_output><empty_output><jupyter_text>Creating a list with authors names and their DBLP corresponding URLs<jupyter_code>author_names = soup.find_all('span', itemprop='author')

names_list=[]
for names in author_names:
    name = names['itemprop'].split("."), names.text
    url = names.a['href']
    names_list.append([name[1], url])<jupyter_output><empty_output><jupyter_text>Creation of the dataframe<jupyter_code>df = DataFrame( names_list, columns = ['Authors', 'Url'])
df['Conference'] = title

df = df.drop_duplicates()<jupyter_output><empty_output><jupyter_text>1.2 Parse authors Affiliation<jupyter_code>import time

start_time = time.time()
afilliation_list=[]
count = 0

for url in df.Url:
    page_auth = requests.get(url)
    soup_authors = BeautifulSoup(page_auth.content, 'html.parser')
    
    author_affilitation = soup_authors.find('div', class_='hide-body')
    
    for names in author_affilitation:
        try:
            name = names.span['itemprop'].split("."), names.text
            name= name[1]
            name = name.replace("affiliation: ", "")
            afilliation_list.append([name, url])
            count+=1
            if count%50 == 0:
                print('Already parsed:',count,'/',len(df.Url), '--- %s minutes ---' %round((time.time() - start_time)/60, 2))
            
        except:
            afilliation_list.append([ None, url])
            count+=1
            if count%50 ==0:
                print('Already parsed:',count,'/',len(df.Url), '--- %s minutes ---' %round((time.time() - start_time)/60, 2))
df_affiliation = DataFrame( afilliation_list, columns = ['Affiliation', 'Url'])
df_affiliation[8:13]
print('Total number of entries:', len(df_affiliation))
print('Number of Null Affiliations:', len(df_affiliation[df_affiliation['Affiliation'].isnull()]))<jupyter_output>Total number of entries: 417
Number of Null Affiliations: 346<jupyter_text>Merge DatAffiliationframes<jupyter_code>df_complete = df.merge(df_affiliation, how='left', on='Url')<jupyter_output><empty_output>