<jupyter_start><jupyter_text>Today, we will scrape Producthunt with SeleniumSelenium is a tool initially created to automate tests on websites. It is therefore very useful when information is accessible by clicking on links. A button for example is an element on which it is very difficult to obtain the link. Beautifull soup then becomes limited.In this case, use Selenium  Load libraries<jupyter_code>import bs4
import requests
from bs4 import BeautifulSoup
import numpy as np
import pandas as pd
import json
import re
import lxml.html
import time
import random
from random import randint
import logging
import collections
from time import gmtime, strftime

import re
from tabulate import tabulate
import os
date=strftime("%Y-%m-%d")<jupyter_output><empty_output><jupyter_text>Install Selenium according to this manualhttps://selenium-python.readthedocs.io/installation.htmldownloading-python-bindings-for-selenium/binnb: Linux: put your geckodriver (the downloaded extension) in the equivalent path at home to /home/YOURNAME/.local/bin We will simulate a search on the official Phython website<jupyter_code>import selenium

# The selenium.webdriver module provides all the implementations of WebDriver. WebDriver implementations 
# Currently supported are Firefox, Chrome, IE and Remote. The Keys class provides keys in 
# the keyboard such as RETURN, F1, ALT etc.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# Then, the instance of Firefox WebDriver is created.
driver = webdriver.Firefox()
# The driver.get method will lead to a page given by the URL. WebDriver will wait until the page is fully completed 
# loaded (i.e. the "onload" event has been triggered) before returning the control to your script. 
# It should be noted that if your page uses a lot of AJAX when loading, WebDriver may not know 
# when it was fully charged:
driver.get("http://www.python.org")
# The following line is a statement confirming that the title contains the word "Python"
assert "Python" in driver.title
# WebDriver offers several methods to search for items using one of the methods 
# find_element_by_by_ * . For example, the input text element can be located by its name attribute by 
# using the find_element_by_name method 
elem = driver.find_element_by_name("q")
# Then we send keys. This is similar to entering keys using your keyboard. 
# Special keys can be sent using the imported selenium.webdriver.common.keys Keys class. 
# For security reasons, we will delete any pre-filled text in the input field 
# (for example, "Search") so that it does not affect our search results:


elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)

#After submitting the page, you should get the result if there is one. To ensure that certain results 
# are found, make an assertion:
assert "No results found." not in driver.page_source
driver.close()<jupyter_output><empty_output><jupyter_text>Open the source code of this page (HTML) and check that the search area (field) is called "q".input id="id-search-field" name="q" type="search" role="textbox" class="search-field placeholder" placeholder="Search" value="" tabindex="1"  Getting a phone number from an ad in the right place<jupyter_code>from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By

url='https://www.leboncoin.fr/sports_hobbies/1536839557.htm/'

driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.get(url)


python_button = driver.find_elements_by_xpath('//div[@data-reactid="269"]')[0]
python_button.click()

# And then it's like Beautiful soup
soup=BeautifulSoup(driver.page_source)

driver.close()

# And then it's like Beautiful soup
for elem in soup.find_all('a',attrs={"data-qa-id" :"adview_number_phone_contact" }):
    print(elem.text)<jupyter_output><empty_output><jupyter_text>Starting from an ad in the right place, collect all the information available to define the product being sold. Use selenium for the telephone number  API (Application Program Interface) A set of tools and methods that allow different applications to interact with each other. In the case of a web service, we can retrieve data dynamically. By using an API correctly, we can thus obtain in real time, the modifications made on a "parent" site.For example, we will retrieve online news, for example from the "L'équipe" website. Follow the instructions at https://newsapi.org/s/lequipe-api to retrieve an "API key" connection key Your API key is: 73bbb95f8ecb49b499113a46481b4af1It is frequent that a key does not work after a while 5mins 30 mins 1 d ...So don't jump up if you get an error message back.<jupyter_code>import requests
key='<KEY>'
url ='https://newsapi.org/v2/top-headlines?sources=lequipe&apiKey='+key
response = requests.get(url)

# Here the response format is a json file, it is used as a dictionary

print (response.json())
dic=response.json()
print(dic.keys())
for elem in list(dic.keys()):
    print('##############################################')
    print("clé: ",elem,"// values: ", dic[elem])
# And now we have lists in dictionaries it's json but it's always the same thing
# We will discover the information of the article key
for elem in enumerate(dic['articles']):
    print('###############################################')
    print(elem)
# So if we keep going, it gives us another dictionary!
for elem in dic['articles'][0].keys():
    print(' Key : ',elem,'Values : ',dic['articles'][0][elem])<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup