<jupyter_start><jupyter_text>In this blog post, I will demonstrate webscraping using the scrapy package. We will be taking all of the actors of my favorite TV show, Narcos: Mexico, and attempt to recommend shows based on how many actors from the orginal show the new shows share.   WebscrapingWebscraping is a powerful tool that makes it easy to 'scrape' information off of a website in which the data that you want is not in a format that is easy to recieve. I will demostrate my method of recieving the shows that all the actors of Narcos: Mexico have been on, and visualizing that in an attractive plot.  The parse Function The parse function will utilize the starting url (in our case it is the home page for Narcos: Mexico on IMBD). It will then navigate to the Cast & Crew page so we can see every actor that appeared in this series. The comments in the function will walk through this in more detail.<jupyter_code>def parse(self, response):
    '''
    Redirect scraper to webpage of full cast and crew of desired TV show 
    (under assumption you start on the TV Show's main IMDB page)

    @param self
    @param response: represents the website
    @yield: redirection to full cast and crew site
    '''
    
    #find where there is a hyperlink on the subgroup titled "Top Cast"
    topCst = response.css("div.ipc-title__wrapper a").attrib["href"] 

    #if it exists, take that link embedded in the hyperlink and
    #redirect scraper there
    if topCst:
        topCst = response.urljoin(topCst)

        yield scrapy.Request(topCst, callback = self.parse_full_credits)<jupyter_output><empty_output><jupyter_text>The parse_full_credits Function The parse_full_credits function works only if it is called on the Cast & Crew page. It will go through the page and redirect through to each actor's main page. The comments in the function will walk through this in more detail.<jupyter_code>def parse_full_credits(self, response):
        '''
        Redirect sraper to actor's main webpage of each actor selected

        @param self
        @param response: represents the website
        @yield: redirection to actor main webpage
        '''

        #put actor name/ hyperlink in a list for all actors in the full cast and crew
        actLst = [a.attrib["href"] for a in response.css("td.primary_photo a")]

        #redirect to the actor page for each actor in the list
        for actor in actLst:
            actrPg = response.urljoin(actor)
            yield scrapy.Request(actrPg, callback = self.parse_actor_page)<jupyter_output><empty_output><jupyter_text>The parse_actor_page Function The parse_actor_page function will only work if it is called on the actor's main webpage. It will yield a dictionary of the actor's name and every show that they have appeared in. The comments in the function will walk through this in more detail.<jupyter_code>def parse_actor_page(self, response):
        '''
        Add to dictionary of every actor and every work they have acted in 

        @param self
        @param response: represents the website        
        @yield: adds dictionary  names and the works they have appeared in
        '''

        #grab the actors name
        actNam = response.css('title::text').get()[0:-7]
        
        #list of every work they have been apart of (producing, acting, etc...)
        flmLst = response.xpath("//*[@class='filmo-category-section']/div/b/a/text()").extract()
        
        #take the number of acting credits they have from 'Acting' tab
        #try with 'actor' tag, if that doesn't work, try with 'actress'
        #this depends on the individuals webpage
        try:
            actNum = int(response.xpath("//*[@id='filmo-head-actor']/text()").extract()[-1][2:-10])
        except:
            actNum = int(response.xpath("//*[@id='filmo-head-actress']/text()").extract()[-1][2:-10])

        #takes only the first actNum of works they been in, because this is the works they have acted in
        actLst = flmLst[:actNum]

        #yeild dictionary of the actors and a list of all their works
        yield {"actor" : actNam, "movie_or_TV_name" : actLst}<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup