<jupyter_start><jupyter_text>Cleaning and EDA<jupyter_code>## each observation is a video that has trended in Asia, Europe or North America
## dataset gives out details like number of views acquired, comments recieved, likes/dislikes. 
## Datetime : trending_date
## factor_vector: ratings_disabled, comments disabled, video glitched/removed, country of origin.
videos.head()
## Each row has some basic essential details about different videos published on youtube.
## Creating one complete dataset appending Asia, Europe and NA dataframes. 
trending = asia_trending.append(eu_trending).append(na_trending)
## Cleaning the dataset and changing data types wherever necessary
trending['trending_date'] = pd.to_datetime(trending['trending_date'], format = '%y.%d.%m')
pd.isna(trending['views']).sum() #9850 na values detected
pd.isna(trending['comment_count']).sum() #7396 nan values detected

# splitting likes_dislikes column into separate columns and adding them back to the dataframe
#trending['likes/dislikes'].str.split(pat = '/', n = -1, expand = True)
trending.columns = ['video_id', 'trending_date', 'views', 'likes_dislikes', 
                    'comment_count', 'thumbnail_link', 'comments_disabled', 
                    'ratings_disabled', 'video_error_or_removed', 'country']

trending[['likes','dislikes']] = trending.likes_dislikes.str.split(pat = '/', expand = True)
trending["likes"] = trending['likes'].astype("int64")
trending["dislikes"] = trending['dislikes'].astype("int64")

## converting ratings_disabled and video_error_or_removed into numeric type
trending['ratings_disabled'] = trending['ratings_disabled'].astype('int64')
trending['video_error_or_removed'] = trending['video_error_or_removed'].astype('int64')

## Removing unwanted columns
trending = trending.drop(['thumbnail_link','likes_dislikes'], axis = 1)
## Merging videos df with Trending df to get the final dataset
category.rename(columns = {'ID':'category_id'}, inplace = True)
category.head()
trending0 = trending.merge(videos, on = 'video_id', how = 'left')
trending1 = trending0.merge(category, on = 'category_id', how = 'left')


trending1["country"].str.len()
trending1["country"] = trending1["country"].str.strip()

## Converting publish time into datetime object
trending1["publish_time"] = pd.to_datetime(trending1["publish_time"])
trending1.head()<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code># Q1: In which country did youtube videos trend the most in the given span of time? 
trend_country = trending1.groupby('country').count().sort_values('video_id', ascending = False)["video_id"]
trend_country

## Youtube experiences the most number of trending videos in the US, while the least in Japan. 
# Q2: Which channel has trended the most number of times?
trend_channel = trending1.groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
trend_channel

## The most trending channel over Asia, Europe and North-America is 'The Late Show with <NAME>'.
## By country
# US
#trending1[trending1["country"] == "US"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in the US is INSIDER. 

# Canada
trending1[trending1["country"] == "Canada"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in Canada is 'VikatanTV'.

# Germany
trending1[trending1["country"] == "Germany"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in Germany is 'Galileo'.

# France
trending1[trending1["country"] == "France"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in France is 'Troom Troom FR'

# UK
trending1[trending1["country"] == "UK"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in the UK is 'PewDiePie'.

# India 
trending1[trending1["country"] == "India"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in India is 'ViketanTV'.

#Japan
trending1[trending1["country"] == "Japan"].groupby('channel_title').count().sort_values('video_id', ascending = False)["video_id"]
## The most trending channel in Japan is 'DHCテレビ '.'

## Interesting to see that the ViketanTV is the most trending channel in both India and Canada. 
## This could point towards there being some similarity in the tastes of viewers of both countries. 
# Q3: Which is the best time to publish a video on youtube such that it trends well? 
# This could be infered from analysing the publish time of the already trended videos. 

# Extract hour from the publish time 
def hr_func(ts):
    return ts.hour
trending1["hour"] = trending1["publish_time"].apply(hr_func)

trend_hour = trending1.groupby('hour').count().sort_values('video_id', ascending = False)["video_id"]
trend_hour

## From an inspection of the trend dataset, we can infer that an ideal time to publish a video on youtube (in the 3 continents, on an average) so that is become likely to trend is
## 3:00pm - 5:59pm
trending1[trending1["country"] == "India"].groupby('hour').count().sort_values('video_id', ascending = False)["video_id"]
## Specifically in India, a more suitable time would be 12-2 pm. 
#Q4(a): Which channel seems to capitalize on bad fame i.e. it trends via videos which garners more likes < dislikes? 
#trending1[trending1["likes"] < trending1["dislikes"]].groupby("channel_title").count().sort_values(by = "dislikes", ascending = False)
disliked_channels = trending1[trending1["likes"] < trending1["dislikes"]]
disliked_channels = disliked_channels.drop_duplicates(subset = ["video_id"], keep = "last")
pd.set_option('display.max_rows', None)
disliked_channels.groupby("channel_title").count().sort_values(by = "dislikes", ascending = False)['dislikes']

## The channel 'focus nachrichten' has created the most number of trending videos which has more dislikes than likes. 
## We could speculate that the creator make controversial, disturbing or unconventional videos.
## To further confirm our hypothesis, we check for the creator's popular videos that have more likes than dislikes.
print(trending1[trending1["channel_title"] == "focus nachrichten"].drop_duplicates(subset = ["video_id"], keep = "last").shape) #39 distinct publishes
## Focus Nachriten has 26 distict videos where likes < dislikes. The total videos that it has published is 39. 
26/39*100
## 66% of his videos have more dislikes than likes.
## Hence it could be reasonably infered that it capitalizes on disturbing or disliked content. 
# Q4(b): Which category seems to capitalize on bad fame i.e. it trends via videos which garners more likes < dislikes? 
disliked_category = trending1[trending1["likes"] < trending1["dislikes"]]
disliked_category = disliked_category.drop_duplicates(subset = ["video_id"], keep = "last")
disliked_category.shape
pd.set_option('display.max_rows', None)
print(disliked_category.groupby("Category name").count().sort_values(by = "dislikes", ascending = False)['dislikes'])
## Entertainment Category has 617 distict videos where likes < dislikes.
print(trending1[trending1["Category name"] == "Entertainment"].drop_duplicates(subset = ["video_id"], keep = "last").shape)
## The total videos that have trended in the Entertainment category is 34383. 

617/34383*100
## 1.79% of all videos of the genre capitalises on bad fame with controversial or unconventional content
# Q5. What is the engagement rate of each genre/category of videos?
# Engagement is defined as the number of people who either like, dislike or comment on the video, per 100 viewers of the video
# Engagement=(likes+dislikes+comments)/view

#Dropping the videos that have comments or ratings disables
engagement_df1 = trending1[trending1['comments_disabled']!=1]
engagement_df = engagement_df1[engagement_df1['ratings_disabled']!=1]
engagement_df=engagement_df.dropna()

engagement_df['engagement'] = (engagement_df['likes']+engagement_df['dislikes']+engagement_df['comment_count'])/engagement_df['views']*100
engagement_by_genre = engagement_df.groupby('Category name').mean().sort_values('engagement', ascending = False)["engagement"]
engagement_by_genre
## The most engagement is observed in 'Nonprofits & Activism' videos with 7% of the viewers either rating the video (likes/dislikes) or commenting on it. 
# Q6. What is the impact of the creator disabling comments on the video's likes & dislikes?
## For this, we plot a boxplot to ascertain the IQR of likes when (i) comments are enabled & (ii) comments are disabled
Boxplot_likes = trending1.boxplot(by ='comments_disabled', column =['likes'], grid = False, showfliers = False)
Boxplot_likes.set_ylim(0,40000)
Boxplot_likes.set_title('Likes on Videos when Comments are Enabled v/s Disabled')
Boxplot_likes.set_xlabel('Comments Enabled or Disabled')
Boxplot_likes.set_ylabel('Number of Likes')
trending1['likes'].groupby(trending1['comments_disabled']).describe()

## The IQR of the amount of likes on a video is wider & higher when the comment section is enabled. This also means greater engagement.
#Plotting a boxplot to ascertain the IQR of dislikes when (i) comments are enabled & (ii) comments are disabled
Boxplot_dislikes = trending1.boxplot(by ='comments_disabled', column =['dislikes'], grid = False, showfliers = False)
Boxplot_dislikes.set_title('Disikes on Videos when Comments are Enabled v/s Disabled')
Boxplot_dislikes.set_xlabel('Comments Enabled or Disabled')
Boxplot_dislikes.set_ylabel('Number of Dislikes')

## The IQR of the amount of dislikes on a video is wider & higher when the comment section is enabled. This also means greater engagement.
#Q7. What is the penetration of Youtube (specifically trending videos) in the listed countries?
trending_2=trending1.dropna()
activeness = trending_2.groupby('country').mean().sort_values('views', ascending = False)["views"]
type(activeness)

# Activeness is a series. Converting it into a data frame
country_activeness=activeness.to_frame()
country_activeness

#The names of the country is given as an index. Adding a column 
country_activeness['country']=['UK', 'US', 'Canada', 'India', 'Germany', 'France', 'Japan']
#Adding a column with the population of the respective countries in 2018
country_activeness['population']=[6.610000e+07,3.251000e+08 , 3.650000e+07, 1.338700e+09, 82700000, 6.690000e+07, 1.268000e+08]
country_activeness

activeness = (country_activeness['views'])/ (country_activeness['population'])*1000
activeness.sort_values(ascending = False)
## The highest penetration of Youtube (limited to trending videos) is in the UK with 92 per 1000 people watching the videos. 
## After a huge margin, this is followed by Canada with 31 per 1000 people.
## Q8: Which video has trended the most number of time in the entire dataset? 
trending1.value_counts(["title"])

# "Drake - Nice For What" is the most trending video in the dataset. It has trended 96 times
df = trending1.groupby("title")
df = df.agg({"trending_date": "nunique"})
df = df.reset_index()
df[df["title"] == "Drake - Nice For What"] 

df = trending1.groupby("title")
df = df.agg({"country": "nunique"})
df = df.reset_index()
df[df["title"] == "Drake - Nice For What"]

# This video trended for a total of 31 days and in 5 countries. 
## Q9: How many videos have trended across all 7 countries? 
df = trending1.groupby("title")
df = df.agg({"country": "nunique"})
df = df.reset_index()
df[df["country"] == 7].shape

## 22 videos in the dataset have trended across all 7 countries.<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup