<jupyter_start><jupyter_text>Data Cleaning & FeaturesThis step is composed by the investigation of any still missing data, feature selection and creation of the tables that will be used on Tableau.<jupyter_code>INTERIM_DATA_PATH = Path('data/interim')
PROCESSED_DATA_PATH = Path('data/processed')

# Data Loading
SPOTIFY_BILLBOARD_DATA_PATH = INTERIM_DATA_PATH / 'spotify_billboard_data.csv'

# Datasets
ARTISTS_DATA_FILEPATH = PROCESSED_DATA_PATH / 'artists.csv'
LYRICS_DATA_FILEPATH  = PROCESSED_DATA_PATH / 'lyrics.csv'
TAGS_DATA_FILEPATH = PROCESSED_DATA_PATH / 'tags.csv'
SONGS_DATA_FILEPATH = PROCESSED_DATA_PATH / 'songs.csv'

PROCESSED_DATA_PATH.mkdir(exist_ok=True, parents=True)<jupyter_output><empty_output><jupyter_text>Data Loading<jupyter_code>spotify_billboard_df = pd.read_csv(SPOTIFY_BILLBOARD_DATA_PATH, index_col=0)

print(spotify_billboard_df.shape)
print(spotify_billboard_df.columns)
spotify_billboard_df.head()<jupyter_output>(4028, 40)
Index(['index', 'name', 'artist', 'api_name', 'api_artists', 'danceability',
       'energy', 'key', 'loudness', 'mode', 'speechiness', 'acousticness',
       'instrumentalness', 'liveness', 'valence', 'tempo', 'type', 'uri',
       'track_href', 'analysis_url', 'duration_ms', 'time_signature', 'year',
       'lyrics', 'tags', 'num_syllables', 'pos', 'year.1', 'fog_index',
       'flesch_index', 'num_words', 'num_lines', 'sentiment_neg',
       'sentiment_neu', 'sentiment_pos', 'sentiment_compound', 'title',
       'f_k_grade', 'difficult_words', 'num_dupes'],
      dtype='object')<jupyter_text>Missing ValuesBy looking at the missing values, there isn't any big problem, so nothing needs to be done<jupyter_code>msno.matrix(spotify_billboard_df, figsize=(20, 5))<jupyter_output><empty_output><jupyter_text>Feature SelectionBelow are the features that we are going to use for the exploration.<jupyter_code>FEATURE_COLUMNS = [
    # Identification Features
    'name', 'artist', 'api_artists',
    
    # Spotify Features
    'danceability', 'energy', 'key', 'loudness', 'mode', 'speechiness',
    'acousticness', 'instrumentalness', 'liveness', 'valence', 'tempo',
    'duration_ms', 'time_signature',
    
    # Billboard Features
    'year', 'lyrics', 'tags', 'num_syllables', 'pos',
    'fog_index', 'flesch_index', 'num_words', 'num_lines',
    'sentiment_neg','sentiment_neu', 'sentiment_pos', 'sentiment_compound',
    'f_k_grade', 'difficult_words', 'num_dupes'
]

data_df = spotify_billboard_df[FEATURE_COLUMNS]<jupyter_output><empty_output><jupyter_text>Creating and splitting data into multiple tablesTo help the exploration, my approach here was to split the data into 4 main tables: (i) artists tables, (ii) lyrics table, (iii) tags table and (iv) songs table. All these tables are index by a numerical index.  Artists tableThis table contain all the artists that are credited in the song (not only the main artist). This information was retrieved from Spotify API.<jupyter_code>artists_df = pd.DataFrame()
artists_col = data_df.api_artists.dropna()

for ix, value in tqdm(artists_col.iteritems(), total=len(artists_col)):
    if ix is not None and value is not None:
        artists = [(ix, artist) for artist in value.split(",")]
        artists_df = artists_df.append(artists)
        
artists_df = artists_df.rename(columns = {0: "index", 1: "artist"})
artists_df = artists_df.set_index("index")
artists_df.to_csv(ARTISTS_DATA_FILEPATH)

artists_df.head()<jupyter_output><empty_output><jupyter_text>Lyrics TableTo understand better the lyrics of the songs, each row of this table represents a lemma found on the song lyrics. Other informations presented in the rows are if they are a stop word or not and the frequency of the lemma in the song. **NOTE:** For more information about lemmas, read [Lemmatisation](https://en.wikipedia.org/wiki/Lemmatisation) on Wikipedia.<jupyter_code>lyrics_list = []

for row in tqdm(data_df.itertuples(), total=len(data_df)):
    
    # Song & Lyrics Info
    song_name = row.name
    song_artist = row.artist
    song_api_index = row.Index
    doc = nlp(row.lyrics.lower())
    
    # Tokens and Counts
    word_freq_non_stop = Counter([
        token.lemma_ for token in doc 
        if token.is_stop != True 
        and token.is_punct != True 
        and '\n' not in token.text
    ])
    unique_words_non_stop = list(word_freq_non_stop.keys())
    freq_non_stop = list(word_freq_non_stop.values())

    word_freq_stop = Counter([
        token.lemma_ for token in doc 
        if token.is_stop == True 
        and token.is_punct != True 
        and '\n' not in token.text
    ])
    unique_words_stop = list(word_freq_stop.keys())
    freq_stop = list(word_freq_stop.values())
    
    # Creating DataFrame
    number_of_words = len(unique_words_stop) + len(unique_words_non_stop)
    
    lyrics_song_df = pd.DataFrame({
        "index": [song_api_index] * number_of_words,
        "song": [song_name]  * number_of_words,
        "artist": [song_artist] * number_of_words,
        "lemma": unique_words_stop + unique_words_non_stop,
        "count": freq_stop + freq_non_stop,
        "is_stop": ([True] * len(unique_words_stop)) + ([False] * len(unique_words_non_stop))
    })
    
    lyrics_list.append(lyrics_song_df)
    
lyrics_df = pd.concat(lyrics_list)
lyrics_df = lyrics_df.set_index("index")
lyrics_df.to_csv(LYRICS_DATA_FILEPATH)

lyrics_df.head()<jupyter_output><empty_output><jupyter_text>Tags TableAgain here, each rows of the table is composed by a tag that the song has received from the billboard dataset<jupyter_code>tags_df = pd.DataFrame()
tags_col = data_df.tags.dropna()

for ix, value in tqdm(tags_col.iteritems(), total=len(tags_col)):
    if ix is not None and value is not None:
        tags = [(ix, tag) for tag in ast.literal_eval(value)]
        tags_df = tags_df.append(tags)
        
tags_df = tags_df.rename(columns = {0: "index", 1: "tag"})
tags_df = tags_df.set_index("index")
tags_df.to_csv(TAGS_DATA_FILEPATH)

tags_df.head()<jupyter_output>100%|██████████| 4028/4028 [00:08<00:00, 485.24it/s]<jupyter_text>Songs TableThe songs table is composed by any other information not included in the other tables.<jupyter_code>data_df = data_df.rename(columns = {'artist': 'main_artist'})
data_df = data_df.drop(columns=['api_artists', 'tags'])
data_df.to_csv(SONGS_DATA_FILEPATH)

data_df.head()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup