<jupyter_start><jupyter_text>Map Reduce Algorithm Map reduce is very functional algorithm where three parts of it can easily executed on different machines. In this assignment, we will try to implement this algorithm into a >100 million rows dataset.For comparison, we will have two ways of getting counts of carriers.1. Serial way - Looping through each record and counting each airline's flight2. Map reduce way - map, reduce and sort, and collect way to counn the flights<jupyter_code>import os
import glob
import pandas as pd
import utils
import data_handler<jupyter_output><empty_output><jupyter_text>Data Preperation<jupyter_code>data_handler.download_dataset()<jupyter_output>data downloaded. you can skip this step or delete data folder to download again.<jupyter_text>Variables<jupyter_code># all files under data folder
file_list = sorted(glob.glob(os.path.join('data', '*.csv.bz2')))
carrier_counts = {}<jupyter_output><empty_output><jupyter_text>Serial WayHere, we are getting a list of files under the data folder. The serial way requires to loop through all of the records and update one by one. If you think your computer is fast enough, you can also try `read_as_dataframe` in data_handler module. You can simply use the following code rather than looping through each file.```py inital value dictionarycarrier_counts = {} read all datasets into onedf = data_handler.read_as_dataframe() unique careerscarriers = df.UniqueCarrier.unique() get the countsfor carrier in df.UniqueCarrier:    carrier_counts[carrier] += 1```Since my computer wasn't able to finish it, I prefered the loop. Hopefully, you won't run into these issues thanks to the Map Reduce Algorithm!<jupyter_code>for ind, data_file in enumerate(file_list):
    # read current data
    df = pd.read_csv(data_file, encoding='ISO-8859-1', memory_map=True, low_memory=False)
    
    # unique airlines in dataset
    carriers = df.UniqueCarrier.unique()
    
    # update the global carrier_count
    for key in carriers:
        if key not in carrier_counts:
            carrier_counts.update({key: 0})
    
    # loop through each row in dataframe 
    for carrier in df.UniqueCarrier:
        carrier_counts[carrier] += 1

    # info 
    prefix ='Shape: {} ; {} Mb'.format(
        df.shape, round(df.memory_usage().sum() / 1e+6,2))
    
    # track the progress
    utils.progressbar(len(file_list), ind + 1, prefix=prefix)
# results
carrier_counts<jupyter_output><empty_output><jupyter_text>Map Reduce Way<jupyter_code># todo: your code here<jupyter_output><empty_output><jupyter_text>Conclusion<jupyter_code># todo: your conclusion here<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup