<jupyter_start><jupyter_text>Data loading and churn label creation: Demo The raw data contains all the users' activities from March 30 to May 12, 2017. (Fresh data!!) Since the data amount is so huge for a PC, reading all data into memory is a mission impossible. On the other hand, it's not necessary to get all the activity information to determine if a user is a churn. To be more specific, whether one has activity is important, while the detail of the activities shall not be considered for the moment.In this demo, the file operations are walked through, from unzipping original data in .tar.gz zipped files to eventually print the churn user ids to a .txt file. For simplicity, only the data of the first day and the last day are used. More slim codes for processing all the data is given in another notebook  Procedure:    1. Unzip all the "play" activity data in a batch.        2. Choose a cut-off date for the churn labeling: Any user who is active before this cut-off date but has no activity after the date shall be labeled as a churn.        3. Read only "user_id","date/time" of each activity log before and after the cut-off date, and save them in two new file respectively.        4. Get two sets of active users, before and after the cut-off date: {active_before}, {active_after}        5. Output the churns to a .txt file. {Churn} = {active_before} - {active_before} & {active_after}      1. Unzip the 400+ .tar.gz compressed files of raw data7z can only unzip the .tar.gz files to .tar, so the complete unzip takes two steps:        1. Batch unzip from .tar.gz --> .tar    2. Batch unzip from .tar --> log files<jupyter_code>## In windows powershell, run the following iteration commands in the raw data directory:
'''
$files = Get-ChildItem "../data/raw/" -Filter *_play.log.tar.gz

foreach ($f in $files) {7z e $f -oC../data/raw/unzip}

cd unzip

$files = Get-ChildItem "../raw/unzip" -Filter *_play.log.tar

foreach ($f in $files) {7z e $f}
'''<jupyter_output><empty_output><jupyter_text>References:__[for loop in windows powershell](https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using-powershell)____[Unzip .tar.gz files using 7z commands](https://stackoverflow.com/questions/1359793/programmatically-extract-tar-gz-in-a-single-step-on-windows-with-7zip)__  2. Cut-off date = April 21st.  In this demo, the first file of 3/30 is before the cut-off date, while the second file of 5/12 is after the cut-off. In reality there are three weeks before cut-off, and three weeks after. The cut off might need changing later, depending on the model's performance.  3. File operations on the play logs    1. Open *play.log files one by one        2. "Read - Append" for each line        3. Write the new line to a new file for all lines of all files.        4. Meawhile, extract the user_id (first item) in each line, and save them into two python sets for logs before and after the "snapshot date"        5. Save the sets into .log files.     1. Open all play logs, using * wildcard<jupyter_code>import glob

filepath = 'C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\*play.log'
files = glob.glob(filepath)
len(files)
files
log_amounts = []
for the_file in files:
    f = open(the_file, 'r')
    lines = f.readlines()
    log_amounts.append(len(lines))
    f.close()
log_amounts<jupyter_output><empty_output><jupyter_text>2. Read the files, append the date to each line<jupyter_code>with open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\20170512_3_play.log','r') as f:
    content = f.readlines()
len(content)
first_line = content[0]
first_line
first_line_fields = content[0].strip('\n').split('\t')
#first_line_fields.append(f.name.split('\\')[-1][8])
first_line_fields
first_line_fields.append(f.name.split('\\')[-1][:8])
'\t'.join(first_line_fields)
new_line = '\t'.join(first_line_fields) + '\n'
new_line<jupyter_output><empty_output><jupyter_text>3. write the appended lines into a new file        Using the f.write() method<jupyter_code>output = open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\output\\all.log','a')
for the_file in files:
    with open(the_file, 'r') as f:
        lines = f.readlines()
        for line in lines:
            contents = line.strip('\n').split('\t')
            contents.append(f.name.split('\\')[-1][:8])
            output.write('\t'.join(contents)+'\n')

output.close()
with open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\output\\all.log','r') as output:
    lines = output.readlines()
len(lines)
sum(log_amounts)<jupyter_output><empty_output><jupyter_text>4. Save the user_ids into sets            Delete the all.log just created, as the procedure can be done at the same time<jupyter_code>from sets import Set

list_of_sets = []
# for each day's data, set the active users' user_id into a set.
with open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\output\\all.log','a') as output:
    for the_file in files:
        with open(the_file, 'r') as f:
            lines = f.readlines()
            list_of_sets.append(Set([line.split('\t')[0] for line in lines]))
            for line in lines:
                contents = line.strip('\n').split('\t')
                contents.append(f.name.split('\\')[-1][:8])
                output.write('\t'.join(contents)+'\n')
[len(each_set) for each_set in list_of_sets]<jupyter_output><empty_output><jupyter_text>5. Churn labeling and file saving            Save the user_id of churns into a new file.<jupyter_code>active_before, active_after = list_of_sets[0],list_of_sets[1]
#. set method: s.intersection(t) returns to a new set of s & t
loyal_users = active_before.intersection(active_after)
len(loyal_users)
#. set method: s.difference(t) returns to a new set of items in s but not in t
churn = active_before.difference(active_after)
len(churn)
new_users = active_after.difference(active_before)
len(new_users)
# Use loyal_user as an example

with open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\loyal.log','a') as loyal_file:
    loyal_file.write('\n'.join(list(loyal_users))+'\n')
# check loyal_file

with open('C:\\Users\\Sean\\Documents\\BitTiger\\Capston_music_player_python\\loyal.log','r') as loyal_file:
    lines = loyal_file.readlines()

lines<jupyter_output><empty_output>