<jupyter_start><jupyter_text>Create Benign vs X Files [not the show]This notebook exists to create multiple data files out of one large dataset. The dataset contains multiple samples with various classifications, such as 'Adware', 'Benign', 'Ransomware', 'Scareware', and 'SMSmalware'. The goal of the notebook is to generate data files that contain only 'Benign' and some other malware classification (Ex. `adware_vs_benign.csv` would only contain 'Adware' and 'Benign').  To start out, we need a file that was generated previously through either another notebook or a python script. The dataset I'm using in this notebook contains all of the features available and all of the classifications possible (`full_features_full_data.csv`). In this case, the file was generated using the `create full data file` notebook.  Due to potentially kernel-killing memory constraints, you may have to use the [script version](https://github.com/rambasnet/DeepLearning-AndroidMalware/tree/master/data_scripts/create_benign_vs_x_files.py) of this file available in the `data_scripts` directory instead of using Jupyter Notebook.<jupyter_code>import pandas as pd

print('Imports complete.')
path = '../../malware_dataset/'
df = pd.read_csv(path + 'full_features_full_data.csv')

# Initialize the random_state for later use
random_state=1

df.head()<jupyter_output><empty_output><jupyter_text>Dataset CompositionThe datasets that we are creating will be 50/50 balanced, meaning that the 'Benign' and 'Adware' samples, for example, will have a 1:1 ratio. However, since the lowest number of samples is in the 'SMSmalware' category, we will limit the amount of each category to this amount.<jupyter_code># Define our target variable
dep_var = 'Label'
df[dep_var].value_counts()
# The lowest amount of data is SMSMALWARE at 229,275 samples (or it's close enough)
low_count = 229275<jupyter_output><empty_output><jupyter_text>Creating the FilesTo begin, we need to isolate all of the instances that are of each given classification and take the random sample we need. We will use these variables to then create the csv files later. A consequence of the work here is that all of the data files we create will have the same 'Benign' samples. All of these experiments down the line are against exactly the same instances of the 'Benign' class. Additionally, since we've given a `random_state`, as long as we don't change this value, every execution of the notebook will yield the same data sets.<jupyter_code># Pull out all of the samples into separate dataframes

# Benign data
benign_data = ( df.loc[ df[dep_var] == 'BENIGN']).sample(low_count, random_state=random_state)
print(benign_data[dep_var].value_counts())

# Adware data
adware_data = ( df.loc[ df[dep_var] == 'ADWARE']).sample(low_count, random_state=random_state)
print(adware_data[dep_var].value_counts())

# Scareware data
scareware_data = ( df.loc[ df[dep_var] == 'SCAREWARE']).sample(low_count, random_state=random_state)
print(scareware_data[dep_var].value_counts())

# Ransomware data
ransomware_data = ( df.loc[ df[dep_var] == 'RANSOMWARE']).sample(low_count, random_state=random_state)
print(ransomware_data[dep_var].value_counts())

# SMSMalware data
smsmalware_data = ( df.loc[ df[dep_var] == 'SMSMALWARE']).sample(low_count, random_state=random_state)
print(smsmalware_data[dep_var].value_counts())<jupyter_output><empty_output><jupyter_text>Now, we actually will create the files. Every file starts with the 'Benign' class and we then append the malware data on the end. We have to be careful here or, otherwise, we would append the headers or overwrite the files we just made.<jupyter_code># Write the data out to files
print('Creating Adware vs Benign file...', end='')
filen = 'adware_vs_benign.csv'
benign_data.to_csv(path + filen)
adware_data.to_csv(path + filen, mode='a', header=False)
print('done')

print('Creating Ransomware vs Benign file...', end='')
filen = 'ransomware_vs_benign.csv'
benign_data.to_csv(path + filen)
ransomware_data.to_csv(path + filen, mode='a', header=False)
print('done')

print('Creating Scareware vs Benign file...', end='')
filen = 'scareware_vs_benign.csv'
benign_data.to_csv(path + filen)
scareware_data.to_csv(path + filen, mode='a', header=False)
print('done')

print('Creating SMSmalware vs Benign file...', end='')
filen = 'smsmalware_vs_benign.csv'
benign_data.to_csv(path + filen)
smsmalware_data.to_csv(path + filen, mode='a', header=False)
print('done')
print('All datasets created! Exiting...')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup