<jupyter_start><jupyter_text>Predictive maintenance Part 1: Data PreparationThe original data can be [downloaded from this link.](https://ti.arc.nasa.gov/tech/dash/groups/pcoe/prognostic-data-repository/turbofan) Since the content in the train and test datasets is different, we are making it uniform before we start the data exploration and the model buiding process. We will convert the data into a more natural format for Vaex.<jupyter_code>import vaex<jupyter_output><empty_output><jupyter_text>Read the dataThe data contains a list of sensors. These are their names and meanings:| Name      |Description                      |Unit     |    |-----------|---------------------------------|---------|    | T2        | Total temperature at fan inlet  | °R      |    | T24       | Total temperature at LPC outlet | °R      |    | T30       | Total temperature at HPC outlet | °R      |    | T50       | Total temperature at LPT outlet | °R      |    | P2        | Pressure at fan inlet           | psia    |    | P15       | Total pressure in bypass-duct   | psia    |    | P30       | Total pressure at HPC outlet    | psia    |    | Nf        | Physical fan speed              | rpm     |    | Nc        | Physical core speed             | rpm     |    | epr       | Engine pressure ratio (P50/P2)  | --      |    | Ps30      | Static pressure at HPC outlet   | psia    |    | phi       | Ratio of fuel flow to Ps30      | pps/psi |    | NRf       | Corrected fan speed             | rpm     |    | NRc       | Corrected core speed            | rpm     |    | BPR       | Bypass Ratio                    | --      |    | farB      | Burner fuel-air ratio           | --      |    | htBleed   | Bleed Enthalpy                  | --      |    | Nf_dmd    | Demanded fan speed              | rpm     |    | PCNfR_dmd | Demanded corrected fan speed    | rpm     |    | W31       | HPT coolant bleed               | lbm/s   |    | W32       | LPT coolant bleed               | lbm/s   |<jupyter_code>column_names = ['unit_number', 'time_in_cycles', 'setting_1', 'setting_2', 'setting_3',
                'T2', 'T24', 'T30', 'T50', 'P2', 'P15', 'P30', 'Nf', 'Nc', 'epr', 'Ps30', 'phi', 
                'NRf', 'NRc', 'BPR', 'farB', 'htBleed', 'Nf_dmd', 'PCNfR_dmd', 'W31', 'W32']


# The training data
train_data = vaex.read_csv("./data/train_FD001.txt", sep='\s+', names=column_names)

# The testing data
test_data = vaex.read_csv("./data/test_FD001.txt", sep='\s+', names=column_names)

# The "answer" to the test data
y_test = vaex.read_csv('./data/RUL_FD001.txt', names=['remaining_cycles'])
y_test['unit_number'] = vaex.vrange(1, 101)
y_test['unit_number'] = y_test.unit_number.astype('int')<jupyter_output><empty_output><jupyter_text>Create proper train and test datasets- in the training set, the engines are run until failure occurs, so we can calculate the target varuable, i,e, the RUL (Remaining Useful Life) based on when a particular engines running;- in the test set the engines are run for some time, and our goal is to predict their RULs. Their RUL are provided in a separate file, so we need to join it so it can be made available for evaluating scores and estimateing model performance<jupyter_code>def prepare_data(data, y=None):
    df = data.copy()  # As to not modify the underlying dataframe
    # Count how many cycles each unit is run for - groupby and count
    g = df.groupby('unit_number').agg({'max_cycles': vaex.agg.count('time_in_cycles')})
    # Join to the main data - basically adds the "max_cycle" column
    df = df.join(other=g, on='unit_number', how='left')
    
    # Calculate the RUL:
    if y is None:  # This is for the train data -> last point is the point of failure
        # Calculate the RUL
        df['RUL'] = df.max_cycles - df.time_in_cycles
        # Drop the column that is not needed anymore
        df = df.drop(columns=['max_cycles'])
    else:  # This is for the test data -> add the answer to calculate the RUL
        # Join the answers
        df = df.join(y, on='unit_number', how='left')
        # Calculate the RUL
        df['RUL'] = df.max_cycles + df.remaining_cycles - df.time_in_cycles
        # Drop the columns that are not needed anymore
        df = df.drop(columns=['remaining_cycles', 'max_cycles'])    
    # Done
    return df
# Add the RUL to the train and test sets
df_train = prepare_data(train_data)
df_test = prepare_data(test_data, y=y_test)<jupyter_output><empty_output><jupyter_text>Quick preview of the datasets<jupyter_code>df_train
df_test<jupyter_output><empty_output><jupyter_text>Export the datasets to HDF5<jupyter_code>df_train.export_hdf5('./data/data_train.hdf5')
df_test.export_hdf5('./data/data_test.hdf5')<jupyter_output><empty_output>