<jupyter_start><jupyter_text>About![prjpic](doc/media_main/story.png)This is the main demo of this repo, it is about a concept study on the MIMII dataset to detect anomalies of machines or machine parts like fans, slider, pump and valves by means of classic machine learning and deep learning methods.In runs through the essentials to demonstrate the steps* feature extraction* indvidual model training within the ensemble* ensemble evaluation* summary and scoresFor more in depth information and some insights about the decisions we made, take a look at the subnotebooks and utilities as mentioned in the README.  Imports<jupyter_code>#===============================================
# Basic Imports
BASE_FOLDER = './'
TARGET_FOLDER_FE = r'\dataset\extdia_v1_essential' # output folder for ffeat. extraction
# import the repo-local utility py files
%run -i utility\feature_extractor\JupyterLoad_feature_extractor.py
%run -i utility\modeling\JupyterLoad_modeling.py

# feature extraction diagram
%run -i feature_extraction_diagrams\extdia_v1_essential
%run -i utility\extractor_batch.py

# helper
from tqdm.auto import tqdm
import glob
import gc

# sklearn 
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA, FastICA

feat_ext_folder = os.path.abspath(BASE_FOLDER + TARGET_FOLDER_FE)<jupyter_output>load feature_extractor_mother
load feature_extractor_mel_spectra
load feature_extractor_psd
load feature_extractor_ICA2
load feature_extractore_pre_nnFilterDenoise
load extractor_diagram_mother
load Simple_FIR_HP
load TimeSliceAppendActivation
load load_data
Load split_data
Load anomaly_detection_models
Load pseudo_supervised_models
Load tensorflow models
Load detection_pipe
load # extractor diagram V1 essential
load extractor_batch<jupyter_text>General configTo customize the following notebook execution, set the specifications here:The execution time on a local desktop PC for all four IDs of one machine and one SNR is circa 1 hour. We recommend only executing a set of 4 variations at a time (for instance over one machine).<jupyter_code>#===============================================
# Possible variations
# 
# SNRs = ['6dB', '0dB', 'min6dB']
# machines = ['pump', 'valve', 'fan', 'slider']
# IDs = ['00', '02', '04', '06']

SNRs = ['6dB', 'min6dB']
machines = ['pump']
IDs = ['04', '06']

# if we want to include augmented model (SVM) set True
aug = True

# note: increase n_jobs to max. CPUs you have use all hyperthreading cores (there is no auto detect just now)
n_jobs = 7<jupyter_output><empty_output><jupyter_text>Utility wrapper functions<jupyter_code>def feat_ext_process_set(FileFindDict, main_channel=0, sporadic=False, augment=False,FileCountLimit=None, n_jobs=4):
    
    if sporadic:
        dt = 1 # 1 means time slicing 
    else:
        dt = 0
        
    if augment:
        ag = 0 # augment only normal operation
    else:
        ag = -2 # not existing class = no augment
        
    extractor_batch(base_folder= BASE_FOLDER, 
                    target_folder=TARGET_FOLDER_FE, 
                    extdia = extdia_v1_essential, 
                    FileFindDict = FileFindDict,
                    n_jobs = n_jobs,
                    target_class_map = {'abnormal':1, 'normal': 0},
                    FileCountLimit = FileCountLimit,
                    datset_folder_from_base = 'dataset',
                    fHP=120,
                    DeviceType=dt,
                    main_channel = main_channel,
                    augment=ag)
    
    gc.collect()
def find_data_file(SNR, machine, ID):
    '''
    Function to find existing feature data files.
    If one is found, it returns the path.
    Multiple files will raise error.
    '''
    
    # list all the files that match the mask
    path = glob.glob(BASE_FOLDER 
              + '/dataset/extdia_v1_essential/{}{}{}_EDiaV1HP'.format(machine, SNR, ID) 
              + "*pandaDisc*.pkl", recursive=True)
    
    if len(path) == 0:
        return None
    elif len(path) == 1:
        return path[0]
    else:
        raise Exception('More than one file found:', path)<jupyter_output><empty_output><jupyter_text>Feat Extraction  Feature extraction diagram![exdia](doc/media_feature_extraction/exdia_v1_essential.png)in order to modify the diagram go the class definition:  /feature_extraction_diagrams/extdia_v1_essential.py  Note to the main_channelThe main channel is picking one microphone out of the 8, this can be seen as if the demo version is strictly in working monoOr a DOA could be used to find the main direction see : feature_extraction_diagrams/A21_DirectionOfArrival_DOA/pyroomacustic_DOA.ipynb<jupyter_code>if not find_data_file('6dB', 'pump', '02'):
    ExampleFileFilter = {'SNR': ['6dB'],'machine': ['pump'],'ID': ['02']}
    # create some 
    feat_ext_process_set(ExampleFileFilter,
                        main_channel=2,
                        sporadic = False,
                        augment = True,
                        FileCountLimit= None,
                        n_jobs=n_jobs)<jupyter_output><empty_output><jupyter_text>Spot Check the output<jupyter_code># This code reloads pkl files that have been stored
# in the step above - notice only created files can be loaded
# then a plot is made form n and n+1 output ports
# this cell is ment as a spot check before running the batch that might,
# take much more time !

d_MEL_den = pickle.load( open( feat_ext_folder + r'\pump6dB02_EDiaV1HPaug0_outpMEL_den.pkl', "rb" ))
d_MEL_raw = pickle.load( open( feat_ext_folder + r'\pump6dB02_EDiaV1HPaug0_outpMEL_raw.pkl', "rb" ))
d_PSD_raw = pickle.load( open( feat_ext_folder + r'\pump6dB02_EDiaV1HPaug0_outpPSD_raw.pkl', "rb" ))
n1=2
n2=7
plt.figure(figsize=(16,9))
plt.subplot(321)
feature_extractor_from_dict(d_MEL_raw[n1],BASE_FOLDER).plot(False)
plt.subplot(322)
feature_extractor_from_dict(d_MEL_raw[n2],BASE_FOLDER).plot(False)
plt.subplot(323)
feature_extractor_from_dict(d_MEL_den[n1],BASE_FOLDER).plot(False)
plt.subplot(324)
feature_extractor_from_dict(d_MEL_den[n2],BASE_FOLDER).plot(False)
plt.subplot(325)
feature_extractor_from_dict(d_PSD_raw[n1],BASE_FOLDER).plot(True)
plt.subplot(326)
feature_extractor_from_dict(d_PSD_raw[n2],BASE_FOLDER).plot(True)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Batch creation of feature data<jupyter_code># Create the batch of feature data
# note: there is still a deepcopy issue you may experience memory leak : https://github.com/BA-HanseML/NF_Prj_MIMII_Dataset/issues/58
for SNR in SNRs:
    for machine in machines:
        # some metaparameters for feature generation according to each machine
        # main_channel is the channel in direction of the machine (according to MIMII-paper setup)
        # for sporadic machinery we use the activation time detection
        if machine == 'valve':
            main_channel = 0
            sporadic = True
        elif machine == 'pump':
            main_channel = 2
            sporadic = False
        elif machine == 'fan':
            main_channel = 4
            sporadic = False
        elif machine == 'slider':
            main_channel = 6
            sporadic = True
        
        for ID in IDs:
            # check if files already exist
            if not find_data_file(SNR, machine, ID):
                BatchFileFilter = {'SNR': SNR,'machine': machine,'ID': ID}
                feat_ext_process_set(BatchFileFilter,
                                    main_channel=main_channel,
                                    sporadic=sporadic,
                                    augment=True,
                                    n_jobs=n_jobs)<jupyter_output><empty_output><jupyter_text>Modeling  Ensemble![diagram](doc\media_modeling\ensemble_structure.png)The above diagram shows the final ensemble structure. We have four individual pipes with different sets of feature data, preprocessing steps and models. The class definition below will combine them in an ensemble. This class definition is only used to evaluate the found ensemble structure. As input pre extracted feature data is used.<jupyter_code>class uni_Ensemble(object):
    def __init__(self, SNR, machine, ID, aug=True):
        self.SNR = SNR
        self.machine = machine
        self.ID = ID

        # individual model weights for the blender
        self.weights = [1.3, 1.0, 0.9]
        
        # feature constellation for the individual models
        feature_constellation = [('MEL_den', {'function':'frame', 'frames':5}), # Autoencoder MEL spectrum
                          ('MEL_den', {'function':'frame', 'frames':5}), # Isolation Forest MEL spectrum
                          ('PSD_raw', {'function':'flat'}), # Isolation Forest Welch method
                          ('PSD_raw', {'function':'flat'})] # Isolation Forest Welch method
        
        # augmented is optional
        if aug: 
            feature_constellation.append(
                ('PSD_raw', {'function':'flat'}))
            self.weights.append(0.8)
        
        # set up all the model tasks
        self.tasks = [{
            'path_descr': find_data_file(SNR, machine, ID),
            'feat': feature[1],
            'feat_col':feature[0],
            'SNR':SNR,
            'machine':machine,
            'ID':ID,
            'BASE_FOLDER':BASE_FOLDER
        } for feature in feature_constellation]
        
        # set up all the individual model pipelines as shown in the ensemble diagram
        self.pipes = [
            Pipe(preprocessing_steps=[(PCA, {'n_components':64}),(StandardScaler, {})], 
                 modeling_step=(uni_AutoEncoder, {'epochs':50}), 
                 pseudo_sup=False), # Autoencoder MEL spectrum
            
            Pipe(preprocessing_steps=[(PCA, {'n_components':64}),(StandardScaler, {})], 
                 modeling_step=(uni_IsolationForest, {'n_estimators':64, 'max_features':4}), 
                 pseudo_sup=False), # Isolation Forest MEL spectrum
            
            Pipe(preprocessing_steps=[(StandardScaler, {})], 
                 modeling_step=(uni_IsolationForest, {'n_estimators':200, 'max_features':1}), 
                 pseudo_sup=False), # Isolation Forest Welch method        
        ]
        
        # again augmented is optional
        if aug: self.pipes.append(
            Pipe(preprocessing_steps=[(StandardScaler, {})],
                 modeling_step=(uni_svm, {'C': 0.1, 'degree':3,'kernel':'rbf'}), 
                 pseudo_sup=True) # SVM augmented Welch method
                                 )

    def fit(self):
        # training of all the individual models
        for pipe, task in zip(self.pipes, self.tasks):

            # set up the task
            pipe.task = task

            # split data into train and testset
            pipe.split_data()

            # get the data
            print('...loading data')
            data_train, data_test = pipe.get_data()
            print('data loading completed\n\n...preprocessing data')

            # preprocessing
            data_train, data_test = pipe.preprocess(data_train, data_test)
            print('data preprocessing finished\n\n...fitting the model')

            # fitting the model
            pipe.fit_model(data_train)
            print('model fitted successfully\n\n...fitting the prediction scaler')

            # fitting the prediction scaler
            pipe.fit_aggr_score_scaler(data_train, pipe.df_train.path)
            print('prediction scaler fitted successfully\n\n...evaluating model')
            
            # evaluating over ground truth
            pipe.evaluate(data_test)
            print('evaluation successfull, roc_auc:', pipe.roc_auc)

    def evaluate(self):
        # evaluate the individual models
        for i, (pipe, weight) in enumerate(zip(self.pipes, self.weights)):
            
            # get the data
            _, data_test = pipe.get_data()
            
            # preprocess according to individual pipe
            data_test = pipe.preprocess_post(data_test)
            
            # append up weighted predictions
            if i == 0:
                predictions = pipe.predict_aggr_score(
                    data_test, pipe.df_test.path)*weight
                ground_truth = pipe.median_by_file(
                    pipe.ground_truth, pipe.df_test.path)
            else:
                predictions = np.append(predictions, pipe.predict_aggr_score(
                    data_test, pipe.df_test.path)*weight, axis=1)

        # sum up weighted predictions
        prediction = np.sum(predictions, axis=1)
        
        # evaluate predictions
        return roc_auc_score(ground_truth, prediction)
filepath = 'results.dataframe'
columns = ['SNR', 'machine', 'ID', 'augmented', 'roc_auc']

# if the dataframe file exists, load it
# if not, create an empty dataframe
if os.path.exists(filepath):
    results = pd.read_pickle(filepath)
else:
    results = pd.DataFrame(data=None, columns=columns)

# iterate through all the wanted machines
for SNR in SNRs:
    for machine in machines:
        for ID in IDs:
            # check if ensemble result already exists within the dataframe
            # if not, run training and evaluation
            mask = (results.SNR==SNR) & (results.machine==machine) & (results.ID==ID) & (results.augmented==aug)
            if results[mask].shape[0]==0:
                # instantiate ensemble
                ensemble = uni_Ensemble(SNR, machine, ID, aug=aug)
                
                # fit ensemble
                ensemble.fit()
                
                # evaluate ensemble
                roc_auc = ensemble.evaluate()
                
                # create results row for dataframe and append
                df_tmp = pd.DataFrame(data=[[
                    SNR,
                    machine,
                    ID,
                    aug,
                    roc_auc
                ]], columns=columns)
                results = results.append(df_tmp)
            
            # save to file
            results.to_pickle(filepath)<jupyter_output><empty_output><jupyter_text>Summary/ Results The scores can be investigated. The results dataframe is pushed to GitHub with precalculated results. Running this notebook with the config of your interest should reproduce these results.<jupyter_code>results.groupby(by=['augmented', 'SNR']).roc_auc.mean()
results<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup