<jupyter_start><jupyter_text>モデルの確認- ランダムカットフォレストのモデルでバッチ変換- 推論結果を可視化  パラメータ<jupyter_code>training_job_name = 'machine-temperature-iot-training-job'
labeled_test_data_s3_path = 's3://bucket-name/machine_temperature_iot/test.csv'
output_data_s3_path = 's3://bucket-name/machine_temperature_iot/transform'
shingle_size = 288<jupyter_output><empty_output><jupyter_text>バッチ変換用にデータ形式を変更テスト用データのラベル部分を削除して、バッチ変換に使える形にした上で、S３にアップロードする<jupyter_code>import pandas as pd

transform_input_data_local_path = '/tmp/test_transform.csv'
pd.read_csv(labeled_test_data_s3_path, header=None)\
    .drop(columns=0)\
    .to_csv(transform_input_data_local_path, header=None, index=None)

from os import path
transform_input_data_s3_path = path.join(path.dirname(labeled_test_data_s3_path), 'test_transform.csv')
!aws s3 cp $transform_input_data_local_path $transform_input_data_s3_path<jupyter_output><empty_output><jupyter_text>バッチ変換<jupyter_code>from sagemaker.estimator import Estimator
model = Estimator.attach(training_job_name=training_job_name)

transformer = model.transformer(
    instance_count=1,
    instance_type='ml.m4.xlarge',
    strategy='MultiRecord',
    assemble_with='Line',
    output_path=output_data_s3_path
)
transformer.transform(
    transform_input_data_s3_path,
    content_type='text/csv',
    split_type='Line'
)
transformer.wait()<jupyter_output><empty_output><jupyter_text>推論結果を取得<jupyter_code>temp_path = '/tmp/transform_output/'
!aws s3 sync $transformer.output_path $temp_path
from os import path
# バッチ変換の結果ファイルは入力ファイル名に.outが付いている
output_path = path.join(temp_path, path.basename(transform_input_data_s3_path))+'.out'

!head $output_path
import json
with open(output_path) as f:
    lines = f.readlines()
    scores = list(map(lambda l : json.loads(l)['score'], lines))


import pandas as pd
df = pd.read_csv(labeled_test_data_s3_path, header=None)

# 異常度スコアを入れる
df_format =  pd.DataFrame(data={
    'is_anomaly': df.iloc[:, 0], # 異常値かどうか
    'value': df.iloc[:, shingle_size], # 温度の値
    'score': scores, # 異常度スコア
    'anomaly_threshold': [0]*len(df) # 異常度の閾値(この後計算する)
})<jupyter_output><empty_output><jupyter_text>異常度スコアの閾値を計算<jupyter_code>score_mean = df_format.score.mean()
score_std = df_format.score.std()
score_cutoff = score_mean + 1 * score_std
df_format['anomaly_threshold'] = pd.Series([score_cutoff]*len(df_format), df_format.index)<jupyter_output><empty_output><jupyter_text>結果を可視化<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# pyplotで描画する図を綺麗にする
sns.set()

# 図がインラインで描画されるようにする(JupyterLabでは不要)
%matplotlib inline

fig, ax1 = plt.subplots()
ax2 = ax1.twinx()

ax1.plot(df_format.value, color='C0', alpha=0.7)
ax2.plot(df_format.score, color='C1', alpha=0.7)

# 異常値のラベルデータ
anomalies_true = df_format[df_format.is_anomaly == 1]
ax1.plot(anomalies_true.value, 'ko' )

# 推論による異常値
anomalies_infer = df_format.score[df_format.score >= score_cutoff]
ax2.plot(anomalies_infer, 'ro' )
ax2.plot(df_format.anomaly_threshold, 'r', alpha=0.5)

ax1.grid(which='major', axis='both')
ax1.set_ylabel('Machine Temperature', color='C0')
ax2.set_ylabel('Anomaly Score', color='C1')
ax1.tick_params('y', colors='C0')
ax2.tick_params('y', colors='C1')
ax1.set_ylim(0, max(df_format.value))
ax2.set_ylim(min(df_format.score), 1.5*max(max(df_format.score), score_cutoff)+1)
fig.set_figwidth(12)
plt.show()<jupyter_output><empty_output>