<jupyter_start><jupyter_text>Histogram / Density plot of nr of trees per ensemble<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt

from arboreto.core import *
XGB_KWARGS
meta_df = pd.read_csv('/home/tmo/work/kuleuven/papers/arboreto/meta_out_macosko_40k.tsv', sep='\t')
meta_df.head()
df = macosko_df

all_zeros = df.loc[:, (df == 0).all()]
set(all_zeros.columns)
macosko_df = pd.read_csv('/media/tmo/data/work/datasets/macosko/in/macosko_40k.tsv.gz', sep='\t')<jupyter_output><empty_output><jupyter_text>* the targets with 5000 estimators are the all-zero columns! * these targets should have been filtered out before proceeding the analysis<jupyter_code>clean_df = meta_df[meta_df['n_estimators'] < 2000]
fig, ax = plt.subplots()
clean_df.n_estimators.plot.hist(bins=1000, ax=ax, figsize=(12,8))
ax.set_yscale('log')
ax.set_xlabel('Number of estimators')
plt.show()
len(clean_df.n_estimators)
clean_df.n_estimators.sum() / (len(clean_df.n_estimators) * 1000)
clean_df
nonzero_df = pd.DataFrame(macosko_df.astype(bool).sum(axis=0)).reset_index()
nonzero_df.columns = ['target', 'n_nonzero']
nonzero_df
fig, ax = plt.subplots()
clean_df.n_estimators.plot.hist(bins=1000, ax=ax, figsize=(12,8))
ax.set_yscale('log')
ax.set_xlabel('Number of estimators')
plt.show()
fig, ax = plt.subplots()
nonzero_df.merge(clean_df[['target', 'n_estimators']]).plot.scatter(ax=ax, x='n_nonzero', y='n_estimators', figsize=(12,8), alpha=.05)

ax.set_xscale('log')
#ax.set_yscale('log')
plt.show()
nonzero_df<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup