<jupyter_start><jupyter_text>When I first ran this, my dataframes weren't "aligned".  So it's very important to check your datasets after every load.  The correspondence between dates and topics and numerical features is critical for training!<jupyter_code>print(len(dates))
print(len(topics))
print(len(nums))
print(sum(nums.favorite_count >= 1))
sum(nums.index == dates.index) == len(dates)
sum(nums.index == topics.index) == len(dates)
sgd = SGDRegressor()
sgd
sgd = SGDRegressor().fit(topics.values, nums.favorite_count)<jupyter_output><empty_output><jupyter_text>Well, that was **much** faster...<jupyter_code>predicted_favorites = sgd.predict(topics.values)
predicted_favorites
np.sum(predicted_favorites >= 1)<jupyter_output><empty_output><jupyter_text>Well that seems more "balanced" at least.  And it's nice to have a continuous score.<jupyter_code>np.sum(nums.favorite_count.values >= 1)
from pug.nlp.stats import Confusion
results = pd.DataFrame()
results['predicted'] = pd.Series(predicted_favorites >= 1)
results['truth'] = pd.Series(nums.favorite_count >= 1)
conf = Confusion(results)
conf
results.predicted.corr(results.truth)<jupyter_output><empty_output><jupyter_text>Wait, why are we classifying with a regressor anyway?<jupyter_code>pd.Series(predicted_favorites).corr(nums.favorite_count)<jupyter_output><empty_output><jupyter_text>Not so hot...Balance the training again?  Get rid of some negatives?<jupyter_code>pos = np.array(nums.favorite_count >= 1)
neg = ~pos
portion_pos = 2 * float(sum(pos)) / len(nums)
mask = ((np.random.binomial(1, portion_pos, size=len(nums)).astype(bool) & neg) | pos)
sgd = SGDRegressor().fit(topics[mask], nums.favorite_count[mask] >= 1)
print(portion_pos)
print(sum(mask))
print(sum(pos) * 2)
print(sum(neg))
len(nums)
results = pd.DataFrame()
# you have to adjust the threshold as you add more negatives
results['predicted'] = pd.Series(sgd.predict(topics.values) > .3)
results['truth'] = pd.Series(nums.favorite_count.values >= 1)
conf = Confusion(results)
conf
results.predicted.corr(results.truth)
conf.stats_dict<jupyter_output><empty_output><jupyter_text>So once again about 38% correlation is the best we can do...    Don't try this at home!  It never finished after 10 hours of running on my laptop<jupyter_code>svr = SVR().fit(topics.values, nums.favorite_count)<jupyter_output><empty_output>