<jupyter_start><jupyter_text>*Data Science Unit 4 Sprint 3 Assignment 1* Recurrent Neural Networks and Long Short Term Memory (LSTM)![Monkey at a typewriter](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Chimpanzee_seated_at_typewriter.jpg/603px-Chimpanzee_seated_at_typewriter.jpg)It is said that [infinite monkeys typing for an infinite amount of time](https://en.wikipedia.org/wiki/Infinite_monkey_theorem) will eventually type, among other things, the complete works of Wiliam Shakespeare. Let's see if we can get there a bit faster, with the power of Recurrent Neural Networks and LSTM.This text file contains the complete works of Shakespeare: https://www.gutenberg.org/files/100/100-0.txtUse it as training data for an RNN - you can keep it simple and train character level, and that is suggested as an initial approach.Then, use that trained RNN to generate Shakespearean-ish text. Your goal - a function that can take, as an argument, the size of text (e.g. number of characters or lines) to generate, and returns generated text of that size.Note - Shakespeare wrote an awful lot. It's OK, especially initially, to sample/use smaller data and parameters, so you can have a tighter feedback loop when you're trying to get things running. Then, once you've got a proof of concept - start pushing it more!<jupyter_code>!pip install tf-nightly-gpu
try:
  import tensorflow.compat.v2 as tf
except Exception:
  pass
tf.enable_v2_behavior()
print(tf.__version__)
from __future__ import print_function

from tensorflow.keras.callbacks import LambdaCallback
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding
from tensorflow.keras.layers import LSTM

import numpy as np
import random
import sys
import os
tf.test.gpu_device_name()
data_files = os.listdir('./shakespeare')
# Read in Data

data = []

for file in data_files:
    if file[-3:] == 'txt':
        with open(f'./shakespeare/{file}', 'r', encoding='utf-8') as f:
            data.append(f.read())
len(data)
data[0][:500]
# Encode data as characters
char_list = set()
for play in data:
    chars = set(play)
    char_list = char_list.union(chars)
    
char_list = list(char_list)
len(char_list)
# Lookup tables
char_int = {c:i for i, c in enumerate(char_list)}
int_char = {i:c for i, c in enumerate(char_list)}
# Create the sequence data

maxlen = 100
step = 1

sequences = []
next_char = []

for play in data:
    encoded = [char_int[c] for c in play]
    for i in range(0, len(encoded) - maxlen, step):
        sequences.append(encoded[i : i + maxlen])
        next_char.append(encoded[i + maxlen])
        
print('Number of sequences: ', len(sequences))
sequences[0][:50]
# Create x and y

x = np.zeros((len(sequences), maxlen, len(char_list)), dtype=np.bool)
y = np.zeros((len(sequences), len(char_list)), dtype=np.bool)

for i, sequence in enumerate(sequences):
    for t, char in enumerate(sequence):
        x[i, t, char] = 1
        
    y[i, next_char[i]] = 1
x.shape, y.shape
# Build the model: a single LSTM
model = Sequential()
model.add(LSTM(512, input_shape=(maxlen, len(char_list)), dropout=0.2))
model.add(Dense(len(char_list), activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()
def sample(preds):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / 1
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)
def on_epoch_end(epoch, _):
    # Function invoked at end of each epoch. Prints generated text.
    
    print()
    print('----- Generating text after Epoch: %d' % epoch)
    
    rand_play = random.randint(0, len(data) - 1)
    start_index = random.randint(0, len(data[rand_play]) - maxlen - 1)
    
    generated = ''
    
    sentence = data[rand_play][start_index: start_index + maxlen]
    generated += sentence
    
    print('----- Generating with seed: "' + sentence + '"')
    sys.stdout.write(generated)
    
    for i in range(1000):
        x_pred = np.zeros((1, maxlen, len(char_list)))
        for t, char in enumerate(sentence):
            x_pred[0, t, char_int[char]] = 1
            
        preds = model.predict(x_pred, verbose=0)[0]
        next_index = sample(preds)
        next_char = int_char[next_index]
        
        sentence = sentence[1:] + next_char
        
        sys.stdout.write(next_char)
        sys.stdout.flush()
    print()


print_callback = LambdaCallback(on_epoch_end=on_epoch_end)
# Fit the model
model.fit(x, y,
          batch_size=512,
          epochs=50,
          callbacks=[print_callback],
          shuffle=True)
import h5py
rand_play = random.randint(0, len(data) - 1)
start_index = random.randint(0, len(data[rand_play]) - maxlen - 1)
    
generated = ''
    
sentence = data[rand_play][start_index: start_index + maxlen]
    
sentence = "OTHELLO\nO my fair warrior!\n\nDESDEMONA\nMy dear Othello!\n\nOTHELLO\nIt gives me wonder great as my content"
sentence = sentence[:maxlen]
generated += sentence
    
print('----- Generating with seed: "' + sentence + '"')
print('-----------------')
sys.stdout.write(generated)
    
for _ in range(1000):
    x_pred = np.zeros((1, maxlen, len(char_list)))
    for t, char in enumerate(sentence):
        x_pred[0, t, char_int[char]] = 1
            
    preds = model.predict(x_pred, verbose=0)[0]
    next_index = sample(preds)
    next_char = int_char[next_index]
        
    sentence = sentence[1:] + next_char
        
    sys.stdout.write(next_char)
    sys.stdout.flush()
model.save('shakespeare_lstm_512.h5')<jupyter_output><empty_output>