<jupyter_start><jupyter_text>Understanding Embeddings on Texts<jupyter_code># Based on
# https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.2-understanding-recurrent-neural-networks.ipynb
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
%pylab inline
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
print(tf.__version__)
# https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification
max_features = 1000  # number of words to consider as features
maxlen = 20  # cut texts after this number of words (among top max_features most common words)

# each review is encoded as a sequence of word indexes
# indexed by overall frequency in the dataset
# output is 0 (negative) or 1 (positive) 
imdb = tf.keras.datasets.imdb.load_data(num_words=max_features)
(raw_input_train, y_train), (raw_input_test, y_test) = imdb
# tf.keras.datasets.imdb.load_data?
y_train.min()
y_train.max()
# 25000 texts
len(raw_input_train)
# first text has 218 words
len(raw_input_train[0])
raw_input_train[0]
# tf.keras.preprocessing.sequence.pad_sequences?
# https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences

input_train = tf.keras.preprocessing.sequence.pad_sequences(raw_input_train, maxlen=maxlen)
input_test = tf.keras.preprocessing.sequence.pad_sequences(raw_input_test, maxlen=maxlen)
input_train.shape, input_test.shape, y_train.shape, y_test.shape
# left padded with zeros
# As a convention, "0" does not stand for a specific word, but instead is used to encode any unknown word.
input_train[0]<jupyter_output><empty_output><jupyter_text>We can use a randomly initialized embedding without any training<jupyter_code># tf.keras.layers.Embedding?
embedding_dim = 3

random_model = tf.keras.Sequential()
# Parameters: max_features * embedding_dim 
random_model.add(tf.keras.layers.Embedding(name='embedding',input_dim=max_features, output_dim=embedding_dim, input_length=maxlen))

random_model.summary()
random_model.predict(input_train[:1])<jupyter_output><empty_output><jupyter_text>Training the embedding together with the whole model is more reasonableAlternative: use a pre-trained model, probably trained using skip-gram<jupyter_code>embedding_dim = 3

model = tf.keras.Sequential()
# Parameters: max_features * embedding_dim 
model.add(tf.keras.layers.Embedding(name='embedding', input_dim=max_features, output_dim=embedding_dim, input_length=maxlen))

# Output: maxlen * embedding_dim (8)
model.add(tf.keras.layers.Flatten(name='flatten'))

# binary classifier
model.add(tf.keras.layers.Dense(name='fc', units=32, activation='relu'))
model.add(tf.keras.layers.Dense(name='classifier', units=1, activation='sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model.summary()
batch_size = 128

%time history = model.fit(input_train, y_train, epochs=10, batch_size=batch_size, validation_split=0.2)
train_loss, train_accuracy = model.evaluate(input_train, y_train, batch_size=batch_size)
train_accuracy
test_loss, test_accuracy = model.evaluate(input_test, y_test, batch_size=batch_size)
test_accuracy
# precition
model.predict(input_test[0:5])
# ground truth
y_test[0:5]<jupyter_output><empty_output><jupyter_text>How does the output of the trained embedding look like?<jupyter_code>embedding_layer = model.get_layer('embedding')
model_stub= tf.keras.Model(inputs=model.input, outputs=embedding_layer.output)
embedding_prediction = model_stub.predict(input_test[0:5])
# 5 sample reviews, 500 words per review, 8 dimensions per word
embedding_prediction.shape
# 8 embedding dimensions of first word of first sample review
embedding_prediction[0][0]<jupyter_output><empty_output><jupyter_text>Comparing trained to untrained model<jupyter_code>input_train[0]
model_stub.predict(input_train[:1])
random_model.predict(input_train[:1])<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup