add dense sentiment classifier

%% Cell type:markdown id: tags:

# Dense Sentiment Classifier

%% Cell type:markdown id: tags:

In this notebook, we build a dense neural net to classify IMDB movie reviews by their sentiment.

%% Cell type:markdown id: tags:

#### Load dependencies

%% Cell type:code id: tags:

``` python

import keras

from keras.datasets import imdb

from keras.preprocessing.sequence import pad_sequences

from keras.models import Sequential

from keras.layers import Dense, Flatten, Dropout

from keras.layers import Embedding # new!

from keras.callbacks import ModelCheckpoint # new!

import os # new!

from sklearn.metrics import roc_auc_score, roc_curve # new!

import pandas as pd

import matplotlib.pyplot as plt # new!

%matplotlib inline

```

%% Cell type:markdown id: tags:

#### Set hyperparameters

%% Cell type:code id: tags:

``` python

# output directory name:

output_dir = 'model_output/dense'

# training:

epochs = 4

batch_size = 128

# vector-space embedding:

n_dim = 64

n_unique_words = 5000 # as per Maas et al. (2011); may not be optimal

n_words_to_skip = 50 # ditto

max_review_length = 100

pad_type = trunc_type = 'pre'

# neural network architecture:

n_dense = 64

dropout = 0.5

```

%% Cell type:markdown id: tags:

#### Load data

%% Cell type:markdown id: tags:

For a given data set:

* the Keras text utilities [here](https://keras.io/preprocessing/text/) quickly preprocess natural language and convert it into an index

* the `keras.preprocessing.text.Tokenizer` class may do everything you need in one line:

    * tokenize into words or characters

    * `num_words`: maximum unique tokens

    * filter out punctuation

    * lower case

    * convert words to an integer index

%% Cell type:code id: tags:

``` python

(x_train, y_train), (x_valid, y_valid) = imdb.load_data(num_words=n_unique_words, skip_top=n_words_to_skip)

```

%% Cell type:code id: tags:

``` python

x_train[0:6] # 0 reserved for padding; 1 would be starting character; 2 is unknown; 3 is most common word, etc.

```

%% Cell type:code id: tags:

``` python

for x in x_train[0:6]:

    print(len(x))

```

%% Cell type:code id: tags:

``` python

y_train[0:6]

```

%% Cell type:code id: tags:

``` python

len(x_train), len(x_valid)

```

%% Cell type:markdown id: tags:

#### Restoring words from index

%% Cell type:code id: tags:

``` python

word_index = keras.datasets.imdb.get_word_index()

word_index = {k:(v+3) for k,v in word_index.items()}

word_index["PAD"] = 0

word_index["START"] = 1

word_index["UNK"] = 2

```

%% Cell type:code id: tags:

``` python

word_index

```

%% Cell type:code id: tags:

``` python

index_word = {v:k for k,v in word_index.items()}

```

%% Cell type:code id: tags:

``` python

x_train[0]

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in x_train[0])

```

%% Cell type:code id: tags:

``` python

(all_x_train,_),(all_x_valid,_) = imdb.load_data()

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in all_x_train[0])

```

%% Cell type:markdown id: tags:

#### Preprocess data

%% Cell type:code id: tags:

``` python

x_train = pad_sequences(x_train, maxlen=max_review_length, padding=pad_type, truncating=trunc_type, value=0)

x_valid = pad_sequences(x_valid, maxlen=max_review_length, padding=pad_type, truncating=trunc_type, value=0)

```

%% Cell type:code id: tags:

``` python

x_train[0:6]

```

%% Cell type:code id: tags:

``` python

for x in x_train[0:6]:

    print(len(x))

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in x_train[0])

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in x_train[5])

```

%% Cell type:markdown id: tags:

#### Design neural network architecture

%% Cell type:code id: tags:

``` python

# CODE HERE

```

%% Cell type:code id: tags:

``` python

model.summary() # so many parameters!

```

%% Cell type:code id: tags:

``` python

# embedding layer dimensions and parameters:

n_dim, n_unique_words, n_dim*n_unique_words

```

%% Cell type:code id: tags:

``` python

# ...flatten:

max_review_length, n_dim, n_dim*max_review_length

```

%% Cell type:code id: tags:

``` python

# ...dense:

n_dense, n_dim*max_review_length*n_dense + n_dense # weights + biases

```

%% Cell type:code id: tags:

``` python

# ...and output:

n_dense + 1

```

%% Cell type:markdown id: tags:

#### Configure model

%% Cell type:code id: tags:

``` python

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

```

%% Cell type:code id: tags:

``` python

modelcheckpoint = ModelCheckpoint(filepath=output_dir+"/weights.{epoch:02d}.hdf5")

```

%% Cell type:code id: tags:

``` python

if not os.path.exists(output_dir):

    os.makedirs(output_dir)

```

%% Cell type:markdown id: tags:

#### Train!

%% Cell type:code id: tags:

``` python

# 84.7% validation accuracy in epoch 2

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_valid, y_valid), callbacks=[modelcheckpoint])

```

%% Cell type:markdown id: tags:

#### Evaluate

%% Cell type:code id: tags:

``` python

model.load_weights(output_dir+"/weights.01.hdf5") # zero-indexed

```

%% Cell type:code id: tags:

``` python

y_hat = model.predict_proba(x_valid)

```

%% Cell type:code id: tags:

``` python

len(y_hat)

```

%% Cell type:code id: tags:

``` python

y_hat[0]

```

%% Cell type:code id: tags:

``` python

plt.hist(y_hat)

_ = plt.axvline(x=0.5, color='orange')

```

%% Cell type:code id: tags:

``` python

pct_auc = roc_auc_score(y_valid, y_hat)*100.0

```

%% Cell type:code id: tags:

``` python

"{:0.2f}".format(pct_auc)

```

%% Cell type:code id: tags:

``` python

float_y_hat = []

for y in y_hat:

    float_y_hat.append(y[0])

```

%% Cell type:code id: tags:

``` python

ydf = pd.DataFrame(list(zip(float_y_hat, y_valid)), columns=['y_hat', 'y'])

```

%% Cell type:code id: tags:

``` python

ydf.head(10)

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in all_x_valid[0])

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in all_x_valid[6])

```

%% Cell type:code id: tags:

``` python

ydf[(ydf.y == 0) & (ydf.y_hat > 0.9)].head(10)

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in all_x_valid[489])

```

%% Cell type:code id: tags:

``` python

ydf[(ydf.y == 1) & (ydf.y_hat < 0.1)].head(10)

```

%% Cell type:code id: tags:

``` python

' '.join(index_word[id] for id in all_x_valid[927])

```

%% Cell type:code id: tags:

``` python

```