add NLP best practices (aab5ed0b) · Commits · live-training / deep-learning

notebooks/natural_language_preprocessing_best_practices_LT.ipynb

0 → 100644

+447 −0

Changes for notebooks/natural_language_preprocessing_best_practices_LT.ipynb: 447 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		%% Cell type:markdown id: tags:

		# Best Practices for Preprocessing Natural Language Data

		%% Cell type:markdown id: tags:

		In this notebook, we improve the quality of our Project Gutenberg word vectors by adopting best-practices for preprocessing natural language data.

		N.B.: Some, all or none of these preprocessing steps may be helpful to a given downstream application.

		%% Cell type:markdown id: tags:

		#### Load dependencies

		%% Cell type:code id: tags:

		``` python
		# the initial block is copied from creating_word_vectors_with_word2vec.ipynb
		import nltk
		from nltk import word_tokenize, sent_tokenize
		from nltk.corpus import stopwords
		from nltk.stem.porter import *
		import gensim
		from gensim.models.word2vec import Word2Vec
		from gensim.models.phrases import Phraser, Phrases
		from sklearn.manifold import TSNE
		import pandas as pd
		from bokeh.io import output_notebook, output_file
		from bokeh.plotting import show, figure
		import string
		%matplotlib inline
		```

		%% Cell type:code id: tags:

		``` python
		nltk.download('punkt')
		```

		%% Cell type:code id: tags:

		``` python
		nltk.download('stopwords')
		```

		%% Cell type:markdown id: tags:

		#### Load data

		%% Cell type:code id: tags:

		``` python
		nltk.download('gutenberg')
		```

		%% Cell type:code id: tags:

		``` python
		from nltk.corpus import gutenberg
		```

		%% Cell type:code id: tags:

		``` python
		len(gutenberg.fileids())
		```

		%% Cell type:code id: tags:

		``` python
		gutenberg.fileids()
		```

		%% Cell type:code id: tags:

		``` python
		gberg_sent_tokens = sent_tokenize(gutenberg.raw())
		```

		%% Cell type:code id: tags:

		``` python
		gberg_sent_tokens[0:5]
		```

		%% Cell type:code id: tags:

		``` python
		gberg_sent_tokens[1]
		```

		%% Cell type:code id: tags:

		``` python
		word_tokenize(gberg_sent_tokens[1])
		```

		%% Cell type:code id: tags:

		``` python
		word_tokenize(gberg_sent_tokens[1])[14]
		```

		%% Cell type:code id: tags:

		``` python
		gberg_sents = gutenberg.sents()
		```

		%% Cell type:code id: tags:

		``` python
		gberg_sents[0:5]
		```

		%% Cell type:code id: tags:

		``` python
		len(gutenberg.words())
		```

		%% Cell type:markdown id: tags:

		#### Iteratively preprocess a sentence

		%% Cell type:markdown id: tags:

		##### a tokenized sentence:

		%% Cell type:code id: tags:

		``` python
		gberg_sents[4]
		```

		%% Cell type:markdown id: tags:

		##### to lowercase:

		%% Cell type:code id: tags:

		``` python
		# CODE HERE
		```

		%% Cell type:markdown id: tags:

		##### remove stopwords and punctuation:

		%% Cell type:code id: tags:

		``` python
		stpwrds = stopwords.words('english') + list(string.punctuation)
		```

		%% Cell type:code id: tags:

		``` python
		stpwrds
		```

		%% Cell type:code id: tags:

		``` python
		# CODE HERE
		```

		%% Cell type:markdown id: tags:

		##### stem words:

		%% Cell type:code id: tags:

		``` python
		stemmer = PorterStemmer()
		```

		%% Cell type:code id: tags:

		``` python
		# CODE HERE
		```

		%% Cell type:markdown id: tags:

		##### handle bigram collocations:

		%% Cell type:code id: tags:

		``` python
		phrases = Phrases(gberg_sents) # train detector
		```

		%% Cell type:code id: tags:

		``` python
		bigram = Phraser(phrases) # create a more efficient Phraser object for transforming sentences
		```

		%% Cell type:code id: tags:

		``` python
		bigram.phrasegrams # output count and score of each bigram
		```

		%% Cell type:code id: tags:

		``` python
		"Jon lives in New York City".split()
		```

		%% Cell type:code id: tags:

		``` python
		# CODE HERE
		```

		%% Cell type:markdown id: tags:

		#### Preprocess the corpus

		%% Cell type:code id: tags:

		``` python
		lower_sents = []
		for s in gberg_sents:
		lower_sents.append( # CODE HERE )
		```

		%% Cell type:code id: tags:

		``` python
		lower_sents[0:5]
		```

		%% Cell type:code id: tags:

		``` python
		lower_bigram = Phraser(Phrases(lower_sents))
		```

		%% Cell type:code id: tags:

		``` python
		lower_bigram.phrasegrams # miss taylor, mr woodhouse, mr weston
		```

		%% Cell type:code id: tags:

		``` python
		lower_bigram["jon lives in new york city".split()]
		```

		%% Cell type:code id: tags:

		``` python
		lower_bigram = Phraser(Phrases(lower_sents, min_count=32, threshold=64))
		lower_bigram.phrasegrams
		```

		%% Cell type:code id: tags:

		``` python
		# as in Maas et al. (2001):
		# - leave in stop words ("indicative of sentiment")
		# - no stemming ("model learns similar representations of words of the same stem when data suggests it")
		clean_sents = []
		for s in lower_sents:
		clean_sents.append( # CODE HERE )
		```

		%% Cell type:code id: tags:

		``` python
		clean_sents[0:9]
		```

		%% Cell type:code id: tags:

		``` python
		clean_sents[6] # could consider removing stop words or common words
		```

		%% Cell type:markdown id: tags:

		#### Run word2vec

		%% Cell type:code id: tags:

		``` python
		# max_vocab_size can be used instead of min_count (which has increased here)
		# model = Word2Vec(sentences=clean_sents, size=64, sg=1, window=10, min_count=10, seed=42, workers=8)
		# model.save('../clean_gutenberg_model.w2v')
		```

		%% Cell type:markdown id: tags:

		#### Explore model

		%% Cell type:code id: tags:

		``` python
		# skip re-training the model with the next line:
		model = gensim.models.Word2Vec.load('../clean_gutenberg_model.w2v')
		```

		%% Cell type:code id: tags:

		``` python
		len(model.wv.vocab) # 17k with raw data
		```

		%% Cell type:code id: tags:

		``` python
		len(model['dog'])
		```

		%% Cell type:code id: tags:

		``` python
		model['dog']
		```

		%% Cell type:code id: tags:

		``` python
		model.most_similar('dog')
		```

		%% Cell type:code id: tags:

		``` python
		model.most_similar('think')
		```

		%% Cell type:code id: tags:

		``` python
		model.most_similar('day')
		```

		%% Cell type:code id: tags:

		``` python
		model.doesnt_match("morning afternoon evening dog".split())
		```

		%% Cell type:code id: tags:

		``` python
		model.similarity('morning', 'dog')
		```

		%% Cell type:code id: tags:

		``` python
		model.most_similar('ma_am')
		```

		%% Cell type:code id: tags:

		``` python
		model.most_similar(positive=['father', 'woman'], negative=['man'])
		```

		%% Cell type:markdown id: tags:

		#### Reduce word vector dimensionality with t-SNE

		%% Cell type:code id: tags:

		``` python
		# tsne = TSNE(n_components=2, n_iter=1000)
		```

		%% Cell type:code id: tags:

		``` python
		# X_2d = tsne.fit_transform(model[model.wv.vocab])
		```

		%% Cell type:code id: tags:

		``` python
		# coords_df = pd.DataFrame(X_2d, columns=['x','y'])
		# coords_df['token'] = model.wv.vocab.keys()
		```

		%% Cell type:code id: tags:

		``` python
		# coords_df.to_csv('../clean_gutenberg_tsne.csv', index=False)
		```

		%% Cell type:markdown id: tags:

		#### Visualise

		%% Cell type:code id: tags:

		``` python
		coords_df = pd.read_csv('../clean_gutenberg_tsne.csv')
		```

		%% Cell type:code id: tags:

		``` python
		coords_df.head()
		```

		%% Cell type:code id: tags:

		``` python
		_ = coords_df.plot.scatter('x', 'y', figsize=(12,12), marker='.', s=10, alpha=0.2)
		```

		%% Cell type:code id: tags:

		``` python
		output_notebook()
		```

		%% Cell type:code id: tags:

		``` python
		subset_df = coords_df.sample(n=5000)
		```

		%% Cell type:code id: tags:

		``` python
		p = figure(plot_width=800, plot_height=800)
		_ = p.text(x=subset_df.x, y=subset_df.y, text=subset_df.token)
		```

		%% Cell type:code id: tags:

		``` python
		show(p)
		```

		%% Cell type:code id: tags:

		``` python
		# output_file() here
		```