Initial commit

%% Cell type:markdown id: tags:

# Mining the Social Web

## Mining Mailboxes

This Jupyter Notebook provides an interactive way to follow along with and explore the examples from the video series. The intent behind this notebook is to reinforce the concepts in a fun, convenient, and effective way.

%% Cell type:markdown id: tags:

## Converting a toy mailbox to JSON

%% Cell type:code id: tags:

``` python

import mailbox # pip install mailbox

import json

```

%% Cell type:code id: tags:

``` python

MBOX = 'data/northpole.mbox'

# Create an mbox that can be iterated over and transform each of its

# messages to a convenient JSON representation

mbox = mailbox.mbox(MBOX)

for msg in mbox:

    part = [p for p in msg.walk()][0]

    print(part)

```

%% Cell type:code id: tags:

``` python

# A routine that makes a ton of simplifying assumptions

# about converting an mbox message into a Python object

# given the nature of the northpole.mbox file in order

# to demonstrate the basic parsing of an mbox with mail

# utilities

def objectify_message(msg):

    # Map in fields from the message

    o_msg = dict([ (k, v) for (k,v) in msg.items() ])

    # Assume one part to the message and get its content

    # and its content type

    part = [p for p in msg.walk()][0]

    o_msg['contentType'] = part.get_content_type()

    o_msg['content'] = part.get_payload()

    return o_msg

```

%% Cell type:code id: tags:

``` python

messages = []

for msg in mbox:

    messages.append(objectify_message(msg))

print(json.dumps(messages, indent=1))

```

%% Cell type:markdown id: tags:

## Downloading the Enron email corpus

%% Cell type:code id: tags:

``` python

import sys

from urllib.request import urlopen

import time

import os

import envoy # pip install envoy

URL = "http://www.cs.cmu.edu/~enron/enron_mail_20110402.tgz"

DOWNLOAD_DIR = "data"

# Downloads a file and displays a download status every 5 seconds

def download(url, download_dir):

    file_name = url.split('/')[-1]

    u = urlopen(url)

    f = open(os.path.join(download_dir, file_name), 'wb')

    meta = u.info()

    file_size = int(meta['Content-Length'])

    print("Downloading: %s Bytes: %s" % (file_name, file_size))

    file_size_dl = 0

    block_sz = 8192

    last_update = time.time()

    while True:

        buffer = u.read(block_sz)

        if not buffer:

            break

        file_size_dl += len(buffer)

        f.write(buffer)

        download_status = r"%10d MB  [%3.2f%%]" % (file_size_dl / 1000000.0, file_size_dl * 100.0 / file_size)

        download_status = download_status + chr(8)*(len(download_status)+1)

        if time.time() - last_update > 5:

            print(download_status)

            sys.stdout.flush()

            last_update = time.time()

    f.close()

    return f.name

# Extracts a gzipped tarfile. e.g. "$ tar xzf filename.tgz"

def tar_xzf(f):

    # Call out to the shell for a faster decompression.

    # This will still take a while because Vagrant synchronizes

    # thousands of files that are extracted to the host machine

    r = envoy.run("tar xzf %s -C %s" % (f, DOWNLOAD_DIR))

    print(r.std_out)

    print(r.std_err)

f = download(URL, DOWNLOAD_DIR)

print("Download complete: %s" % (f,))

tar_xzf(f)

print("Decompression complete")

print("Data is ready")

```

%% Cell type:markdown id: tags:

## Converting the Enron corpus to a standardized mbox format

%% Cell type:code id: tags:

``` python

import re

import email

from time import asctime

import os

import sys

from dateutil.parser import parse # pip install python_dateutil

# Download the Enron corpus to resources/ch06-mailboxes/data

# and unarchive it there.

MAILDIR = 'data/enron_mail_20110402/maildir'

# Where to write the converted mbox

MBOX = 'data/enron.mbox'

# Create a file handle that we'll be writing into...

mbox = open(MBOX, 'w+')

# Walk the directories and process any folder named 'inbox'

for (root, dirs, file_names) in os.walk(MAILDIR):

    if root.split(os.sep)[-1].lower() != 'inbox':

        continue

    # Process each message in 'inbox'

    for file_name in file_names:

        file_path = os.path.join(root, file_name)

        message_text = open(file_path, errors='ignore').read()

        # Compute fields for the From_ line in a traditional mbox message

        _from = re.search(r"From: ([^\r\n]+)", message_text).groups()[0]

        _date = re.search(r"Date: ([^\r\n]+)", message_text).groups()[0]

        # Convert _date to the asctime representation for the From_ line

        _date = asctime(parse(_date).timetuple())

        msg = email.message_from_string(message_text)

        msg.set_unixfrom('From {0} {1}'.format(_from, _date))

        mbox.write(msg.as_string(unixfrom=True) + "\n\n")

mbox.close()

```

%% Cell type:markdown id: tags:

## Loading the mailbox data into Pandas

%% Cell type:code id: tags:

``` python

import pandas as pd # pip install pandas

import mailbox

MBOX = 'data/enron.mbox'

mbox = mailbox.mbox(MBOX)

mbox_dict = {}

for i, msg in enumerate(mbox):

    mbox_dict[i] = {}

    for header in msg.keys():

        mbox_dict[i][header] = msg[header]

    mbox_dict[i]['Body'] = msg.get_payload().replace('\n', ' ').replace('\t', ' ').replace('\r', ' ').strip()

```

%% Cell type:code id: tags:

``` python

df = pd.DataFrame.from_dict(mbox_dict, orient='index')

df.head()

```

%% Cell type:code id: tags:

``` python

df.index = df['Date'].apply(pd.to_datetime)

# Remove non-essential columns

cols_to_keep = ['From', 'To', 'Cc', 'Bcc', 'Subject', 'Body']

df = df[cols_to_keep]

```

%% Cell type:code id: tags:

``` python

df.head()

```

%% Cell type:markdown id: tags:

## Describe the DataFrame

%% Cell type:code id: tags:

``` python

df.describe()

```

%% Cell type:markdown id: tags:

## Investigate email volume by month

%% Cell type:code id: tags:

``` python

start_date = '2000-1-1'

stop_date = '2003-1-1'

datemask = (df.index > start_date) & (df.index <= stop_date)

vol_by_month = df.loc[datemask].resample('1M').count()['To']

vol_by_month.head()

```

%% Cell type:code id: tags:

``` python

from prettytable import PrettyTable

pt = PrettyTable(field_names=['Year', 'Month', 'Num Msgs'])

pt.align['Num Msgs'], pt.align['Month'] = 'r', 'r'

[ pt.add_row([ind.year, ind.month, vol])

  for ind, vol in zip(vol_by_month.index, vol_by_month)]

print(pt)

```

%% Cell type:code id: tags:

``` python

import matplotlib.pyplot as plt

%matplotlib inline

vol_by_month[::-1].plot(kind='barh', figsize=(5,8), title='Email Volume by Month')

```

%% Cell type:markdown id: tags:

## Searching by keyword

%% Cell type:code id: tags:

``` python

search_term = 'raptor'

query = (df['Body'].str.contains(search_term, case=False) | df['Subject'].str.contains(search_term, case=False))

print('{0} results found.'.format(query.sum()))

```

%% Cell type:code id: tags:

``` python

results = df[query]

results.head()

```

%% Cell type:code id: tags:

``` python

print(results.iloc[0]['Subject'])

print()

print(results.iloc[0]['Body'])

```

%% Cell type:markdown id: tags:

## See who is sending and receiving the most email

%% Cell type:code id: tags:

``` python

senders = df.groupby('From')

receivers = df.groupby('To')

```

%% Cell type:code id: tags:

``` python

import numpy as np

senders = senders.count()['To']

receivers = receivers.count()['From']

# Get the ordered indices of the top senders and receivers in descending order

top_snd_ord = np.argsort(senders)[::-1]

top_rcv_ord = np.argsort(receivers)[::-1]

```

%% Cell type:code id: tags:

``` python

top_senders = senders[top_snd_ord]

top_receivers = receivers[top_rcv_ord]

```

%% Cell type:code id: tags:

``` python

from prettytable import PrettyTable

top10 = top_senders[:10]

pt = PrettyTable(field_names=['Rank', 'Sender', 'Messages Sent'])

pt.align['Messages Sent'] = 'r'

[ pt.add_row([i+1, email, vol]) for i, email, vol in zip(range(10), top10.index.values, top10.values)]

print(pt)

```

%% Cell type:code id: tags:

``` python

from prettytable import PrettyTable

top10 = top_receivers[:10]

pt = PrettyTable(field_names=['Rank', 'Receiver', 'Messages Received'])

pt.align['Messages Sent'] = 'r'

[ pt.add_row([i+1, email, vol]) for i, email, vol in zip(range(10), top10.index.values, top10.values)]

print(pt)

```

From santa@northpole.example.org Fri Dec 25 00:06:42 2009

Message-ID: <16159836.1075855377439@mail.northpole.example.org>

References: <88364590.8837464573838@mail.northpole.example.org>

In-Reply-To: <194756537.0293874783209@mail.northpole.example.org>

Date: Fri, 25 Dec 2001 00:06:42 -0000 (GMT)

From: St. Nick <santa@northpole.example.org>

To: rudolph@northpole.example.org

Subject: RE: FWD: Tonight

Mime-Version: 1.0

Content-Type: text/plain; charset=us-ascii

Content-Transfer-Encoding: 7bit

Sounds good. See you at the usual location.

Thanks,

-S

 -----Original Message-----

From:   Rudolph  

Sent:   Friday, December 25, 2009 12:04 AM

To: Claus, Santa

Subject:    FWD:  Tonight

Santa -

Running a bit late. Will come grab you shortly. Standby.

Rudy

Begin forwarded message:

> Last batch of toys was just loaded onto sleigh. 

>

> Please proceed per the norm.

>

> Regards,

> Buddy

>

> --

> Buddy the Elf

> Chief Elf

> Workshop Operations

> North Pole

> buddy.the.elf@northpole.example.org

From buddy.the.elf@northpole.example.org Fri Dec 25 00:03:34 2009

Message-ID: <88364590.8837464573838@mail.northpole.example.org>

Date: Fri, 25 Dec 2001 00:03:34 -0000 (GMT)

From: Buddy <buddy.the.elf@northpole.example.org>

To: workshop@northpole.example.org

Subject: Tonight

Mime-Version: 1.0

Content-Type: text/plain; charset=us-ascii

Content-Transfer-Encoding: 7bit

Last batch of toys was just loaded onto sleigh. 

Please proceed per the norm.

Regards,

Buddy

--

Buddy the Elf

Chief Elf

Workshop Operations

North Pole

buddy.the.elf@northpole.example.org

## Example files for the title:  

# Mining the Social Web - Mailboxes, by Mikhail Klassen

[![Mining the Social Web - Mailboxes, by Mikhail Klassen](http://akamaicovers.oreilly.com/images/9781491989890/cat.gif)](https://www.safaribooksonline.com/)

The following applies to example files from material published by O’Reilly Media, Inc. Content from other publishers may include different rules of usage. Please refer to any additional usage rights explained in the actual example files or refer to the publisher’s website.

O'Reilly books are here to help you get your job done. In general, you may use the code in O'Reilly books in your programs and documentation. You do not need to contact us for permission unless you're reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from our books does not require permission. Answering a question by citing our books and quoting example code does not require permission. On the other hand, selling or distributing a CD-ROM of examples from O'Reilly books does require permission. Incorporating a significant amount of example code from our books into your product's documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN.

If you think your use of code examples falls outside fair use or the permission given here, feel free to contact us at <permissions@oreilly.com>.

Please note that the examples are not production code and have not been carefully testing. They are provided "as-is" and come with no warranty of any kind.