added ipython notebooks for IMDB example, and apologies

%% Cell type:markdown id: tags:

# Introduction

In this notebook, we'll take the existing files from last time and turn them into a pandas dataframe, then use Altair (=vega-lite) to explore a little

To install altair:

    $ conda install altair --channel conda-forge

4/26: for some reason, this is really incredibly slow. And there's a LOT of crappy movies here no one cares about

Worth noting: there are 279304 movies here. We could probably drop one or two without high cost

%% Cell type:code id: tags:

``` python

from altair import *

from math import *

import pandas as pd

import os

```

%% Cell type:code id: tags:

``` python

file = r"F:\danyelf\OneDrive - Microsoft\Projects\2016 Making Sense of Data\datasets\cleanratings.list"

data = pd.read_csv( file , sep='\t', encoding = 'Windows-1252')

```

%% Cell type:code id: tags:

``` python

data.head()

```

%% Cell type:markdown id: tags:

# Looking at the data

%% Cell type:code id: tags:

``` python

Chart(data).mark_bar().encode(

    x = X('raters', bin = Bin( maxbins = 20)),

    y = 'count(*):Q',

)

```

%% Cell type:markdown id: tags:

Looks like almost every rating is a very small number. What's the real curve look like? Can we get a bin?

%% Cell type:code id: tags:

``` python

data['lograters'] =  data['raters'].map(  lambda x : log(x))

Chart(data).mark_bar().encode(

    x = X('lograters', bin = Bin( maxbins = 20)),

    y = 'count(*):Q',

)

```

%% Cell type:code id: tags:

``` python

medianraters = data['raters'].median()

topquarter = data['raters'].quantile ( q= 0.75)

```

%% Cell type:code id: tags:

``` python

Chart(data).mark_bar().encode(

    x = X('score', bin = Bin( maxbins = 100)),

    y = 'count(*):Q',

)

```

%% Cell type:markdown id: tags:

Sanity check: is there a correlation between score & # of raters? That would seem to be biasing.

%% Cell type:code id: tags:

``` python

Chart(data).mark_circle().encode(

       x = X('score'), bin = Bin(maxbins = 10),

       y = Y('lograters', bin = Bin(maxbins = 10)),

       size = 'count(*):Q'

)

```

%% Cell type:code id: tags:

``` python

smalldata = data['score'] > topquarter

```

The ipython notebooks in this section are based on the versions of the IMDB data that was attainable by FTP as of mid-July 2017.

Since then, IMDB has changed to distributing this data via Amazon Web Services. I have not yet had an opportunity to update the notebooks. IMDB's license does not allow for redistribution of their data. My apologies for the rotten bits. -DAF

http://www.imdb.com/interfaces/

https://s3-ap-southeast-2.amazonaws.com/scico-labs/docs/lab-jupyter-aws.pdf

    { "name": "static", "value": true,

      "bind": {"input": "checkbox"} },

    { "description": "throttles the number of nodes rendered; 77 covers 'all'",

      "name": "numNodes", "value": 77 },

      "name": "numNodes", "value": 200 },

    {

      "description": "State variable for active node fix status.",

      "name": "fix", "value": 0,