Initial commit

%% Cell type:markdown id: tags:

# PySpark

%% Cell type:markdown id: tags:

Installing Python and Spark

Spark Fundamentals

Transformations

Actions

Key-Value Pair RDDs

Input/Output

Performance

Running on a Cluster

Advanced Spark

%% Cell type:code id: tags:

``` python

from pyspark.sql import SQLContext

```

%% Cell type:code id: tags:

``` python

sqlContext = SQLContext(sc)

```

%% Cell type:code id: tags:

``` python

s = sqlContext.read.json("sample.json")

```

%% Cell type:code id: tags:

``` python

s.printSchema()

```

%% Output

    root

     |-- id: long (nullable = true)

     |-- name: string (nullable = true)

     |-- price: double (nullable = true)

%% Cell type:code id: tags:

``` python

s.show()

```

%% Output

    +---+-------------+-----+

    | id|         name|price|

    +---+-------------+-----+

    |  1| A green door| 12.5|

    |  2|A yellow door|  1.5|

    |  3|  A blue door| 11.5|

    |  4|   A red door|120.5|

    +---+-------------+-----+

%% Cell type:code id: tags:

``` python

s.filter(s['price'] > 12).select(s['name']).show()

```

%% Output

    +------------+

    |        name|

    +------------+

    |A green door|

    |  A red door|

    +------------+

%% Cell type:code id: tags:

``` python

s.registerTempTable("sample")

```

%% Cell type:code id: tags:

``` python

sqlContext.sql("SELECT name FROM sample WHERE price >= 12").collect()

```

%% Output

    [Row(name=u'A green door'), Row(name=u'A red door')]

%% Cell type:code id: tags:

``` python

```

%% Cell type:raw id: tags:

spark.mllib - OLD, low level

%% Cell type:raw id: tags:

spark.ml - NEW, higher level, works with dataframes

%% Cell type:code id: tags:

``` python

from pyspark.streaming import StreamingContext

```

%% Cell type:code id: tags:

``` python

ssc = StreamingContext(sc, 1)

```

%% Cell type:code id: tags:

``` python

input_stream = ssc.textFileStream("sampleTextDir")

```

%% Cell type:code id: tags:

``` python

word_pairs = input_stream.flatMap(

    lambda l:l.split(" ")).map(lambda w: (w,1))

```

%% Cell type:code id: tags:

``` python

counts = word_pairs.reduceByKey(lambda x,y: x + y)

```

%% Cell type:code id: tags:

``` python

counts.pprint()

```

%% Cell type:code id: tags:

``` python

ssc.start()

```

%% Cell type:code id: tags:

``` python

```

{ "id": 1, "name": "A green door", "price": 12.50 }

{ "id": 2, "name": "A yellow door", "price": 1.50 }

{ "id": 3, "name": "A blue door", "price": 11.50 }

{ "id": 4, "name": "A red door", "price": 120.50 }