Initial commit (b677301e) · Commits · examples / Analyzing Data Using Spark 20 DataFrames With Python

Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 1/No Work Files.rtf

0 → 100755

+7 −0

Changes for Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 1/No Work Files.rtf: 7 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
		{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
		{\colortbl;\red255\green255\blue255;}
		\margl1440\margr1440\vieww9000\viewh8400\viewkind0
		\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

		\f0\fs24 \cf0 No work files for this lesson...}
		No newline at end of file

Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 2/No Work Files.rtf

0 → 100755

+7 −0

Changes for Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 2/No Work Files.rtf: 7 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
		{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
		{\colortbl;\red255\green255\blue255;}
		\margl1440\margr1440\vieww9000\viewh8400\viewkind0
		\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

		\f0\fs24 \cf0 No work files for this lesson...}
		No newline at end of file

Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 3/A Review of Python.ipynb

0 → 100755

+1585 −0

File added.

Preview size limit exceeded, changes collapsed.

Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 3/Creating New Columns.ipynb

0 → 100755

+134 −0

Changes for Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 3/Creating New Columns.ipynb: 134 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		%% Cell type:markdown id: tags:

		# Creating New Columns

		%% Cell type:code id: tags:

		``` python
		from pyspark.sql import SparkSession
		```

		%% Cell type:code id: tags:

		``` python
		spark = SparkSession.builder.appName('basics').getOrCreate()
		```

		%% Cell type:code id: tags:

		``` python
		df = spark.read.json('people.json')
		```

		%% Cell type:code id: tags:

		``` python
		df.show()
		```

		%% Output

		+----+-------+
		\| age\| name\|
		+----+-------+
		\|null\|Michael\|
		\| 30\| Andy\|
		\| 19\| Justin\|
		+----+-------+


		%% Cell type:code id: tags:

		``` python
		df.withColumn('newage',df['age']).show()
		```

		%% Output

		+----+-------+------+
		\| age\| name\|newage\|
		+----+-------+------+
		\|null\|Michael\| null\|
		\| 30\| Andy\| 30\|
		\| 19\| Justin\| 19\|
		+----+-------+------+


		%% Cell type:code id: tags:

		``` python
		df.withColumnRenamed('age','supernewage').show()
		```

		%% Output

		+-----------+-------+
		\|supernewage\| name\|
		+-----------+-------+
		\| null\|Michael\|
		\| 30\| Andy\|
		\| 19\| Justin\|
		+-----------+-------+


		%% Cell type:code id: tags:

		``` python
		df.withColumn('doubleage',df['age']*2).show()
		```

		%% Output

		+----+-------+---------+
		\| age\| name\|doubleage\|
		+----+-------+---------+
		\|null\|Michael\| null\|
		\| 30\| Andy\| 60\|
		\| 19\| Justin\| 38\|
		+----+-------+---------+


		%% Cell type:code id: tags:

		``` python
		df.withColumn('add_one_age',df['age']+1).show()
		```

		%% Output

		+----+-------+-----------+
		\| age\| name\|add_one_age\|
		+----+-------+-----------+
		\|null\|Michael\| null\|
		\| 30\| Andy\| 31\|
		\| 19\| Justin\| 20\|
		+----+-------+-----------+


		%% Cell type:code id: tags:

		``` python
		mynew_age_df = df.withColumn('half_age',df['age']/2)
		```

		%% Cell type:code id: tags:

		``` python
		mynew_age_df.show()
		```

		%% Output

		+----+-------+--------+
		\| age\| name\|half_age\|
		+----+-------+--------+
		\|null\|Michael\| null\|
		\| 30\| Andy\| 15.0\|
		\| 19\| Justin\| 9.5\|
		+----+-------+--------+


		%% Cell type:code id: tags:

		``` python
		```

Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 3/Creating a DataFrame.ipynb

0 → 100755

+42 −0

Changes for Analyzing Data Using Spark 2.0 DataFrames With Python - Working Files/Chapter 3/Creating a DataFrame.ipynb: 42 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		%% Cell type:markdown id: tags:

		# Creating a DataFrame

		%% Cell type:code id: tags:

		``` python
		from pyspark.sql import SparkSession
		```

		%% Cell type:code id: tags:

		``` python
		spark = SparkSession.builder.appName("create").getOrCreate()
		```

		%% Cell type:code id: tags:

		``` python
		df = spark.createDataFrame([(0,'a'),(1,'b')],['id','letter'])
		```

		%% Cell type:code id: tags:

		``` python
		df.show()
		```

		%% Output

		+---+------+
		\| id\|letter\|
		+---+------+
		\| 0\| a\|
		\| 1\| b\|
		+---+------+


		%% Cell type:code id: tags:

		``` python
		```