Initial commit

## Example files for the title:  

# Using Kudu with Apache Spark and Apache Flume, by Ryan Bosshart

[![Using Kudu with Apache Spark and Apache Flume, by Ryan Bosshart](http://akamaicovers.oreilly.com/images/9781491985717/cat.gif)](https://www.safaribooksonline.com/library/view/title/9781491985724//)

The following applies to example files from material published by O’Reilly Media, Inc. Content from other publishers may include different rules of usage. Please refer to any additional usage rights explained in the actual example files or refer to the publisher’s website.

O'Reilly books are here to help you get your job done. In general, you may use the code in O'Reilly books in your programs and documentation. You do not need to contact us for permission unless you're reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from our books does not require permission. Answering a question by citing our books and quoting example code does not require permission. On the other hand, selling or distributing a CD-ROM of examples from O'Reilly books does require permission. Incorporating a significant amount of example code from our books into your product's documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN.

If you think your use of code examples falls outside fair use or the permission given here, feel free to contact us at <permissions@oreilly.com>.

Please note that the examples are not production code and have not been carefully testing. They are provided "as-is" and come with no warranty of any kind.

### Integrating Kudu with Flume and Spark

Before running these examples, you'll need flume, git, and maven installed. You'll also need some sample data downloaded and staged in Kudu. There's a script in the project to automate this:  

    cd integrations/

    ./bootstrap.sh

About these examples: 

* Segment 1 demonstrates integrating Kudu with Apache Flume. The steps to get flume running are in `flume_example.sh`. 

* Segment 2 is the first of two segments looking at the integration of Kudu with Apache Spark. See code in `CreateTable.scala` and `SparkExample.scala`. 

* Segment 3 is the second Kudu-Spark segment, create a new Kudu table using Spark and then some fun with Spark MLLib. See `SparkExample.scala`. 

 No newline at end of file

Due to ongoing efforts to ensure you have the best possible learning experience, we may update the working files for this course from time to time.

We recommend that you download the latest version of these files from:

http://www.infiniteskills.com/02470

 No newline at end of file

#!/usr/bin/env bash

set -x

set -e

wget -P /tmp/ --no-check-certificate https://raw.githubusercontent.com/bosshart/kuduscreencast/master/api-sql-basics/user_ratings.txt

hadoop fs -ls /user/demo/

hadoop fs -mkdir -p /user/demo/moviedata/

hadoop fs -put /tmp/user_ratings.txt /user/demo/moviedata/

impala-shell -i localhost -d default -q "CREATE EXTERNAL TABLE user_item_ratings(

	ts INT,

	userid INT,

	movieid INT,

	rating INT,

	age INT,

	gender STRING,

	occupation STRING,

	zip INT,

	movietitle STRING,

	releasedate STRING,

	videoreleasedate STRING,

	url STRING)

    ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS TEXTFILE LOCATION '/user/demo/moviedata/';"

impala-shell -i localhost -d default -q "CREATE TABLE kudu_user_ratings PRIMARY KEY(movieid,userid)

PARTITION BY HASH(movieid)

PARTITIONS 4 STORED AS KUDU tblproperties('kudu.table_name'='kudu_user_ratings')

AS SELECT movieid, userid, rating, age, gender, occupation, zip, movietitle, releasedate, videoreleasedate, url, ts  FROM user_item_ratings;"

sudo echo "[cloudera-cdh5]

# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64

name=Cloudera's Distribution for Hadoop, Version 5

baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.10/

gpgkey = http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

gpgcheck = 1" > /etc/yum.repos.d/cloudera-cdh5.repo

# Need all kinds of dependencies to build the kudu client and flume jar

sudo yum -y install git flume-ng flume-ng-agent autoconf automake libtool gcc gcc-c++

wget -P /tmp/ https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz -O /tmp/protobuf-2.6.1.tar.gz

tar -xvf /tmp/protobuf-2.6.1.tar.gz -C /tmp/

cd /tmp/protobuf-2.6.1/

./configure

make

sudo make install

sudo service flume-ng-agent start

sudo service flume-ng-agent status

wget -P /tmp/ http://apache.cs.utah.edu/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

sudo tar xzf /tmp/apache-maven-3.3.9-bin.tar.gz -C /usr/local

cat <<'EOF'>> maven.sh

export M2_HOME=/usr/local/apache-maven-3.3.9

export PATH=${M2_HOME}/bin:${PATH}

EOF

sudo mv maven.sh /etc/profile.d/maven.sh

source /etc/profile

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.kuduscreencast</groupId>

  <artifactId>integrations</artifactId>

  <version>1.0-SNAPSHOT</version>

  <build>

    <plugins>

      <plugin>

        <artifactId>maven-compiler-plugin</artifactId>

        <version>3.6.0</version>

        <configuration>

          <source>1.7</source>

          <target>1.7</target>

        </configuration>

      </plugin>

      <plugin>

        <artifactId>maven-shade-plugin</artifactId>

        <version>1.7.1</version>

        <executions>

          <execution>

            <phase>package</phase>

            <goals>

              <goal>shade</goal>

            </goals>

          </execution>

        </executions>

        <configuration>

          <filters>

            <filter>

              <artifact>*:*</artifact>

              <excludes>

                <exclude>META-INF/*.SF</exclude>

                <exclude>META-INF/*.DSA</exclude>

                <exclude>META-INF/*.RSA</exclude>

              </excludes>

            </filter>

          </filters>

        </configuration>

      </plugin>

    </plugins>

  </build>

  <repositories>

    <repository>

      <snapshots>

        <enabled>false</enabled>

      </snapshots>

      <id>cdh.repo</id>

      <name>Cloudera Repositories</name>

      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>

    </repository>

  </repositories>

  <properties>

    <kudu.version>1.2.0</kudu.version>

    <log4j.version>4.10</log4j.version>

    <spark.version>1.6.0-cdh5.10.0</spark.version>

  </properties>

</project>