Commit 62d7025c authored by O'Reilly Media, Inc.'s avatar O'Reilly Media, Inc.
Browse files

Initial commit

parents

Building a Near Real-Time Analytical Application with Kudu - Working Files/README.md

0 → 100755
+117 −0
Changes for Building a Near Real-Time Analytical Application with Kudu - Working Files/README.md: 117 added lines, 0 removed lines.
Original line number Diff line number Diff line 
### Realtime Timeseries Example using "KIKS Stack" (Apache Kafka, Impala, Kudu, Spark Streaming)



This example shows how to build and run a project that demonstrates realtime data ingest, processing, and query using Apache Kudu. The project uses a [FIX message generator](https://github.com/cloudera-labs/envelope/tree/master/examples/fix) to simulate the continious creation of Fix financial messages which are published to Kafka and then persisted to Kudu using a Spark Streaming job. Once the FIX data is persisted into Kudu, a small web application uses Impala (via JDBC) to feed a realtime chart that analyzes peak order activity in near-realtime. 



This example was tested using Kudu version 1.1, Kafka version 0.9.0 (included with Cloudera Distribution of Kafka version 2.0), Spark 1.6 and Impala 2.5 (included with CDH 5.8), and the [Impala JDBC Driver v2.5.36](http://www.cloudera.com/downloads/connectors/impala/jdbc/2-5-36.html). 



#### 1. Preparing the Kudu quickstart VM

When running this project, you'll need an environment running Kafka, Kudu, Impala, and Spark. These instructions assume you're using the [Kudu Quickstart VM](https://kudu.apache.org/docs/quickstart.html), but can easily be adapted to run on a "real", fully-distributed cluster. Since the Kudu quickstart doesn't come with Kafka, we'll need to install Spark, Kafka, and Zookeeper. If you're running on a fully distributed Cloudera cluster, I'd recommend bypassing this step and going straight to Step 2, you can easily use Cloudera Manager to accomplish the same thing. 



Run the 'quickstart_setup' to install the necessary software on the quickstart VM: 

    

    git clone https://github.com/bosshart/kuduscreencast.git

    cd kuduscreencast/timeseries-with-kudu/

    chmod u+x quickstart_setup.sh

    sudo ./quickstart_setup.sh

    



#### 2. Install JDBC drivers and build the code. 



Run the following on the host where you want to build your application.



The webapp that feeds the FIX message visualization connects to Kudu through Impala, meaning you will need to download the Impala JDBC driver jars and make them available to maven as a dependency. For more information on how to build and run a Maven-based project to execute SQL queries on Impala using JDBC, see [here](https://github.com/onefoursix/Cloudera-Impala-JDBC-Example). 

 

The pom dependency file for our project includes references to the following jars that are not available in public repos. 



    (1)  ImpalaJDBC41.jar

    (2)  TCLIServiceClient.jar

    (3)  hive_metastore.jar

    (4)  hive_service.jar

    (5)  ql.jar

    

Download the appropriate JDBC connector jars from the [cloudera website](http://www.cloudera.com/downloads/connectors/impala/jdbc/2-5-36.html), scp it (if needed), and unzip the file. 



    scp impala_jdbc_2.5.36.2056.zip demo@quickstart:/tmp

    unzip impala_jdbc_2.5.36.2056.zip

    cd 2.5.36.1056\ GA/

    unzip Cloudera_ImpalaJDBC41_2.5.36.zip



Install the JDBC drivers in your local maven repo: 



    mvn install:install-file -Dfile=ImpalaJDBC41.jar -DgroupId=com.cloudera.impala.jdbc -DartifactId=ImpalaJDBC41 -Dversion=2.5.36 -Dpackaging=jar

    mvn install:install-file -Dfile=hive_service.jar -DgroupId=com.cloudera.impala.jdbc -DartifactId=hive_service -Dversion=2.5.36 -Dpackaging=jar

    mvn install:install-file -Dfile=hive_metastore.jar -DgroupId=com.cloudera.impala.jdbc -DartifactId=hive_metastore -Dversion=2.5.36 -Dpackaging=jar

    mvn install:install-file -Dfile=ql.jar -DgroupId=com.cloudera.impala.jdbc -DartifactId=ql -Dversion=2.5.36 -Dpackaging=jar

    mvn install:install-file -Dfile=TCLIServiceClient.jar -DgroupId=com.cloudera.impala.jdbc -DartifactId=TCLIServiceClient -Dversion=2.5.36 -Dpackaging=jar



Clone the repo (if you haven't already) and build the project. 



    git clone https://github.com/bosshart/kuduscreencast.git

    cd kuduscreencast/timeseries-with-kudu/

    mvn clean

    mvn package



Move the "sample app" to the quickstart VM or edge node as needed. 



    scp target/sample.app-1.0-SNAPSHOT.jar demo@quickstart.cloudera:/tmp/



#### 3. Create FIX Message Kafka Topic, Kudu, and Impala Table



The FIX message generator will publish order and execution messages to a Kafka topic. I named my kafka topic "fixdata" and configured it with one partition and no replicas so it could run on the quickstart VM.



    kafka-topics --zookeeper quickstart.cloudera:2181 --create --topic fixdata --partitions 1 --replication-factor 1

    

If you want, you can test Kafka. In separate ssh sessions, run both: 



    kafka-console-producer --broker-list quickstart:9092 --topic fixdata

    kafka-console-consumer --zookeeper quickstart:2181 --topic fixdata    



You should be able input test in the "producer" side and see it emitted in the consumer. 



Next, create the corresponding Kudu and Impala tables. Specify the kudu master (quickstart), name of table ("fixdata"), number of hash partitions for the stocksymbol (3), and number of range partitions on transaction time (also 3). Also, optionally, specify "quickstart" in order to only create a single replica. If you skip this argument, you'll end up with the default 3 tablet replicas. 

    

    java -cp sample.app-1.0-SNAPSHOT.jar com.kuduscreencast.timeseries.CreateFixTable quickstart fixdata 3 3 quickstart

    impala-shell

    [quickstart.cloudera:21000] > CREATE EXTERNAL TABLE `fixdata` (

                                  `transacttime` BIGINT,

                                  `stocksymbol` STRING,

                                  `clordid` STRING,

                                  `msgtype` STRING,

                                  `orderqty` INT,

                                  `leavesqty` INT,

                                  `cumqty` INT,

                                  `avgpx` DOUBLE,

                                  `lastupdated` BIGINT

                                  )

                                  TBLPROPERTIES(

                                    'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',

                                    'kudu.table_name' = 'fixdata',

                                    'kudu.master_addresses' = 'quickstart.cloudera:7051',

                                    'kudu.key_columns' = 'transacttime, stocksymbol, clordid'

                                  );

    



If you're using CDH 5.10 or later, you'll need different syntax when creating the table in Impala: 



    [quickstart.cloudera:21000] > CREATE EXTERNAL TABLE `fixdata` STORED AS KUDU

                                  TBLPROPERTIES(

                                    'kudu.table_name' = 'fixdata',

                                    'kudu.master_addresses' = 'quickstart.cloudera:7051');



#### 4. Run the Application



    

Run the Fix message generator



    spark-submit --master spark://:quickstart.cloudera  --class com.kuduscreencast.timeseries.FIXGenerator sample.app-1.0-SNAPSHOT.jar quickstart:9092 fixdata



Run the Spark Streaming application to populate Kudu: 

    

    spark-submit --master spark://quickstart:7077  --class com.kuduscreencast.timeseries.KuduFixDataStreamer sample.app-1.0-SNAPSHOT.jar quickstart:9092 fixdata quickstart fixdata quickstart



Finally, run the web application using jetty: 



    mvn jetty:run

    



If you go to \<web-app-hostname\>:8080, you should see a visualization showing the largest order within that ten second window for each stock symbol. The chart should automatically refresh it's data every 10 seconds.

Building a Near Real-Time Analytical Application with Kudu - Working Files/Working Files - README.txt

0 → 100755
+5 −0
Changes for Building a Near Real-Time Analytical Application with Kudu - Working Files/Working Files - README.txt: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line 
Due to ongoing efforts to ensure you have the best possible learning experience, we may update the working files for this course from time to time.



We recommend that you download the latest version of these files from:



http://www.infiniteskills.com/02469
 
 No newline at end of file

Building a Near Real-Time Analytical Application with Kudu - Working Files/dependency-reduced-pom.xml

0 → 100755
+105 −0
Changes for Building a Near Real-Time Analytical Application with Kudu - Working Files/dependency-reduced-pom.xml: 105 added lines, 0 removed lines.
Original line number Diff line number Diff line 
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.kuduscreencast</groupId>

  <artifactId>timeseries.app</artifactId>

  <version>1.0-SNAPSHOT</version>

  <build>

    <plugins>

      <plugin>

        <groupId>org.eclipse.jetty</groupId>

        <artifactId>jetty-maven-plugin</artifactId>

        <version>9.2.7.v20150116</version>

      </plugin>

      <plugin>

        <artifactId>maven-compiler-plugin</artifactId>

        <version>3.6.0</version>

        <configuration>

          <source>1.7</source>

          <target>1.7</target>

        </configuration>

      </plugin>

      <plugin>

        <groupId>net.alchim31.maven</groupId>

        <artifactId>scala-maven-plugin</artifactId>

        <version>3.2.0</version>

        <executions>

          <execution>

            <id>compile</id>

            <phase>compile</phase>

            <goals>

              <goal>compile</goal>

            </goals>

          </execution>

          <execution>

            <id>test-compile</id>

            <phase>test-compile</phase>

            <goals>

              <goal>testCompile</goal>

            </goals>

          </execution>

          <execution>

            <phase>process-resources</phase>

            <goals>

              <goal>compile</goal>

            </goals>

          </execution>

        </executions>

      </plugin>

      <plugin>

        <artifactId>maven-shade-plugin</artifactId>

        <version>1.7.1</version>

        <executions>

          <execution>

            <phase>package</phase>

            <goals>

              <goal>shade</goal>

            </goals>

          </execution>

        </executions>

        <configuration>

          <filters>

            <filter>

              <artifact>*:*</artifact>

              <excludes>

                <exclude>META-INF/*.SF</exclude>

                <exclude>META-INF/*.DSA</exclude>

                <exclude>META-INF/*.RSA</exclude>

              </excludes>

            </filter>

          </filters>

        </configuration>

      </plugin>

    </plugins>

  </build>

  <repositories>

    <repository>

      <id>scala-tools.org</id>

      <name>Scala-tools Maven2 Repository</name>

      <url>http://scala-tools.org/repo-releases</url>

    </repository>

    <repository>

      <snapshots>

        <enabled>false</enabled>

      </snapshots>

      <id>cdh.repo</id>

      <name>Cloudera Repositories</name>

      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>

    </repository>

  </repositories>

  <pluginRepositories>

    <pluginRepository>

      <id>scala-tools.org</id>

      <name>Scala-tools Maven2 Repository</name>

      <url>http://scala-tools.org/repo-releases</url>

    </pluginRepository>

  </pluginRepositories>

  <properties>

    <kudu.version>1.2.0</kudu.version>

    <impala.jdbc.version>2.5.36</impala.jdbc.version>

    <log4j.version>4.10</log4j.version>

    <spark.version>1.6.0-cdh5.10.0</spark.version>

    <kafka.version>0.9.0-kafka-2.0.1</kafka.version>

  </properties>

</project>

Building a Near Real-Time Analytical Application with Kudu - Working Files/pom.xml

0 → 100755
+194 −0
Changes for Building a Near Real-Time Analytical Application with Kudu - Working Files/pom.xml: 194 added lines, 0 removed lines.
Original line number Diff line number Diff line 
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>com.kuduscreencast</groupId>

    <artifactId>timeseries-with-kudu</artifactId>

    <version>1.0-SNAPSHOT</version>

    <properties>

        <kudu.version>1.2.0</kudu.version>

        <spark.version>1.6.0-cdh5.10.0</spark.version>

        <log4j.version>4.10</log4j.version>

        <kafka.version>0.9.0-kafka-2.0.1</kafka.version>

        <impala.jdbc.version>2.5.36</impala.jdbc.version>

    </properties>

    <pluginRepositories>

        <pluginRepository>

            <id>scala-tools.org</id>

            <name>Scala-tools Maven2 Repository</name>

            <url>http://scala-tools.org/repo-releases</url>

        </pluginRepository>

    </pluginRepositories>

    <build>

        <plugins>

            <plugin>

                <groupId>org.eclipse.jetty</groupId>

                <artifactId>jetty-maven-plugin</artifactId>

                <version>9.2.7.v20150116</version>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-compiler-plugin</artifactId>

                <version>3.6.0</version>

                <configuration>

                    <source>1.7</source>

                    <target>1.7</target>

                </configuration>

            </plugin>

            <plugin>

                <groupId>net.alchim31.maven</groupId>

                <artifactId>scala-maven-plugin</artifactId>

                <version>3.2.0</version>

                <executions>

                    <execution>

                        <id>compile</id>

                        <goals>

                            <goal>compile</goal>

                        </goals>

                        <phase>compile</phase>

                    </execution>

                    <execution>

                        <id>test-compile</id>

                        <goals>

                            <goal>testCompile</goal>

                        </goals>

                        <phase>test-compile</phase>

                    </execution>

                    <execution>

                        <phase>process-resources</phase>

                        <goals>

                            <goal>compile</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-shade-plugin</artifactId>

                <version>1.7.1</version>

                <executions>

                    <execution>

                        <phase>package</phase>

                        <goals>

                            <goal>shade</goal>

                        </goals>

                    </execution>

                </executions>

                <configuration>

                    <filters>

                        <filter>

                            <artifact>*:*</artifact>

                            <excludes>

                                <exclude>META-INF/*.SF</exclude>

                                <exclude>META-INF/*.DSA</exclude>

                                <exclude>META-INF/*.RSA</exclude>

                            </excludes>

                        </filter>

                    </filters>

                </configuration>

            </plugin>

        </plugins>

    </build>

    <dependencies>

        <dependency>

            <groupId>org.apache.kudu</groupId>

            <artifactId>kudu-client</artifactId>

            <version>${kudu.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.kudu</groupId>

            <artifactId>kudu-spark_2.10</artifactId>

            <version>${kudu.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_2.10</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-mllib_2.10</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming_2.10</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.kafka</groupId>

            <artifactId>kafka-clients</artifactId>

            <version>${kafka.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming-kafka_2.10</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_2.10</artifactId>

            <version>${spark.version}</version>

            <exclusions>

                <exclusion>

                    <groupId>org.xerial.snappy</groupId>

                    <artifactId>snappy-java</artifactId>

                </exclusion>

            </exclusions>

        </dependency>

        <!-- begin impala/UI dependencies -->

        <dependency>

            <groupId>com.cloudera.impala.jdbc</groupId>

            <artifactId>ImpalaJDBC41</artifactId>

            <version>${impala.jdbc.version}</version>

        </dependency>

        <dependency>

            <groupId>com.cloudera.impala.jdbc</groupId>

            <artifactId>hive_metastore</artifactId>

            <version>${impala.jdbc.version}</version>

        </dependency>

        <dependency>

            <groupId>com.cloudera.impala.jdbc</groupId>

            <artifactId>hive_service</artifactId>

            <version>${impala.jdbc.version}</version>

        </dependency>

        <dependency>

            <groupId>com.cloudera.impala.jdbc</groupId>

            <artifactId>ql</artifactId>

            <version>${impala.jdbc.version}</version>

        </dependency>

        <dependency>

            <groupId>com.cloudera.impala.jdbc</groupId>

            <artifactId>TCLIServiceClient</artifactId>

            <version>${impala.jdbc.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.thrift</groupId>

            <artifactId>libfb303</artifactId>

            <version>0.9.0</version>

        </dependency>

        <dependency>

            <groupId>org.apache.thrift</groupId>

            <artifactId>libthrift</artifactId>

            <version>0.9.0</version>

        </dependency>

    </dependencies>

    <repositories>

        <repository>

            <id>scala-tools.org</id>

            <name>Scala-tools Maven2 Repository</name>

            <url>http://scala-tools.org/repo-releases</url>

        </repository>

        <repository>

            <id>cdh.repo</id>

            <name>Cloudera Repositories</name>

            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>

            <snapshots>

                <enabled>false</enabled>

            </snapshots>

        </repository>

    </repositories>



</project>
 
 No newline at end of file

Building a Near Real-Time Analytical Application with Kudu - Working Files/quickstart_setup.sh

0 → 100755
+44 −0
Changes for Building a Near Real-Time Analytical Application with Kudu - Working Files/quickstart_setup.sh: 44 added lines, 0 removed lines.
Original line number Diff line number Diff line 
#!/usr/bin/env bash



set -x

set -e



sudo echo "[cloudera-kafka]

# Packages for Cloudera's Distribution for kafka, Version 2, on RedHat  or CentOS 6 x86_64

name=Cloudera's Distribution for kafka, Version 2

baseurl=http://archive.cloudera.com/kafka/redhat/6/x86_64/kafka/2.0.1/

gpgkey = http://archive.cloudera.com/kafka/redhat/6/x86_64/kafka/RPM-GPG-KEY-cloudera

gpgcheck = 1" > /etc/yum.repos.d/cloudera-kafka.repo



sudo echo "[cloudera-cdh5]

# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64

name=Cloudera's Distribution for Hadoop, Version 5

baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.10/

gpgkey = http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

gpgcheck = 1" > /etc/yum.repos.d/cloudera-cdh5.repo



sudo yum -y install git

sudo yum -y install zookeeper-server

mkdir -p /var/lib/zookeeper

sudo chown -R zookeeper /var/lib/zookeeper/

sudo service zookeeper-server init

sudo service zookeeper-server start



sudo yum -y install kafka kafka-server

sudo yum -y install spark-core spark-master spark-worker



#Replace the relevant portion of /etc/spark/conf/spark-env.sh to point to the host where the spark-master runs

sed -i "s/export STANDALONE_SPARK_MASTER_HOST=\`hostname\`/export STANDALONE_SPARK_MASTER_HOST=quickstart.cloudera/" /etc/spark/conf/spark-env.sh



sudo service kafka-server start

sudo service spark-master start

sudo service spark-worker start



wget -P /tmp/ http://apache.cs.utah.edu/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

sudo tar xzf /tmp/apache-maven-3.3.9-bin.tar.gz -C /usr/local

cat <<'EOF'>> maven.sh

export M2_HOME=/usr/local/apache-maven-3.3.9

export PATH=${M2_HOME}/bin:${PATH}

EOF

sudo mv maven.sh /etc/profile.d/maven.sh

source /etc/profile

© 2026 O’Reilly Media, Inc. All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. Terms of Service • Privacy Policy • Editorial Independence