[KYLIN-2421] Add spark engine to Integration Test

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.0.0
• Fix Version/s: v2.0.0
• Component/s: Job Engine
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 04/Feb/17 05:58
• Updated: 30/Apr/17 12:21
• Resolved: 05/Feb/17 12:54

Description

Let Jenkins to test Spark cubing daily. I have changed the engine type for "ci_inner_join_cube" from "2" (MRV2) to "4" (spark).

After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test:

1. manually install spark-1.6.3-bin-hadoop2.6 in a local folder like /usr/local/spark
cd /usr/local
tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz
ln -s spark-1.6.3-bin-hadoop2.6 spark

2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS):
hadoop fs -mkdir /kylin/spark/
hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/

3. manually create local temp folder for hbase client (if it doesn't exist):
mkdir -p /hadoop/hbase/local/jars
chmod 777 /hadoop/hbase/local/jars

4. Specify SPARK_HOME property when run mvn:

mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox