[PIG-4174] e2e tests for Spark

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Praveen Rachabattuni
• Reporter: Praveen Rachabattuni
• Created: 16/Sep/14 13:45
• Updated: 21/Jun/17 09:18
• Resolved: 10/Oct/14 10:43

Description

Setup e2e tests for pig on spark like that pig on map-reduce and pig on tez.

Steps to setup e2e tests:

1. Initialize Variables
export HADOOP_HOME=/path/to/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/conf
export HADOOP_BIN=$HADOOP_HOME/bin/hadoop

2. Generate Data
ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dharness.hadoop.home=$HADOOP_HOME test-e2e-deploy-local
(You might want to install necessary cpan modules incase of any dependency errors https://cwiki.apache.org/confluence/display/PIG/HowToTest#HowToTest-End-to-endTesting)

3. Copy data to hdfs to use with Spark
hadoop fs -put test/e2e/pig/testdist/data ./

4. Run particular test
ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dharness.hadoop.home=$HADOOP_HOME -Dtests.to.run="-t Checkin_1" test-e2e-spark

5. Run all tests
ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dharness.hadoop.home=$HADOOP_HOME test-e2e-spark

Issue Links

• is related to: PIG-4206 e2e tests for Spark can not work in hadoop env
• relates to: PIG-4206 e2e tests for Spark can not work in hadoop env