[PIG-4233] Package pig along with dependencies into a fat jar while job submission to Spark cluster

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Praveen Rachabattuni
• Reporter: Praveen Rachabattuni
• Created: 14/Oct/14 12:37
• Updated: 21/Jun/17 09:18
• Resolved: 14/Oct/14 13:01

Description

Currently we have a fat jar created in legacy directory which contains pig along with dependencies.
Would need to modify build.xml to add spark dependency jars to include in legacy fat jar.

Running job on Spark cluster:

1. export SPARK_HOME=/path/to/spark
2. export SPARK_PIG_JAR=$PIG_HOME/legacy/pig-0.14.0-SNAPSHOT-withouthadoop-h1.jar
3. export SPARK_MASTER=spark://localhost:7077
4 export HADOOP_HOME=/path/to/hadoop
5. Launch the job using ./bin/pig -x spark