[PIG-5246] Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 31/May/17 08:25
• Updated: 25/Jul/17 01:55
• Resolved: 25/Jul/17 01:55

Description

in bin/pig.
we copy assembly jar to pig's classpath in spark1.6.

# For spark mode:
# Please specify SPARK_HOME first so that we can locate $SPARK_HOME/lib/spark-assembly*.jar,
# we will add spark-assembly*.jar to the classpath.
if [ "$isSparkMode"  == "true" ]; then
    if [ -z "$SPARK_HOME" ]; then
       echo "Error: SPARK_HOME is not set!"
       exit 1
    fi

    # Please specify SPARK_JAR which is the hdfs path of spark-assembly*.jar to allow YARN to cache spark-assembly*.jar on nodes so that it doesn't need to be distributed each time an application runs.
    if [ -z "$SPARK_JAR" ]; then
       echo "Error: SPARK_JAR is not set, SPARK_JAR stands for the hdfs location of spark-assembly*.jar. This allows YARN to cache spark-assembly*.jar on nodes so that it doesn't need to be distributed each time an application runs."
       exit 1
    fi

    if [ -n "$SPARK_HOME" ]; then
        echo "Using Spark Home: " ${SPARK_HOME}
        SPARK_ASSEMBLY_JAR=`ls ${SPARK_HOME}/lib/spark-assembly*`
        CLASSPATH=${CLASSPATH}:$SPARK_ASSEMBLY_JAR
    fi
fi

after upgrade to spark2.0, we may modify it