Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4233

Package pig along with dependencies into a fat jar while job submission to Spark cluster

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Currently we have a fat jar created in legacy directory which contains pig along with dependencies.
      Would need to modify build.xml to add spark dependency jars to include in legacy fat jar.

      Running job on Spark cluster:

      1. export SPARK_HOME=/path/to/spark
      2. export SPARK_PIG_JAR=$PIG_HOME/legacy/pig-0.14.0-SNAPSHOT-withouthadoop-h1.jar
      3. export SPARK_MASTER=spark://localhost:7077
      4 export HADOOP_HOME=/path/to/hadoop
      5. Launch the job using ./bin/pig -x spark

        Attachments

        1. PIG-4233.patch
          1 kB
          Praveen Rachabattuni

          Activity

            People

            • Assignee:
              praveenr019 Praveen Rachabattuni
              Reporter:
              praveenr019 Praveen Rachabattuni
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: