Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059

Pig on Spark

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch, 0.17.0
    • Component/s: spark
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Setting up your development environment:
      0. download spark release package(currently pig on spark only support spark 1.6).
      1. Check out Pig Spark branch.

      2. Build Pig by running "ant jar" and "ant -Dhadoopversion=23 jar" for hadoop-2.x versions

      3. Configure these environmental variables:
      export HADOOP_USER_CLASSPATH_FIRST="true"
      Now we support “local” and "yarn-client" mode, you can export system variable “SPARK_MASTER” like:
      export SPARK_MASTER=local or export SPARK_MASTER="yarn-client"

      4. In local mode: ./pig -x spark_local xxx.pig
      In yarn-client mode:
      export SPARK_HOME=xx;
      export SPARK_JAR=hdfs://example.com:8020/xxxx (the hdfs location where you upload the spark-assembly*.jar)
      ./pig -x spark xxx.pig

        Attachments

        1. Pig-on-Spark-Design-Doc.pdf
          82 kB
          Praveen Rachabattuni
        2. Pig-on-Spark-Scope.pdf
          549 kB
          Mohit Sabharwal

          Issue Links

            Activity

              People

              • Assignee:
                praveenr019 Praveen Rachabattuni
                Reporter:
                rohini Rohini Palaniswamy
              • Votes:
                22 Vote for this issue
                Watchers:
                73 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: