Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4746

Ensure spark can be run as PIG action in Oozie

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      I was able get PIG on SPARK going with Oozie. But, only in "local" mode. Here is what I did:
      1. Used workflow schema version uri:oozie:workflow:0.2 and passed exectype as an argument.
      2. Copied Spark jars + kyro jar into workflow app lib.

      To get spork going in yarn-client mode, couple of enhancements will need to be made:
      1. Right now, spark launcher reads SPARK_MASTER as env. variable. Need to make this a PIG property.
      2. The spark libraries need to be in classpath of the driver in case of yarn-clientmode. This will need a fix similar to PIG-4667

        Attachments

          Activity

            People

            • Assignee:
              Pratyy Prateek Vaishnav
              Reporter:
              pallavi.rao Pallavi Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: