Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3332

[examples] Spark examples should feature yarn client and cluster modes

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1.0
    • Component/s: examples
    • Labels:
      None

      Description

      The spark and pyspark examples do not have a <mode> element in their workflow.xml files, probably because of the old way the spark CLI worked, in which to run spark on yarn in client mode, you would write

      --master yarn-client
      

      on the command line (https://spark.apache.org/docs/1.5.0/running-on-yarn.html). This still works, but from spark 1.6, the spark documentation (https://spark.apache.org/docs/1.6.0/running-on-yarn.html) says the way to do it is

      --master yarn --mode client
      

      .
      We should update the two examples to have <mode> elements in workflow.xml and corresponding values in job.properties.

        Attachments

        1. OOZIE-3332.patch
          2 kB
          Daniel Becker
        2. OOZIE-3332.2.patch
          3 kB
          Daniel Becker

          Activity

            People

            • Assignee:
              daniel.becker Daniel Becker
              Reporter:
              daniel.becker Daniel Becker

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment