Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3332

[examples] Spark examples should feature yarn client and cluster modes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Trivial
    • Resolution: Fixed
    • None
    • 5.1.0
    • examples
    • None

    Description

      The spark and pyspark examples do not have a <mode> element in their workflow.xml files, probably because of the old way the spark CLI worked, in which to run spark on yarn in client mode, you would write

      --master yarn-client
      

      on the command line (https://spark.apache.org/docs/1.5.0/running-on-yarn.html). This still works, but from spark 1.6, the spark documentation (https://spark.apache.org/docs/1.6.0/running-on-yarn.html) says the way to do it is

      --master yarn --mode client
      

      .
      We should update the two examples to have <mode> elements in workflow.xml and corresponding values in job.properties.

      Attachments

        1. OOZIE-3332.patch
          2 kB
          Daniel Becker
        2. OOZIE-3332.2.patch
          3 kB
          Daniel Becker

        Activity

          People

            daniel.becker Daniel Becker
            daniel.becker Daniel Becker
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: