Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6469

Improving documentation on YARN local directories usage

    Details

    • Type: Documentation
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.3.1, 1.4.0
    • Component/s: Documentation, YARN
    • Labels:
      None

      Description

      According to the Spark YARN doc page, Spark executors will use the local directories configured for YARN, not spark.local.dir which should be ignored.

      However it should be noted that in yarn-client mode, though the executors will indeed use the local directories configured for YARN, the driver will not, because it is not running on the YARN cluster; the driver in yarn-client will use the local directories defined in spark.local.dir

      Can this please be clarified in the Spark YARN documentation above?

        Attachments

        1. TestYarnVars.scala
          0.3 kB
          Christophe Préaud

          Activity

            People

            • Assignee:
              preaudc Christophe Préaud
              Reporter:
              preaudc Christophe Préaud
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: