Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20995

'Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster.
      These configs are used to write to HDFS and connect to the YARN ResourceManager. The
      configuration contained in this directory will be distributed to the YARN cluster so that all
      containers used by the application use the same configuration.

      Sometimes, HADOOP_CONF_DIR is set to the hdfs configuration file path. So, YARN_CONF_DIR should be set to the yarn configuration file path.

        Attachments

          Activity

            People

            • Assignee:
              guoxiaolongzte guoxiaolong
              Reporter:
              guoxiaolongzte guoxiaolong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: