Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3505

yarn APPLICATION_CLASSPATH needs to be overridable

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
      If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time

      The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
      For convenience, here is its current content:

        /**
         * Classpath for typical applications.
         */
        public static final String[] APPLICATION_CLASSPATH =
            new String[] {
              "$HADOOP_CONF_DIR",
              "$HADOOP_COMMON_HOME/share/hadoop/common/*",
              "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
              "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
              "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
              "$YARN_HOME/modules/*",
              "$YARN_HOME/lib/*"
            };
      

      Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.

      1. MAPREDUCE-3505_rev2.patch
        11 kB
        Ahmed Radwan
      2. MAPREDUCE-3505.patch
        7 kB
        Ahmed Radwan

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Eugene Koontz made changes -
          Link This issue relates to MAPREDUCE-3389 [ MAPREDUCE-3389 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Ahmed Radwan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ahmed Radwan made changes -
          Attachment MAPREDUCE-3505_rev2.patch [ 12511621 ]
          Ahmed Radwan made changes -
          Fix Version/s 0.23.1 [ 12318883 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 0.23.1 [ 12318883 ]
          Ahmed Radwan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ahmed Radwan made changes -
          Assignee Ahmed Radwan [ ahmed.radwan ]
          Ahmed Radwan made changes -
          Attachment MAPREDUCE-3505.patch [ 12506990 ]
          Bruno Mahé made changes -
          Field Original Value New Value
          Description Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
          If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time

          The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
          For convenience, here is its current content:
          {quote}
            /**
             * Classpath for typical applications.
             */
            public static final String[] APPLICATION_CLASSPATH =
                new String[] {
                  "$HADOOP_CONF_DIR",
                  "$HADOOP_COMMON_HOME/share/hadoop/common/*",
                  "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
                  "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
                  "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
                  "$YARN_HOME/modules/*",
                  "$YARN_HOME/lib/*"
                };
          {quote}

          Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
          Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
          If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time

          The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
          For convenience, here is its current content:
          {noformat}
            /**
             * Classpath for typical applications.
             */
            public static final String[] APPLICATION_CLASSPATH =
                new String[] {
                  "$HADOOP_CONF_DIR",
                  "$HADOOP_COMMON_HOME/share/hadoop/common/*",
                  "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
                  "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
                  "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
                  "$YARN_HOME/modules/*",
                  "$YARN_HOME/lib/*"
                };
          {noformat}

          Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
          Bruno Mahé created issue -

            People

            • Assignee:
              Ahmed Radwan
              Reporter:
              Bruno Mahé
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development