Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4219

make default container-executor.conf.dir be a path relative to the container-executor binary

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 2.0.0-alpha
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, container-executor binary has an absolute pathname of its configuration file baked in. This prevents an easy relocation of the configuration files when dealing with multiple Hadoop installs on the same node. It would be nice to at least allow for a relative path resolution starting from the location of the container-executor binary itself. Something like

      ../etc/hadoop/
      

      Thoughts?

      1. MAPREDUCE-4219.patch.txt
        7 kB
        Roman Shaposhnik
      2. HADOOP-8332.patch.txt
        6 kB
        Roman Shaposhnik

        Activity

        Hide
        Robert Joseph Evans added a comment -

        I would be a bit nervous about this from a security perspective. It should not be that big of a deal because the container executor validates that the conf file is owned by the correct user, but I have not looked at it that exhaustively.

        Show
        Robert Joseph Evans added a comment - I would be a bit nervous about this from a security perspective. It should not be that big of a deal because the container executor validates that the conf file is owned by the correct user, but I have not looked at it that exhaustively.
        Hide
        Roman Shaposhnik added a comment -

        This patch adds an option of specifying container-executor.conf.dir as a path relative to the location of the container-executor executable itself.

        Show
        Roman Shaposhnik added a comment - This patch adds an option of specifying container-executor.conf.dir as a path relative to the location of the container-executor executable itself.
        Hide
        Roman Shaposhnik added a comment -

        @Robert, I understand your concerns and that's why:

        1. I would like this change to be vetted by as large of a community as possible
        2. my current patch leaves an option of sticking with the absolute path for those who really need that
        Show
        Roman Shaposhnik added a comment - @Robert, I understand your concerns and that's why: I would like this change to be vetted by as large of a community as possible my current patch leaves an option of sticking with the absolute path for those who really need that
        Hide
        Alejandro Abdelnur added a comment -

        +1, patched, built & tested native stuff.

        Also, it would be great if you test it a deployed cluster before we commit it?

        2 NITs:

        • function name, IMO, resolve_config_path would be a more appropriate name.
        • add a comment where HADOOP_CONF is used in main.c that the value comes from the POM property container-executor.conf.dir
        Show
        Alejandro Abdelnur added a comment - +1, patched, built & tested native stuff. Also, it would be great if you test it a deployed cluster before we commit it? 2 NITs: function name, IMO, resolve_config_path would be a more appropriate name. add a comment where HADOOP_CONF is used in main.c that the value comes from the POM property container-executor.conf.dir
        Hide
        Roman Shaposhnik added a comment -

        Please take a look at the new patch.

        Show
        Roman Shaposhnik added a comment - Please take a look at the new patch.
        Hide
        Alejandro Abdelnur added a comment -

        +1, pending feedback on test in a real cluster.

        Show
        Alejandro Abdelnur added a comment - +1, pending feedback on test in a real cluster.
        Hide
        Roman Shaposhnik added a comment -

        I tested Hadoop examples on the fully distributed secure cluster with ../conf baked in as a HADOOP_CONF

        Show
        Roman Shaposhnik added a comment - I tested Hadoop examples on the fully distributed secure cluster with ../conf baked in as a HADOOP_CONF
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525340/MAPREDUCE-4219.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2350//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2350//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525340/MAPREDUCE-4219.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2350//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2350//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Roman. Committed to trunk and branch-2.

        Show
        Alejandro Abdelnur added a comment - Thanks Roman. Committed to trunk and branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2249 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2249/)
        MAPREDUCE-4219. make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2249 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2249/ ) MAPREDUCE-4219 . make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2175 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2175/)
        MAPREDUCE-4219. make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2175 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2175/ ) MAPREDUCE-4219 . make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2193/)
        MAPREDUCE-4219. make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241)

        Result = ABORTED
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2193/ ) MAPREDUCE-4219 . make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1033 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1033/)
        MAPREDUCE-4219. make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1033 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1033/ ) MAPREDUCE-4219 . make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1068 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1068/)
        MAPREDUCE-4219. make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1068 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1068/ ) MAPREDUCE-4219 . make default container-executor.conf.dir be a path relative to the container-executor binary. (rvs via tucu) (Revision 1333241) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333241 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development