Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3259

ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: mrv2, nodemanager
    • Labels:
      None

      Description

      As seen in MAPREDUCE-2915, java.library.path is not being passed when the LCE spawns a JVM for ContainerLocalizer.

      However, unlike branch-0.20-security, the task runtime in 0.23 is unaffected by this. This is because tasks' run-time environment is specified in the launch script by client. Setting LD_LIBRARY_PATH is the primary way of specifying the locations of required native library in this case. The config property, mapreduce.admin.user.env is always set in the job environment and the default value is to add the path to the hadoop native library to LD_LABRARY_PATH.

      For JVM's being launched by the hadoop system scripts, java.library.path is set.

      1. MAPREDUCE-3259.patch.txt
        1 kB
        Kihwal Lee
      2. mapreduce-3259.patch.txt
        2 kB
        Kihwal Lee
      3. mapreduce-3259.patch
        2 kB
        Kihwal Lee

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #845 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/845/)
        MAPREDUCE-3259. Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #845 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/845/ ) MAPREDUCE-3259 . Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #64 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/)
        Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #64 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/ ) Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #873 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/873/)
        MAPREDUCE-3259. Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #873 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/873/ ) MAPREDUCE-3259 . Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #52 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/)
        Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #52 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/ ) Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #68 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/68/)
        Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #68 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/68/ ) Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1182/)
        MAPREDUCE-3259. Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1182/ ) MAPREDUCE-3259 . Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1243/)
        MAPREDUCE-3259. Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1243/ ) MAPREDUCE-3259 . Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1166 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1166/)
        MAPREDUCE-3259. Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1166 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1166/ ) MAPREDUCE-3259 . Added java.library.path of NodeManager to ContainerLocalizer in LinuxContainerExecutor. Contributed by Kihwal Lee. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189667 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #73 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/73/)
        Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #73 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/73/ ) Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Kihwal Lee added a comment -
        • The test failure is not related to the patch. It fails because the test hard codes the user name.
          junit.framework.ComparisonFailure: Incorrect username expected:<[mapred]> but was:<[jenkins]>
        • None of the findbugs warnings are related to this patch or new.
        • Testing require deployment with a set of specific configurations. As reported above, it was done manually.
        Show
        Kihwal Lee added a comment - The test failure is not related to the patch. It fails because the test hard codes the user name. junit.framework.ComparisonFailure: Incorrect username expected:< [mapred] > but was:< [jenkins] > None of the findbugs warnings are related to this patch or new. Testing require deployment with a set of specific configurations. As reported above, it was done manually.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #72 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/72/)
        Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #72 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/72/ ) Merge -c 1189667 from trunk to branch-0.23 to fix MAPREDUCE-3259 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189668 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Kihwal!

        Unit test failure was unrelated and fixed by MAPREDUCE-3279.

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Kihwal! Unit test failure was unrelated and fixed by MAPREDUCE-3279 .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12501054/MAPREDUCE-3259.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501054/MAPREDUCE-3259.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1168//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        The new patch contains only the necessary change pertaining to LinuxContainerExecutor#startLocalizer.

        Show
        Kihwal Lee added a comment - The new patch contains only the necessary change pertaining to LinuxContainerExecutor#startLocalizer.
        Hide
        Arun C Murthy added a comment -

        Ok, the talk about apps confused me. Now I see the real issue: ContainerLocalizer needs this set for UGI etc.

        We shouldn't change MapReduceChildJVM, the changes to LCE.startLocalizer seem fine.

        Show
        Arun C Murthy added a comment - Ok, the talk about apps confused me. Now I see the real issue: ContainerLocalizer needs this set for UGI etc. We shouldn't change MapReduceChildJVM, the changes to LCE.startLocalizer seem fine.
        Hide
        Kihwal Lee added a comment -

        Sorry, my explanation is rather wrong and confusing. I shouldn't have said app-level. That's totally wrong.

        Apps are fine as you said. It's about yarn system level env. When a JVM is directly launched by the LTC, neither of LD_LIBRARY_PATH or java.library.path is set. This is the case for ContainerLocalizer. It will fail to load libhadoop.so if it needs it. If the library is installed in one of the directories ld looks at, it will work without the setting. Or the hadoop lib path may be added to /etc/ld.so.conf or /etc/ld.so.conf.d. Is it still confusing? Or am I totally wrong about this?

        I can leave out the second part in the patch for the client since a proper env can be placed in the container script using the facility you mentioned. Only thing I don't like is the default, which assumes the old dir layout. I could try changing the default so that it has a better chance of working right out of the box, but that's not critical.

        Show
        Kihwal Lee added a comment - Sorry, my explanation is rather wrong and confusing. I shouldn't have said app-level. That's totally wrong. Apps are fine as you said. It's about yarn system level env. When a JVM is directly launched by the LTC, neither of LD_LIBRARY_PATH or java.library.path is set. This is the case for ContainerLocalizer. It will fail to load libhadoop.so if it needs it. If the library is installed in one of the directories ld looks at, it will work without the setting. Or the hadoop lib path may be added to /etc/ld.so.conf or /etc/ld.so.conf.d. Is it still confusing? Or am I totally wrong about this? I can leave out the second part in the patch for the client since a proper env can be placed in the container script using the facility you mentioned. Only thing I don't like is the default, which assumes the old dir layout. I could try changing the default so that it has a better chance of working right out of the box, but that's not critical.
        Hide
        Arun C Murthy added a comment -

        Kihwal, sorry if I wasn't clear.

        If the admins set mapreduce.admin.user.env such that LD_LIBRARY_PATH includes paths for libhadoop.so and JNI-based user-groups mapping, what goes wrong?

        mapreduce.admin.user.env is for cluster-level configs (libhadoop, user-groups mapping etc.) and app-specific deps can go in mapreduce.(map,reduce).env. What am I missing here?

        Show
        Arun C Murthy added a comment - Kihwal, sorry if I wasn't clear. If the admins set mapreduce.admin.user.env such that LD_LIBRARY_PATH includes paths for libhadoop.so and JNI-based user-groups mapping, what goes wrong? mapreduce.admin.user.env is for cluster-level configs (libhadoop, user-groups mapping etc.) and app-specific deps can go in mapreduce.(map,reduce).env. What am I missing here?
        Hide
        Kihwal Lee added a comment -

        Kihwal, won't it suffice to have the right configs (and defaults) for admin.env?

        For executing the task script, that's true. For LTC directly launching JVM to run localizer, either LD_LIBRARY_PATH or -Djava.library.path is need to be set properly. It will appear to work until something requiring the native lib is enabled. When I enabled the JNI-based user groups mapping feature, the localizer wouldn't run.

        Back to running task script. The task jvm may have to load libhadoop.so because of a certain cluster-level config. I think we should treat it as an app-level dependency, rather than a run-time dependency of specific client code. Sometimes even admins won't know whether the library will be required by enabling a feature. I thought it will be simpler to add java.library.path to take care of the dependency at the app-level.

        Show
        Kihwal Lee added a comment - Kihwal, won't it suffice to have the right configs (and defaults) for admin.env? For executing the task script, that's true. For LTC directly launching JVM to run localizer, either LD_LIBRARY_PATH or -Djava.library.path is need to be set properly. It will appear to work until something requiring the native lib is enabled. When I enabled the JNI-based user groups mapping feature, the localizer wouldn't run. Back to running task script. The task jvm may have to load libhadoop.so because of a certain cluster-level config. I think we should treat it as an app-level dependency, rather than a run-time dependency of specific client code. Sometimes even admins won't know whether the library will be required by enabling a feature. I thought it will be simpler to add java.library.path to take care of the dependency at the app-level.
        Hide
        Arun C Murthy added a comment -

        Kihwal, won't it suffice to have the right configs (and defaults) for admin.env?

        Show
        Arun C Murthy added a comment - Kihwal, won't it suffice to have the right configs (and defaults) for admin.env?
        Hide
        Kihwal Lee added a comment -

        Reattaching the same patch with license granted, with the name change for the friendlier mine-type.

        Show
        Kihwal Lee added a comment - Reattaching the same patch with license granted, with the name change for the friendlier mine-type.
        Hide
        Kihwal Lee added a comment -

        The javadoc warning is not caused by this patch.

         [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java:112: warning - @param argument "config" is not a parameter name.
        

        The findbugs warnings don't seem to be new. Even the ones in the NM and the client app have nothing to do with this patch.

        Show
        Kihwal Lee added a comment - The javadoc warning is not caused by this patch. [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java:112: warning - @param argument "config" is not a parameter name. The findbugs warnings don't seem to be new. Even the ones in the NM and the client app have nothing to do with this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12500662/mapreduce-3259.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 165 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500662/mapreduce-3259.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 165 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1134//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        The attached patch is for trunk, but should also apply to branch-0.23.

        Although the client-side environment may be set by users or by the admin, the framework including what may be needed by the hadoop code base in java.library.path seems more reasonable. After all libhadoop.so may be required to run things even if users don't explicitly use any features that directly depend on it.

        Since meaningful tests need specific environment setup and the optional native build, no automated test was added. The risk of this change should be minimal.

        Manual testing was carried out as follows.

        • Enable JniBasedUnixUserGroupsMapping.
        • Build without patch.
        • Test failures observed in node manager and job client.
        • Apply patch and rebuild.
        • Without the library, many tests in node manager and job client failed.
        • Installed the library.
        • All tests pass.
        Show
        Kihwal Lee added a comment - The attached patch is for trunk, but should also apply to branch-0.23. Although the client-side environment may be set by users or by the admin, the framework including what may be needed by the hadoop code base in java.library.path seems more reasonable. After all libhadoop.so may be required to run things even if users don't explicitly use any features that directly depend on it. Since meaningful tests need specific environment setup and the optional native build, no automated test was added. The risk of this change should be minimal. Manual testing was carried out as follows. Enable JniBasedUnixUserGroupsMapping. Build without patch. Test failures observed in node manager and job client. Apply patch and rebuild. Without the library, many tests in node manager and job client failed. Installed the library. All tests pass.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development