Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4253

Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) which are meant to wrap invocations to the container-executor binary. However, not all invocations of container-executor in LinuxContainerExecutor use the PrivilegedOperationExecutor. We should change all such invocations to use PrivilegedOperationExecutor.

      1. YARN-4253.001.patch
        20 kB
        Sidharta Seethana
      2. YARN-4253-branch-2.001.patch
        20 kB
        Sidharta Seethana

        Activity

        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploaded a patch

        Show
        sidharta-s Sidharta Seethana added a comment - Uploaded a patch
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 16s There were no new javac warning messages.
        +1 javadoc 10m 41s There were no new javadoc warning messages.
        -1 release audit 0m 22s The applied patch generated 1 release audit warnings.
        -1 checkstyle 0m 39s The applied patch generated 1 new checkstyle issues (total was 41, now 2).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 44s mvn install still works.
        +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
        +1 findbugs 1m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 9m 4s Tests passed in hadoop-yarn-server-nodemanager.
            50m 4s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766085/YARN-4253.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e617cf6
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9406/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9406/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 16s There were no new javac warning messages. +1 javadoc 10m 41s There were no new javadoc warning messages. -1 release audit 0m 22s The applied patch generated 1 release audit warnings. -1 checkstyle 0m 39s The applied patch generated 1 new checkstyle issues (total was 41, now 2). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 44s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 1m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 9m 4s Tests passed in hadoop-yarn-server-nodemanager.     50m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766085/YARN-4253.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e617cf6 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9406/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9406/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9406/console This message was automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Pre-Patch failure : unrelated to this JIRA
        Tests : There are existing tests in TestLinuxContainerExecutorWithMocks that test the part of code that was modified in this patch. There is no change in behavior so no additional tests are required.
        Release-Audit failure : Not related to this JIRA
        Checkstyle failure : the failure is due to formatting changes to a function that was not otherwise modified. The length of the function was not changed in this patch.

        Varun Vasudev , could you please review this patch?

        Show
        sidharta-s Sidharta Seethana added a comment - Pre-Patch failure : unrelated to this JIRA Tests : There are existing tests in TestLinuxContainerExecutorWithMocks that test the part of code that was modified in this patch. There is no change in behavior so no additional tests are required. Release-Audit failure : Not related to this JIRA Checkstyle failure : the failure is due to formatting changes to a function that was not otherwise modified. The length of the function was not changed in this patch. Varun Vasudev , could you please review this patch?
        Hide
        sidharta-s Sidharta Seethana added a comment -

        It looks like the patch doesn't apply cleanly on branch-2. Uploaded a new patch for branch-2 (changes are trivial).

        Show
        sidharta-s Sidharta Seethana added a comment - It looks like the patch doesn't apply cleanly on branch-2. Uploaded a new patch for branch-2 (changes are trivial).
        Hide
        vvasudev Varun Vasudev added a comment -

        +1. Tested this out on my local machine and there was no change. I'll commit this tomorrow if no one objects.

        Show
        vvasudev Varun Vasudev added a comment - +1. Tested this out on my local machine and there was no change. I'll commit this tomorrow if no one objects.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 32s Pre-patch branch-2 has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 6m 2s There were no new javac warning messages.
        +1 javadoc 10m 10s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 37s The applied patch generated 1 new checkstyle issues (total was 41, now 2).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 15s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 8m 39s Tests passed in hadoop-yarn-server-nodemanager.
            45m 31s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766297/YARN-4253-branch-2.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision branch-2 / 34b8142
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/branch-2FindbugsWarningshadoop-yarn-server-nodemanager.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9424/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9424/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 32s Pre-patch branch-2 has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 6m 2s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 37s The applied patch generated 1 new checkstyle issues (total was 41, now 2). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 15s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 8m 39s Tests passed in hadoop-yarn-server-nodemanager.     45m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766297/YARN-4253-branch-2.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2 / 34b8142 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/branch-2FindbugsWarningshadoop-yarn-server-nodemanager.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9424/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9424/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9424/console This message was automatically generated.
        Hide
        vvasudev Varun Vasudev added a comment -

        Committed to trunk and branch-2. Thanks Sid!

        Show
        vvasudev Varun Vasudev added a comment - Committed to trunk and branch-2. Thanks Sid!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8629 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8629/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8629 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8629/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/CHANGES.txt
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks!

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/)
        YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/ ) YARN-4253 . Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java

          People

          • Assignee:
            sidharta-s Sidharta Seethana
            Reporter:
            sidharta-s Sidharta Seethana
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development