Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6756

ContainerRequest#executionTypeRequest causes NPE

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ContainerRequest#executionTypeRequest is initialized as null, which could cause below " execTypeReq.getExecutionType" unconditionally throw NPE.

        ResourceRequestInfo addResourceRequest(Long allocationRequestId,
            Priority priority, String resourceName, ExecutionTypeRequest execTypeReq,
            Resource capability, T req, boolean relaxLocality,
            String labelExpression) {
          ResourceRequestInfo resourceRequestInfo = get(priority, resourceName,
              execTypeReq.getExecutionType(), capability);
      
      1. YARN-6756.01.patch
        0.9 kB
        Jian He
      2. YARN-6756.02.patch
        2 kB
        Jian He

        Activity

        Hide
        jianhe Jian He added a comment -

        Thanks Arun for the review !
        Committed to trunk and branch-2

        Show
        jianhe Jian He added a comment - Thanks Arun for the review ! Committed to trunk and branch-2
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12268 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12268/)
        YARN-6756. ContainerRequest#executionTypeRequest causes NPE. Contributed (jianhe: rev 8201ed8009e5f04c49568a8133635d47fcde3989)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientContainerRequest.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12268 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12268/ ) YARN-6756 . ContainerRequest#executionTypeRequest causes NPE. Contributed (jianhe: rev 8201ed8009e5f04c49568a8133635d47fcde3989) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientContainerRequest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 6s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 findbugs 0m 31s trunk passed
        +1 javadoc 0m 15s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        -0 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
        +1 mvnsite 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 39s the patch passed
        +1 javadoc 0m 12s the patch passed
              Other Tests
        +1 unit 20m 19s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        41m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884136/YARN-6756.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 60e19670e49e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51881a8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17166/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17166/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17166/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 6s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 24s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 15s trunk passed       Patch Compile Tests +1 mvninstall 0m 21s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -0 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) +1 mvnsite 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 20m 19s hadoop-yarn-client in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 41m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884136/YARN-6756.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 60e19670e49e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51881a8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17166/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17166/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/17166/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        makes sense..
        +1 pending Jenkins, Thanks

        Show
        asuresh Arun Suresh added a comment - makes sense.. +1 pending Jenkins, Thanks
        Hide
        jianhe Jian He added a comment - - edited

        Arun Suresh, ut added.
        there's one more issue, the relaxLocality was by default true for the old ContainerRequest constructors, but with the builder it is by default false
        I changed it to true, looks ok ?

        Show
        jianhe Jian He added a comment - - edited Arun Suresh , ut added. there's one more issue, the relaxLocality was by default true for the old ContainerRequest constructors, but with the builder it is by default false I changed it to true, looks ok ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 15m 21s trunk passed
        +1 compile 0m 24s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 findbugs 0m 35s trunk passed
        +1 javadoc 0m 16s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 42s the patch passed
        +1 javadoc 0m 12s the patch passed
              Other Tests
        +1 unit 20m 16s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        41m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884120/YARN-6756.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 97c7d1acb07a 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51881a8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17164/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17164/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 15m 21s trunk passed +1 compile 0m 24s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 26s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 16s trunk passed       Patch Compile Tests +1 mvninstall 0m 23s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 20m 16s hadoop-yarn-client in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 41m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884120/YARN-6756.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 97c7d1acb07a 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51881a8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17164/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/17164/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks Jian He, Straightforward patch.. LGTM
        Is it possible to put in a small testcase?

        Show
        asuresh Arun Suresh added a comment - Thanks Jian He , Straightforward patch.. LGTM Is it possible to put in a small testcase?
        Hide
        jianhe Jian He added a comment -

        Arun Suresh, can you help check this ?

        Show
        jianhe Jian He added a comment - Arun Suresh , can you help check this ?
        Hide
        jianhe Jian He added a comment -

        A simple patch to fix the npe

        Show
        jianhe Jian He added a comment - A simple patch to fix the npe
        Hide
        jianhe Jian He added a comment - - edited

        I actually run into NPE when I use the builder API, if the caller doesn't explicitly call executionTypeRequest() method to set a dummy object, it always throws NPE, I think we need to initialize it with a default object

        Show
        jianhe Jian He added a comment - - edited I actually run into NPE when I use the builder API, if the caller doesn't explicitly call executionTypeRequest() method to set a dummy object, it always throws NPE, I think we need to initialize it with a default object
        Hide
        asuresh Arun Suresh added a comment -

        yup.. if the ExecutionTypeRequest is explicitly set to 'null' in the ContainerRequest constructor.
        That ideally should not happen. Think we should probably either add it to the null check to the 'sanityCheck()' method and throw an Exception.. or we should handle it by resetting it to 'ExecutionTypeRequest.newInstance()' if the value is null in the ContainerRequest constructor.

        Show
        asuresh Arun Suresh added a comment - yup.. if the ExecutionTypeRequest is explicitly set to 'null' in the ContainerRequest constructor. That ideally should not happen. Think we should probably either add it to the null check to the 'sanityCheck()' method and throw an Exception.. or we should handle it by resetting it to 'ExecutionTypeRequest.newInstance()' if the value is null in the ContainerRequest constructor.
        Hide
        jianhe Jian He added a comment -

        HI Arun Suresh , can you help check this ?

        Show
        jianhe Jian He added a comment - HI Arun Suresh , can you help check this ?

          People

          • Assignee:
            jianhe Jian He
            Reporter:
            jianhe Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development