Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4976

Missing NullPointer check in ContainerLaunchContextPBImpl causes RM to die

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The client can set a null value for any env variable.

      1. YARN-4976.v3.patch
        4 kB
        Giovanni Matteo Fumarola
      2. YARN-4976.v2.patch
        4 kB
        Giovanni Matteo Fumarola
      3. YARN-4976.v1.patch
        1 kB
        Giovanni Matteo Fumarola
      4. YARN-4976.v0.patch
        1 kB
        Giovanni Matteo Fumarola

        Activity

        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Thanks Chris Douglas and Daniel Templeton for reviewing and commit it.
        Thanks Ellen Hui for finding the issue.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Thanks Chris Douglas and Daniel Templeton for reviewing and commit it. Thanks Ellen Hui for finding the issue.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9645/)
        YARN-4976. Missing NullPointer check in ContainerLaunchContextPBImpl. (cdouglas: rev 95a50466075c28110fa7c297e9c5246892076ca8)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerLaunchContextPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/impl/pb/TestApplicationClientProtocolRecords.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9645/ ) YARN-4976 . Missing NullPointer check in ContainerLaunchContextPBImpl. (cdouglas: rev 95a50466075c28110fa7c297e9c5246892076ca8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerLaunchContextPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/impl/pb/TestApplicationClientProtocolRecords.java
        Hide
        chris.douglas Chris Douglas added a comment -

        +1 Committed to trunk, branch-2. Thanks Gio

        Show
        chris.douglas Chris Douglas added a comment - +1 Committed to trunk, branch-2. Thanks Gio
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 44s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_77
        +1 compile 0m 26s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 8s trunk passed
        +1 javadoc 0m 27s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 33s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_77
        +1 javac 0m 20s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.7.0_95
        +1 javac 0m 24s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 24s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
        +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_77.
        +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        20m 36s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799990/YARN-4976.v3.patch
        JIRA Issue YARN-4976
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ad198258ccb0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7da5847
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11161/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11161/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_77 +1 compile 0m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 27s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 33s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 26s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_77 +1 javac 0m 20s the patch passed +1 compile 0m 24s the patch passed with JDK v1.7.0_95 +1 javac 0m 24s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 24s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_77. +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 20m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799990/YARN-4976.v3.patch JIRA Issue YARN-4976 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ad198258ccb0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7da5847 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11161/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/11161/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Uploading a new patch without a whitespace in the comment.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Uploading a new patch without a whitespace in the comment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 34s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_77
        +1 compile 0m 25s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 31s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_77
        +1 javac 0m 20s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 23s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 28s the patch passed with JDK v1.7.0_95
        +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_77.
        +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        19m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799885/YARN-4976.v2.patch
        JIRA Issue YARN-4976
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e12237491d39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1e48eef
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11151/artifact/patchprocess/whitespace-eol.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11151/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11151/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 34s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_77 +1 compile 0m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 31s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 25s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_77 +1 javac 0m 20s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 23s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 28s the patch passed with JDK v1.7.0_95 +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_77. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 19m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799885/YARN-4976.v2.patch JIRA Issue YARN-4976 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e12237491d39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1e48eef Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11151/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11151/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/11151/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Cool, I just uploaded the new version with a test class and an UTC for this particular bug.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Cool, I just uploaded the new version with a test class and an UTC for this particular bug.
        Hide
        templedf Daniel Templeton added a comment -

        Patch looks good. Can you add a quick test for this case? There isn't a test class for the CLCPBImpl, so you'll have to create one. It seems kinda futile to just add this one test, but we have to start somewhere.

        Show
        templedf Daniel Templeton added a comment - Patch looks good. Can you add a quick test for this case? There isn't a test class for the CLCPBImpl, so you'll have to create one. It seems kinda futile to just add this one test, but we have to start somewhere.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 31s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_77
        +1 compile 0m 26s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 31s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 21s the patch passed with JDK v1.8.0_77
        +1 javac 0m 21s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 1s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 25s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
        +1 unit 1m 57s hadoop-yarn-common in the patch passed with JDK v1.8.0_77.
        +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 16s Patch does not generate ASF License warnings.
        20m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799846/YARN-4976.v1.patch
        JIRA Issue YARN-4976
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8cc03f7b6e1e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 63ac2db
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11147/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11147/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 31s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_77 +1 compile 0m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 26s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_77 +1 javac 0m 21s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 25s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 +1 unit 1m 57s hadoop-yarn-common in the patch passed with JDK v1.8.0_77. +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 20m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799846/YARN-4976.v1.patch JIRA Issue YARN-4976 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8cc03f7b6e1e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63ac2db Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11147/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/11147/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 54s trunk passed
        +1 compile 0m 24s trunk passed with JDK v1.8.0_77
        +1 compile 0m 28s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 8s trunk passed
        +1 javadoc 0m 28s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 34s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.8.0_77
        +1 javac 0m 22s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_95
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 25s the patch passed
        +1 javadoc 0m 27s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 31s the patch passed with JDK v1.7.0_95
        +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_77.
        +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        21m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799831/YARN-4976.v0.patch
        JIRA Issue YARN-4976
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 215fb44072a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 63ac2db
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11144/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11144/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 54s trunk passed +1 compile 0m 24s trunk passed with JDK v1.8.0_77 +1 compile 0m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 28s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 34s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 28s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_77 +1 javac 0m 22s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_95 +1 javac 0m 25s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 25s the patch passed +1 javadoc 0m 27s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 31s the patch passed with JDK v1.7.0_95 +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_77. +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 21m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799831/YARN-4976.v0.patch JIRA Issue YARN-4976 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 215fb44072a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63ac2db Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11144/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/11144/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Yes, (by mistake) we set YARN_APPLICATION_LOG_FOLDER equal to null in the client side.

        I uploaded a new version (v1) without the ternary operator.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Yes, (by mistake) we set YARN_APPLICATION_LOG_FOLDER equal to null in the client side. I uploaded a new version (v1) without the ternary operator.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Giovanni Matteo Fumarola. I should have taken a closer look the first time. Can we make

                    String value =
                        (environment.get(key) == null) ? "" : environment.get(key);
        

        be

                    String value = environment.get(key);
        
                    if (value == null) {
                      value = "";
                    } 
        

        It saves a method call in the common case and avoids the ternary operator, which is cute but not as easy to read.

        Did you actually see this issue in live cluster? I just ran through the code and couldn't find anywhere that an env map was passed to the CLC without scrubbing it. A wayward YARN app could do it, but I'm curious how you tripped over the issue.

        Show
        templedf Daniel Templeton added a comment - Thanks, Giovanni Matteo Fumarola . I should have taken a closer look the first time. Can we make String value = (environment.get(key) == null ) ? "" : environment.get(key); be String value = environment.get(key); if (value == null ) { value = ""; } It saves a method call in the common case and avoids the ternary operator, which is cute but not as easy to read. Did you actually see this issue in live cluster? I just ran through the code and couldn't find anywhere that an env map was passed to the CLC without scrubbing it. A wayward YARN app could do it, but I'm curious how you tripped over the issue.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Let me reattached it. Thanks for the quick feedback.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Let me reattached it. Thanks for the quick feedback.
        Hide
        templedf Daniel Templeton added a comment -

        Looks reasonable. The indentation is off, though.

        Show
        templedf Daniel Templeton added a comment - Looks reasonable. The indentation is off, though.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Attached a "nice" way to handle it.

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Attached a "nice" way to handle it.
        Hide
        giovanni.fumarola Giovanni Matteo Fumarola added a comment -

        Missing null check

        @Override
        public StringStringMapProto next() {
        String key = keyIter.next();
        return StringStringMapProto.newBuilder().setKey(key).setValue(
        (environment.get(key))).build();
        }

        environment.get(key) this can result null.

        Log from the RM:
        2016-04-19 12:33:11,796 ERROR org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: State store operation failed
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:381)
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:365)
        at com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
        at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
        at org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllEnvironment(YarnProtos.java:47764)
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addEnvToProto(ContainerLaunchContextPBImpl.java:394)
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:116)
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:129)
        at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:71)
        at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.convertToProtoFormat(ApplicationSubmissionContextPBImpl.java:447)
        at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.mergeLocalToBuilder(ApplicationSubmissionContextPBImpl.java:112)
        at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.mergeLocalToProto(ApplicationSubmissionContextPBImpl.java:139)
        at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.getProto(ApplicationSubmissionContextPBImpl.java:78)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.mergeLocalToBuilder(ApplicationStateDataPBImpl.java:62)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.mergeLocalToProto(ApplicationStateDataPBImpl.java:71)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.getProto(ApplicationStateDataPBImpl.java:54)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:638)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:198)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:180)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:910)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:991)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:986)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
        at java.lang.Thread.run(Thread.java:745)

        Show
        giovanni.fumarola Giovanni Matteo Fumarola added a comment - Missing null check @Override public StringStringMapProto next() { String key = keyIter.next(); return StringStringMapProto.newBuilder().setKey(key).setValue( (environment.get(key))).build(); } environment.get(key) this can result null. Log from the RM: 2016-04-19 12:33:11,796 ERROR org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: State store operation failed at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:381) at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:365) at com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336) at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323) at org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllEnvironment(YarnProtos.java:47764) at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addEnvToProto(ContainerLaunchContextPBImpl.java:394) at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:116) at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:129) at org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:71) at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.convertToProtoFormat(ApplicationSubmissionContextPBImpl.java:447) at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.mergeLocalToBuilder(ApplicationSubmissionContextPBImpl.java:112) at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.mergeLocalToProto(ApplicationSubmissionContextPBImpl.java:139) at org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.getProto(ApplicationSubmissionContextPBImpl.java:78) at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.mergeLocalToBuilder(ApplicationStateDataPBImpl.java:62) at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.mergeLocalToProto(ApplicationStateDataPBImpl.java:71) at org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb.ApplicationStateDataPBImpl.getProto(ApplicationStateDataPBImpl.java:54) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:638) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:198) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:180) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:910) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:991) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:986) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109) at java.lang.Thread.run(Thread.java:745)

          People

          • Assignee:
            giovanni.fumarola Giovanni Matteo Fumarola
            Reporter:
            giovanni.fumarola Giovanni Matteo Fumarola
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development