Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9858

RollingFileSystemSink can throw an NPE on non-secure clusters

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If the sink init fails (such as because the HDFS cluster isn't running) on a non-secure cluster, the init will throw an NPE because of missing properties.

      1. HADOOP-12835.001.patch
        6 kB
        Daniel Templeton
      2. HDFS-9858.002.patch
        16 kB
        Daniel Templeton

        Activity

        Hide
        templedf Daniel Templeton added a comment -

        Oops. Leapt before looking.

        Show
        templedf Daniel Templeton added a comment - Oops. Leapt before looking.
        Hide
        templedf Daniel Templeton added a comment -

        This patch should be better. I also rolled in a lazy initialization of the HDFS connection.

        Show
        templedf Daniel Templeton added a comment - This patch should be better. I also rolled in a lazy initialization of the HDFS connection.
        Hide
        kasha Karthik Kambatla added a comment -

        Looks good. +1, pending Jenkins.

        Show
        kasha Karthik Kambatla added a comment - Looks good. +1, pending Jenkins.
        Hide
        templedf Daniel Templeton added a comment -

        Bumping Jenkins

        Show
        templedf Daniel Templeton added a comment - Bumping Jenkins
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 1s trunk passed
        +1 compile 7m 15s trunk passed with JDK v1.8.0_72
        +1 compile 6m 55s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 32s trunk passed
        +1 javadoc 0m 57s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 42s the patch passed
        +1 compile 7m 12s the patch passed with JDK v1.8.0_72
        +1 javac 7m 12s the patch passed
        +1 compile 6m 58s the patch passed with JDK v1.7.0_95
        +1 javac 6m 58s the patch passed
        -1 checkstyle 0m 19s hadoop-common-project/hadoop-common: patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
        +1 mvnsite 0m 59s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 57s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 11s the patch passed with JDK v1.7.0_95
        +1 unit 8m 58s hadoop-common in the patch passed with JDK v1.8.0_72.
        -1 unit 8m 33s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        65m 46s



        Reason Tests
        JDK v1.7.0_95 Failed junit tests hadoop.fs.TestFsShellReturnCode



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789868/HADOOP-12835.001.patch
        JIRA Issue HADOOP-12835
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fc99300915af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6979cbf
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 1s trunk passed +1 compile 7m 15s trunk passed with JDK v1.8.0_72 +1 compile 6m 55s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 42s the patch passed +1 compile 7m 12s the patch passed with JDK v1.8.0_72 +1 javac 7m 12s the patch passed +1 compile 6m 58s the patch passed with JDK v1.7.0_95 +1 javac 6m 58s the patch passed -1 checkstyle 0m 19s hadoop-common-project/hadoop-common: patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 11s the patch passed with JDK v1.7.0_95 +1 unit 8m 58s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 8m 33s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 65m 46s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.fs.TestFsShellReturnCode Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789868/HADOOP-12835.001.patch JIRA Issue HADOOP-12835 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fc99300915af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6979cbf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8717/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Oops, I did it again. Moving this JIRA over to HDFS because I need to touch the HDFS test classes. sigh

        Show
        templedf Daniel Templeton added a comment - Oops, I did it again. Moving this JIRA over to HDFS because I need to touch the HDFS test classes. sigh
        Hide
        templedf Daniel Templeton added a comment -

        Here's an updated patch that includes a test for the fix.

        Show
        templedf Daniel Templeton added a comment - Here's an updated patch that includes a test for the fix.
        Hide
        templedf Daniel Templeton added a comment -

        Tweaking the last patch a little...

        Show
        templedf Daniel Templeton added a comment - Tweaking the last patch a little...
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 17s trunk passed
        +1 compile 6m 57s trunk passed with JDK v1.8.0_72
        +1 compile 7m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 10s trunk passed
        +1 mvnsite 2m 4s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 3m 39s trunk passed
        +1 javadoc 2m 7s trunk passed with JDK v1.8.0_72
        +1 javadoc 3m 1s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 34s the patch passed
        +1 compile 6m 50s the patch passed with JDK v1.8.0_72
        +1 javac 6m 50s the patch passed
        +1 compile 7m 20s the patch passed with JDK v1.7.0_95
        +1 javac 7m 20s the patch passed
        +1 checkstyle 1m 10s the patch passed
        +1 mvnsite 2m 2s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 13s the patch passed
        +1 javadoc 2m 3s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95
        +1 unit 8m 36s hadoop-common in the patch passed with JDK v1.8.0_72.
        -1 unit 59m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 8m 13s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 unit 54m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        196m 25s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
        JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789970/HDFS-9858.002.patch
        JIRA Issue HDFS-9858
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6e458aa42dfd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6979cbf
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14614/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14614/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 17s trunk passed +1 compile 6m 57s trunk passed with JDK v1.8.0_72 +1 compile 7m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 4s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 39s trunk passed +1 javadoc 2m 7s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 1s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 6m 50s the patch passed with JDK v1.8.0_72 +1 javac 6m 50s the patch passed +1 compile 7m 20s the patch passed with JDK v1.7.0_95 +1 javac 7m 20s the patch passed +1 checkstyle 1m 10s the patch passed +1 mvnsite 2m 2s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 13s the patch passed +1 javadoc 2m 3s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95 +1 unit 8m 36s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 59m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 8m 13s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 54m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 196m 25s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789970/HDFS-9858.002.patch JIRA Issue HDFS-9858 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6e458aa42dfd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6979cbf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14614/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14614/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14614/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Test failures are unrelated.

        Show
        templedf Daniel Templeton added a comment - Test failures are unrelated.
        Hide
        kasha Karthik Kambatla added a comment -

        +1. Checking this in.

        Show
        kasha Karthik Kambatla added a comment - +1. Checking this in.
        Hide
        templedf Daniel Templeton added a comment -
        Show
        templedf Daniel Templeton added a comment - Thanks, Karthik Kambatla !
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks for the fix, Daniel.

        Committed to trunk and branch-2. Had to resolve conflicts on branch-2 manually.

        Show
        kasha Karthik Kambatla added a comment - Thanks for the fix, Daniel. Committed to trunk and branch-2. Had to resolve conflicts on branch-2 manually.
        Hide
        templedf Daniel Templeton added a comment -

        Conflicts on branch-2? That doesn't sound right. The RFSSink code should be identical in both branches...

        Show
        templedf Daniel Templeton added a comment - Conflicts on branch-2? That doesn't sound right. The RFSSink code should be identical in both branches...
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9372 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9372/)
        HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure (kasha: rev c2460dad642feee1086442d33c30c24ec77236b9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9372 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9372/ ) HDFS-9858 . RollingFileSystemSink can throw an NPE on non-secure (kasha: rev c2460dad642feee1086442d33c30c24ec77236b9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java

          People

          • Assignee:
            templedf Daniel Templeton
            Reporter:
            templedf Daniel Templeton
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development