Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10463

TestRollingFileSystemSinkWithHdfs needs some cleanup

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are three primary issues. The most significant is that the testFlushThread() method doesn't clean up after itself, which can cause other tests to fail. The other big issue is that the testSilentAppend() method is testing the wrong thing. An additional minor issue is that none of the tests are careful about making sure the metrics system gets shutdown in all cases.

      1. HDFS-10463.branch-2.001.patch
        5 kB
        Daniel Templeton
      2. HDFS-10463.001.patch
        6 kB
        Daniel Templeton

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s HDFS-10463 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806224/HDFS-10463.001.patch
        JIRA Issue HDFS-10463
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15565/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HDFS-10463 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806224/HDFS-10463.001.patch JIRA Issue HDFS-10463 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15565/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Here's a branch-2 patch.

        Show
        templedf Daniel Templeton added a comment - Here's a branch-2 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 45s Maven dependency ordering for branch
        +1 mvninstall 6m 48s trunk passed
        +1 compile 6m 43s trunk passed
        +1 checkstyle 1m 22s trunk passed
        +1 mvnsite 1m 51s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 13s trunk passed
        +1 javadoc 2m 6s trunk passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 6m 38s the patch passed
        +1 javac 6m 38s the patch passed
        -1 checkstyle 1m 22s root: patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11)
        +1 mvnsite 1m 43s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 24s the patch passed
        +1 javadoc 2m 3s the patch passed
        +1 unit 7m 40s hadoop-common in the patch passed.
        -1 unit 61m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        111m 20s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline
          hadoop.hdfs.TestDFSUpgradeFromImage



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806240/HDFS-10463.001.patch
        JIRA Issue HDFS-10463
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7ec3d98ac29f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3c83cee
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15567/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15567/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 6m 43s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 2m 6s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 6m 38s the patch passed +1 javac 6m 38s the patch passed -1 checkstyle 1m 22s root: patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 2m 3s the patch passed +1 unit 7m 40s hadoop-common in the patch passed. -1 unit 61m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 111m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.TestDFSUpgradeFromImage Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806240/HDFS-10463.001.patch JIRA Issue HDFS-10463 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7ec3d98ac29f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c83cee Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15567/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15567/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15567/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 31s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 2m 45s Maven dependency ordering for branch
        +1 mvninstall 7m 47s branch-2 passed
        +1 compile 6m 36s branch-2 passed with JDK v1.8.0_91
        +1 compile 6m 58s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 1m 29s branch-2 passed
        +1 mvnsite 1m 59s branch-2 passed
        +1 mvneclipse 0m 34s branch-2 passed
        -1 findbugs 2m 6s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings.
        +1 javadoc 2m 8s branch-2 passed with JDK v1.8.0_91
        +1 javadoc 3m 4s branch-2 passed with JDK v1.7.0_101
        0 mvndep 0m 23s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        +1 compile 6m 42s the patch passed with JDK v1.8.0_91
        +1 javac 6m 42s the patch passed
        +1 compile 6m 36s the patch passed with JDK v1.7.0_101
        +1 javac 6m 36s the patch passed
        -1 checkstyle 1m 23s root: patch generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12)
        +1 mvnsite 1m 48s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 4m 4s the patch passed
        +1 javadoc 2m 1s the patch passed with JDK v1.8.0_91
        +1 javadoc 2m 51s the patch passed with JDK v1.7.0_101
        +1 unit 7m 24s hadoop-common in the patch passed with JDK v1.8.0_91.
        -1 unit 53m 14s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 7m 32s hadoop-common in the patch failed with JDK v1.7.0_101.
        -1 unit 50m 57s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 32s Patch does not generate ASF License warnings.
        197m 56s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
        JDK v1.7.0_101 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806254/HDFS-10463.branch-2.001.patch
        JIRA Issue HDFS-10463
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux db163568d184 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / da77040
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15569/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15569/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 45s Maven dependency ordering for branch +1 mvninstall 7m 47s branch-2 passed +1 compile 6m 36s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 58s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 29s branch-2 passed +1 mvnsite 1m 59s branch-2 passed +1 mvneclipse 0m 34s branch-2 passed -1 findbugs 2m 6s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. +1 javadoc 2m 8s branch-2 passed with JDK v1.8.0_91 +1 javadoc 3m 4s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 6m 42s the patch passed with JDK v1.8.0_91 +1 javac 6m 42s the patch passed +1 compile 6m 36s the patch passed with JDK v1.7.0_101 +1 javac 6m 36s the patch passed -1 checkstyle 1m 23s root: patch generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 27s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 4s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_101 +1 unit 7m 24s hadoop-common in the patch passed with JDK v1.8.0_91. -1 unit 53m 14s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 7m 32s hadoop-common in the patch failed with JDK v1.7.0_101. -1 unit 50m 57s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 32s Patch does not generate ASF License warnings. 197m 56s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.balancer.TestBalancer   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_101 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806254/HDFS-10463.branch-2.001.patch JIRA Issue HDFS-10463 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux db163568d184 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / da77040 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15569/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15569/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15569/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        atm Aaron T. Myers added a comment -

        Daniel Templeton - trunk patch looks pretty good to me, but seems like the branch-2 patch had this very test fail. I'll be +1 once that's addressed.

        Show
        atm Aaron T. Myers added a comment - Daniel Templeton - trunk patch looks pretty good to me, but seems like the branch-2 patch had this very test fail. I'll be +1 once that's addressed.
        Hide
        templedf Daniel Templeton added a comment -

        That issue is preexisting: HDFS-10449.

        Show
        templedf Daniel Templeton added a comment - That issue is preexisting: HDFS-10449 .
        Hide
        atm Aaron T. Myers added a comment -

        Good point! +1, looks good to me.

        Thanks, Daniel.

        Show
        atm Aaron T. Myers added a comment - Good point! +1, looks good to me. Thanks, Daniel.
        Hide
        kasha Karthik Kambatla added a comment -

        +1. Checking this in.

        Show
        kasha Karthik Kambatla added a comment - +1. Checking this in.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9867/)
        HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (kasha: rev 55c3e2de3d636482ef2c51bdf88e89a34fc58b32)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9867/ ) HDFS-10463 . TestRollingFileSystemSinkWithHdfs needs some cleanup. (kasha: rev 55c3e2de3d636482ef2c51bdf88e89a34fc58b32) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
        Hide
        kasha Karthik Kambatla added a comment -

        Just committed this to trunk and branch-2.

        Thanks Daniel Templeton for fixing the test, and Aaron T. Myers for the review.

        Show
        kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks Daniel Templeton for fixing the test, and Aaron T. Myers for the review.

          People

          • Assignee:
            templedf Daniel Templeton
            Reporter:
            templedf Daniel Templeton
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development