Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10260

TestFsDatasetImpl#testCleanShutdownOfVolume often fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This test failure occurs in upstream Jenkins. Looking at the test code, I think it should be improved to capture the root cause of failure:
      E.g. change Thread.sleep(1000) to GenericTestUtils.waitFor and use GenericTestUtils.assertExceptionContains to replace

      Assert.assertTrue(ioe.getMessage().contains(info.toString()));
      

      https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/

      java.lang.AssertionError: null
      at org.junit.Assert.fail(Assert.java:86)
      at org.junit.Assert.assertTrue(Assert.java:41)
      at org.junit.Assert.assertTrue(Assert.java:52)
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)

      Standard Error

      Exception in thread "DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
      at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
      at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
      at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
      at java.lang.Thread.run(Thread.java:744)

      1. HDFS-10260-v1.patch
        2 kB
        Rushabh S Shah
      2. HDFS-10260.patch
        2 kB
        Rushabh S Shah

        Issue Links

          Activity

          Hide
          shahrs87 Rushabh S Shah added a comment -

          Thanks Wei-Chiu Chuang for reporting the issue.

          Show
          shahrs87 Rushabh S Shah added a comment - Thanks Wei-Chiu Chuang for reporting the issue.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Sorry for cancelling the patch couple of times.

          Show
          shahrs87 Rushabh S Shah added a comment - Sorry for cancelling the patch couple of times.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Rushabh S Shah thanks for the contribution. I took a quick look at the patch and it looks good to me.
          Were you also able to find out what caused the test failure?

          Thanks

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Rushabh S Shah thanks for the contribution. I took a quick look at the patch and it looks good to me. Were you also able to find out what caused the test failure? Thanks
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Were you also able to find out what caused the test failure?

          I am not able to make it fail on my machine. It never failed on our internal Jenkins server also.
          I ran 20 times in a loop.
          May be after this change we will be able to see the actual exception when it fails on Apache Jenkins server.

          Show
          shahrs87 Rushabh S Shah added a comment - Were you also able to find out what caused the test failure? I am not able to make it fail on my machine. It never failed on our internal Jenkins server also. I ran 20 times in a loop. May be after this change we will be able to see the actual exception when it fails on Apache Jenkins server.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_77
          +1 compile 0m 41s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_77
          +1 javac 0m 35s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
          -1 unit 60m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 55m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          140m 9s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestDecommissionWithStriped
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797539/HDFS-10260-v1.patch
          JIRA Issue HDFS-10260
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 07103262377a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a62637a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15101/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15101/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 42s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_77 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_77 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 60m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 55m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 140m 9s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestDecommissionWithStriped JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797539/HDFS-10260-v1.patch JIRA Issue HDFS-10260 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 07103262377a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a62637a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15101/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15101/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Since I have changed only 1 test case I don't think my patch have caused failures in the other tests.
          So I think the test failures are not related.

          Show
          shahrs87 Rushabh S Shah added a comment - Since I have changed only 1 test case I don't think my patch have caused failures in the other tests. So I think the test failures are not related.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Wei-Chiu Chuang: do you have any additional comments on this patch ?

          Show
          shahrs87 Rushabh S Shah added a comment - Wei-Chiu Chuang : do you have any additional comments on this patch ?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          looks good to me. Thanks for the contribution.

          Show
          jojochuang Wei-Chiu Chuang added a comment - looks good to me. Thanks for the contribution.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 lgtm

          Show
          kihwal Kihwal Lee added a comment - +1 lgtm
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk, branch-2 and branch-2.8. Thanks for reporting it Wei-Chiu Chuang, and Rushabh S Shah for the patch.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.8. Thanks for reporting it Wei-Chiu Chuang , and Rushabh S Shah for the patch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9693 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9693/)
          HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. (kihwal: rev af9b000535cc987f66327e2b2bfe08923ba24c13)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9693 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9693/ ) HDFS-10260 . TestFsDatasetImpl#testCleanShutdownOfVolume often fails. (kihwal: rev af9b000535cc987f66327e2b2bfe08923ba24c13) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Kihwal Lee: Thanks for reviews and committing the patch.
          Wei-Chiu Chuang: Thanks for reporting the issue and reviewing the patch.

          Show
          shahrs87 Rushabh S Shah added a comment - Kihwal Lee : Thanks for reviews and committing the patch. Wei-Chiu Chuang : Thanks for reporting the issue and reviewing the patch.
          Hide
          boky01 Andras Bokor added a comment -

          I am not able to make it fail on my machine. It never failed on our internal Jenkins server also.

          FYI: It come up on my build

          Show
          boky01 Andras Bokor added a comment - I am not able to make it fail on my machine. It never failed on our internal Jenkins server also. FYI: It come up on my build
          Hide
          shahrs87 Rushabh S Shah added a comment -

          FYI: It come up on my build

          By my machine, I meant my mac.
          I found the root cause for the failure.
          Please see HDFS-10372 for more details.

          Show
          shahrs87 Rushabh S Shah added a comment - FYI: It come up on my build By my machine, I meant my mac. I found the root cause for the failure. Please see HDFS-10372 for more details.

            People

            • Assignee:
              shahrs87 Rushabh S Shah
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development