[HDFS-10260] TestFsDatasetImpl#testCleanShutdownOfVolume often fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: datanode, test
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Wei-Chiu Chuang
• Created: 05/Apr/16 14:55
• Updated: 02/Oct/19 17:15
• Resolved: 29/Apr/16 15:08

Description

This test failure occurs in upstream Jenkins. Looking at the test code, I think it should be improved to capture the root cause of failure:
E.g. change Thread.sleep(1000) to GenericTestUtils.waitFor and use GenericTestUtils.assertExceptionContains to replace

Assert.assertTrue(ioe.getMessage().contains(info.toString()));

https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/

java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)

Standard Error

Exception in thread "DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
at java.lang.Thread.run(Thread.java:744)

Issue Links

• breaks: HDFS-9874 Long living DataXceiver threads cause volume shutdown to block.
• is related to: HDFS-10372 Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume