[HDFS-13339] Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.2.0, 3.1.1, 3.0.4
• Component/s: datanode
• Labels: DataNode volumes

Environment

os: Linux 2.6.32-358.el6.x86_64
hadoop version: hadoop-3.2.0-SNAPSHOT
unit: mvn test -Pnative -Dtest=TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart

• Hadoop Flags: Reviewed
• Assignee: Zsolt Venczel
• Reporter: liaoyuxiangqin
• Created: 23/Mar/18 07:40
• Updated: 04/Jun/18 05:56
• Resolved: 04/Jun/18 05:39

Description

When i execute Unit Test of
TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart, the process blocks on waitReplication, detail information as follows:

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 307.492 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
[ERROR] testVolFailureStatsPreservedOnNNRestart(org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting) Time elapsed: 307.206 s <<< ERROR!
java.util.concurrent.TimeoutException: Timed out waiting for /test1 to reach 2 replicas
at org.apache.hadoop.hdfs.DFSTestUtil.waitReplication(DFSTestUtil.java:800)
at org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testVolFailureStatsPreservedOnNNRestart(TestDataNodeVolumeFailureReporting.java:283)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)