[HDFS-11882] Precisely calculate acked length of striped block groups in updatePipeline

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Component/s: erasure-coding, test
• Labels: hdfs-ec-3.0-must-do
• Target Version/s: 3.0.0-beta1
• Assignee: Andrew Wang
• Reporter: Akira Ajisaka
• Created: 25/May/17 04:02
• Updated: 24/Apr/18 20:55
• Resolved: 05/Sep/17 21:16

Description

Some tests of erasure coding fails by the following exception. The following test was removed by HDFS-11823, however, this type of error can happen in real cluster.

Running org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure)  Time elapsed: 38.831 sec  <<< ERROR!
java.lang.IllegalStateException: null
	at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
	at org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
	at org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
	at org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
	at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
	at org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
	at org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381)
	at org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Issue Links

• is broken by: HDFS-9342 Erasure coding: client should update and commit block based on acknowledged size
• is duplicated by: HDFS-11813 TestDFSStripedOutputStreamWithFailure070 failed randomly