[HDFS-10878] TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 2.7.4, 3.0.0-alpha2
• Component/s: hdfs-client
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 20/Sep/16 18:06
• Updated: 04/Oct/16 16:35
• Resolved: 04/Oct/16 16:27

Description

This failed in our internal build

java.util.ConcurrentModificationException: null
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
	at java.util.ArrayList$Itr.next(ArrayList.java:851)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
	at java.lang.String.valueOf(String.java:2982)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
	at com.google.common.base.Joiner.toString(Joiner.java:533)
	at com.google.common.base.Joiner.appendTo(Joiner.java:124)
	at com.google.common.base.Joiner.appendTo(Joiner.java:181)
	at com.google.common.base.Joiner.join(Joiner.java:237)
	at com.google.common.base.Joiner.join(Joiner.java:226)
	at com.google.common.base.Joiner.join(Joiner.java:245)
	at org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
	at org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
	at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete(<generated>)
	at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
	at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
	at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
	at org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)

Its getting NPE in following Log message

TestDFSClientRetries.java

 @Test
  public void testIdempotentAllocateBlockAndClose() throws Exception {
            ...
            public Boolean answer(InvocationOnMock invocation) throws Throwable {
          // complete() may return false a few times before it returns
          // true. We want to wait until it returns true, and then
          // make it retry one more time after that.
          LOG.info("Called complete(: " +
              Joiner.on(",").join(invocation.getArguments()) + ")");
           ...
}