[HBASE-19300] TestMultithreadedTableMapper fails in branch-1.4

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 19/Nov/17 03:33
• Updated: 01/Aug/18 06:22
• Resolved: 27/Nov/17 19:27

Description

From https://builds.apache.org/job/HBase-1.4/1023/jdk=JDK_1_7,label=Hadoop&&!H13/testReport/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/ :

java.lang.AssertionError
	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.verify(TestMultithreadedTableMapper.java:195)
	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.runTestOnTable(TestMultithreadedTableMapper.java:163)
	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.testMultithreadedTableMapper(TestMultithreadedTableMapper.java:136)

I ran the test locally which failed.
Noticed the following in test output:

2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t9] protobuf.ResponseConverter(425): Results sent from server=703. But only got 0 results completely at    client. Resetting the scanner to scan again.
2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t3] protobuf.ResponseConverter(425): Results sent from server=703. But only got 0 results completely at    client. Resetting the scanner to scan again.
2017-11-18 19:28:14,461 ERROR [hconnection-0x11db8653-shared--pool24-t8] protobuf.ResponseConverter(432): Exception while reading cells from result.Resetting the scanner to    scan again.
org.apache.hadoop.hbase.DoNotRetryIOException: Results sent from server=703. But only got 0 results completely at client. Resetting the scanner to scan again.
  at org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:426)
  at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
  at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
  at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
  at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
  at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
  at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
  at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
2017-11-18 19:28:14,464 ERROR [hconnection-0x11db8653-shared--pool24-t2] protobuf.ResponseConverter(432): Exception while reading cells from result.Resetting the scanner to    scan again.
java.io.EOFException: Partial cell read
  at org.apache.hadoop.hbase.codec.BaseDecoder.rethrowEofException(BaseDecoder.java:86)
  at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:70)
  at org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:419)
  at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
  at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
  at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
  at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
  at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
  at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
  at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Premature EOF from inputStream
  at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:202)
  at org.apache.hadoop.hbase.KeyValueUtil.iscreate(KeyValueUtil.java:611)
  at org.apache.hadoop.hbase.codec.KeyValueCodec$KeyValueDecoder.parseCell(KeyValueCodec.java:69)
  at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:67)
  ... 11 more