[BEAM-3605] Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.4.0
• Component/s: io-java-kinesis
• Labels: flake sickbay
• Assignee: Paweł Kaczmarczyk
• Reporter: Kenneth Knowles
• Created: 02/Feb/18 04:03
• Updated: 16/May/20 13:29
• Resolved: 14/Feb/18 13:43

Description

Here's one:

https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/

Filing all test failures as "Critical" so we can sickbay or fix.

The Jenkins build will get GC'd so here is the error:

java.lang.AssertionError: 
Expecting:
  <["shard1", "shard2"]>
to contain only:
  <["shard2"]>
but the following elements were unexpected:
  <["shard1"]>

	at org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)

and stderr

Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator reached end of the shard: streamName=null, shardId=shard1
	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard2 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator reached end of the shard: streamName=null, shardId=shard2
	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
INFO: Closing shard iterators pool
Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
WARNING: Transient exception occurred.
org.apache.beam.sdk.io.kinesis.TransientKinesisException

Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Shard iterator for shard1 shard is closed, finishing the read loop
org.apache.beam.sdk.io.kinesis.KinesisShardClosedException

Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:26 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
INFO: Closing shard iterators pool
Feb 01, 2018 11:24:26 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
WARNING: Thread was interrupted, finishing the read loop
java.lang.InterruptedException: sleep interrupted
	at java.lang.Thread.sleep(Native Method)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.lambda$shouldInterruptKinesisReadingAndStopShortly$0(ShardReadersPoolTest.java:150)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator$$EnhancerByMockitoWithCGLIB$$791f33eb.readNextBatch(<generated>)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
	at org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Feb 01, 2018 11:24:26 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished
Feb 01, 2018 11:24:26 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
INFO: Kinesis Shard read loop has finished

Issue Links

• links to: GitHub Pull Request #4661
• links to: GitHub Pull Request #4668