[FLINK-4514] ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Resolved
Affects Version/s: 1.1.0, 1.1.1
Fix Version/s: 1.1.3, 1.2.0
Component/s: Connectors / Kinesis
Labels:
None

Description

Original mailing thread for the reported issue:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html

Normally, the exception is thrown when the consumer uses the same shard iterator after 5 minutes since it was retrieved. I've still yet to clarify & reproduce the root cause of the ExpiredIteratorException, because from the code this seems to be impossible. I'm leaning towards suspecting this is a Kinesis-side issue (from the description in the ML, the behaviour also seems indeterminate).

Either way, the exception can be fairly easily handled so that the consumer doesn't just fail. When caught, we request a new shard iterator from Kinesis with the last sequence number.