Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4514

ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: 1.1.0, 1.1.1
    • Fix Version/s: 1.2.0, 1.1.3
    • Component/s: Kinesis Connector
    • Labels:
      None

      Description

      Original mailing thread for the reported issue:
      http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html

      Normally, the exception is thrown when the consumer uses the same shard iterator after 5 minutes since it was retrieved. I've still yet to clarify & reproduce the root cause of the ExpiredIteratorException, because from the code this seems to be impossible. I'm leaning towards suspecting this is a Kinesis-side issue (from the description in the ML, the behaviour also seems indeterminate).

      Either way, the exception can be fairly easily handled so that the consumer doesn't just fail. When caught, we request a new shard iterator from Kinesis with the last sequence number.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: