Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10536

Flink Kinesis Consumer leading to job failure due to ProvisionedThroughputExceededException

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.5.2
    • Fix Version/s: None
    • Component/s: Connectors / Kinesis
    • Labels:
      None

      Description

      Background:

      Flink Kinesis consumer, while consuming message from a shard, it retries for a configured number of attempts and then after it throws "ProvisionedThroughputExceededException" and which bubbles up to top level and eventually which leads to failure of the running streaming job.

      Improvement:

      In some scenario it may be desirable that to skip some of the records by skipping that "shardIterator" and move to next iterator in "ShardConsumer" and log this as metrics. This can be configurable through some property. This way those application which can tolerate to skip some messages can continue to keep running.

       

      If this idea seems useful, i would like to contribute on this. Please let me know the process

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              raviratnakar Ravi Bhushan Ratnakar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: