Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19899

[Kinesis][EFO] Optimise error handling to use a separate exception delivery mechanism

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: Connectors / Kinesis
    • Labels:
      None

      Description

      Background

      There is a queue used to pass events between the network client and consumer application. When an error is thrown in the network thread, the queue is cleared to make space for the error event. This means that records will be thrown away to make space for errors (the records would be subsequently reloaded from the shard).

      Scope

      Add a new mechanism to pass exceptions between threads, meaning data does not need to be discarded. When an error is thrown, the error event will be processed by the consumer once all of the records have been processed.

        Attachments

          Activity

            People

            • Assignee:
              danny.cranmer Danny Cranmer
              Reporter:
              danny.cranmer Danny Cranmer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: