Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21228

[Kinesis][Producer] Deadlock in KinesisProducer

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Background
      Kinesis sink failed and resulted in deadlock:

      • Indefinite backpressure being applied
      • Exception never thrown causing job to fail

      Application running with:

      flinkKinesisProducer.setQueueLimit(1);
      flinkKinesisProducer.setFailOnError(true); 
      
      • KinesisProducer is waiting for queue to empty before sending the next record (code)
      • KPL ran out of memory, which raised an error, however this is processed async (code)
      • KinesisProducer would have rethrown the error and restarted the job, however operator stuck in an infinite loop enforcing the queue limit (which never clears) (code)

      Proposal

      • checkAndPropagateAsyncError() while enforcing queue limit in enforceQueueLimit() to break deadlock

        Attachments

          Activity

            People

            • Assignee:
              dannycranmer Danny Cranmer
              Reporter:
              danny.cranmer Danny Cranmer

              Dates

              • Created:
                Updated:

                Issue deployment