Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23802

[kinesis][efo] Reduce ReadTimeoutExceptions for Kinesis Consumer

    XMLWordPrintableJSON

    Details

      Description

       Background

      The Kinesis EFO consumer uses an async AWS SDK Netty client to read records from Kinesis. When the client is inactive for 30 seconds a ReadTimeoutException is thrown by Netty. The consumer will terminate the subscription, backoff and retry. Jobs with high backpressure can result in frequent ReadTImeoutException and the frequent backoff and retry can cause unnecessary overhead.

      What?

      Reduce/eliminate ReadTimeoutException from the EFO consumer

      How?

      There are 2 improvements to be made:
      1. Request next record from the Flink source thread rather than the AWS SDK response thread. This means that there will always be space in the input buffer queue. The AWS SDK async response thread is no longer blocking on this queue. Backpressure is now applied by the Flink source thread rather than the AWS SDK thread.
      2. Increase the Read Timeout (30s) to be higher than the maximum Shard subscription duration (5m) and enable TCP keep alive

      References

      This has already been implemented and tested in amazon-kinesis-connector-flink:

       

        Attachments

          Activity

            People

            • Assignee:
              dannycranmer Danny Cranmer
              Reporter:
              dannycranmer Danny Cranmer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: