Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9897

Further enhance adaptive reads in Kinesis Connector to depend on run loop time

    XMLWordPrintableJSON

    Details

      Description

      In FLINK-9692, we introduced the ability for the shardConsumer to adaptively read more records based on the current average record size to optimize the 2 Mb/sec shard limit. The feature maximizes  maxNumberOfRecordsPerFetch of 5 reads/sec (as prescribed by Kinesis limits). In the case where applications take more time to process records in the run loop, they are no longer able to read at a frequency of 5 reads/sec (even though their fetchIntervalMillis maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch should be calculated based on the time that the run loop actually takes as opposed to fetchIntervalMillis. 

        Attachments

          Activity

            People

            • Assignee:
              glaksh100 Lakshmi Rao
              Reporter:
              glaksh100 Lakshmi Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: