[FLINK-9897] Further enhance adaptive reads in Kinesis Connector to depend on run loop time

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.1
• Fix Version/s: 1.6.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Lakshmi Rao
• Reporter: Lakshmi Rao
• Created: 19/Jul/18 19:34
• Updated: 01/Aug/18 13:30
• Resolved: 01/Aug/18 13:30

Description

In FLINK-9692, we introduced the ability for the shardConsumer to adaptively read more records based on the current average record size to optimize the 2 Mb/sec shard limit. The feature maximizes  maxNumberOfRecordsPerFetch of 5 reads/sec (as prescribed by Kinesis limits). In the case where applications take more time to process records in the run loop, they are no longer able to read at a frequency of 5 reads/sec (even though their fetchIntervalMillis maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch should be calculated based on the time that the run loop actually takes as opposed to fetchIntervalMillis. 

Issue Links

• links to: GitHub Pull Request #6408
• links to: GitHub Pull Request #6408
• links to: GitHub Pull Request #6409