[FLINK-4018] Configurable idle time between getRecords requests to Kinesis shards

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0
• Component/s: Connectors / Common, Connectors / Kinesis
• Assignee: Tzu-Li (Gordon) Tai
• Reporter: Tzu-Li (Gordon) Tai
• Created: 04/Jun/16 02:59
• Updated: 12/Jul/16 09:01
• Resolved: 12/Jul/16 08:18

Description

Currently, the Kinesis consumer is calling getRecords() right after finishing previous calls. This results in easily reaching Amazon's limitation of 5 GET requests per shard per second. Although the code already has backoff & retry mechanism, this will affect other applications consuming from the same Kinesis stream.

Along with this new configuration and the already existing `KinesisConfigConstants.CONFIG_SHARD_RECORDS_PER_GET`, users will have more control on the desired throughput behaviour for the Kinesis consumer.

Issue Links

• links to: GitHub Pull Request #2071