[FLINK-7223] Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Connectors / Kinesis
• Assignee: Bowen Li
• Reporter: Bowen Li
• Created: 19/Jul/17 04:43
• Updated: 02/Oct/19 17:43
• Resolved: 03/Oct/17 22:16

Description

Background: DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS in org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants is the default value for Flink to call Kinesis's describeStream() API.

Problem: Right now, its value is 10,000millis (10sec), which is too short. We ran into problems that Flink-kinesis-connector's call of describeStream() exceeds Kinesis rate limit, and broken Flink taskmanager.

According to http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,
"This operation has a limit of 10 transactions per second per account.". What it means is that the 10transaction/account is a limit on a single organization's AWS account...... We contacted AWS Support, and confirmed this. If you have more applications (either other Flink apps or non-Flink apps) competing aggressively with your Flink app on this API, your Flink app breaks.

I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from 10,000millis(10sec) to preferably 300,000 (5min). Or at least 60,000 (1min) if anyone has a solid reason arguing that 5min is too long,

This is also related to https://issues.apache.org/jira/browse/FLINK-6365