[FLINK-7367] Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Connectors / Kinesis
• Assignee: Bowen Li
• Reporter: Bowen Li
• Created: 04/Aug/17 00:05
• Updated: 07/Sep/17 06:24
• Resolved: 07/Sep/17 06:24

Description

Right now, FlinkKinesisProducer only expose two configs for the underlying KinesisProducer:

• AGGREGATION_MAX_COUNT
• COLLECTION_MAX_COUNT

Well, according to AWS doc and their sample on github, developers can set more to make the max use of KinesisProducer, and make it fault-tolerant (e.g. by increasing timeout).

I select a few more configs that we need when using Flink with Kinesis:

• MAX_CONNECTIONS
• RATE_LIMIT
• RECORD_MAX_BUFFERED_TIME
• RECORD_TIME_TO_LIVE
• REQUEST_TIMEOUT

Flink is using KPL's default values. They make Flink writing too fast to Kinesis, which fail Flink job too frequently. We need to parameterize FlinkKinesisProducer to pass in the above params, in order to slowing down Flink's write rate to Kinesis.

Issue Links

• links to: GitHub Pull Request #4473