[SPARK-20140] Remove hardcoded kinesis retry wait and max retries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0, 2.3.0
• Component/s: DStreams
• Labels: kinesis recovery
• Assignee: Yash Sharma
• Reporter: Yash Sharma
• Created: 29/Mar/17 12:22
• Updated: 03/Jul/17 11:15
• Resolved: 16/May/17 22:11

Description

The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES.

This change is critical for kinesis checkpoint recovery when the kinesis backed rdd is huge.
Following happens in a typical kinesis recovery :

• kinesis throttles large number of requests while recovering
• retries in case of throttling are not able to recover due to the small wait period
• kinesis throttles per second, the wait period should be configurable for recovery

The patch picks the spark kinesis configs from:

• spark.streaming.kinesis.retry.wait.time
• spark.streaming.kinesis.retry.max.attempts

Issue Links

• links to: [Github] Pull Request #17467 (yssharma)