[SPARK-7661] Support for dynamic allocation of resources in Kinesis Spark Streaming

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 2.0.0
• Component/s: DStreams

Environment

AWS-EMR

• Flags: Important
• Assignee: Unassigned
• Reporter: Murtaza Kanchwala
• Created: 15/May/15 06:35
• Updated: 11/Mar/17 10:50
• Resolved: 11/Mar/17 10:50

Description

Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis Stream.

My Requirement is that if I use this Resharding util for Amazon Kinesis :

Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils

Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only).

Issue Links

• is related to: SPARK-12133 Support dynamic allocation in Spark Streaming