[SPARK-3660] Initial RDD for updateStateByKey transformation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Soumitra Kumar
• Created: 23/Sep/14 20:16
• Updated: 12/Nov/14 20:26
• Resolved: 12/Nov/14 20:26

Description

How to initialize state tranformation updateStateByKey?

I have word counts from previous spark-submit run, and want to load that in next spark-submit job to start counting over that.

One proposal is to add following argument to updateStateByKey methods.
initial : Option [RDD [(K, S)]] = None

This will maintain the backward compatibility as well.

I have a working code as well.

This thread started on spark-user list at:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-initialize-updateStateByKey-operation-td14772.html

Issue Links

• links to: [Github] Pull Request #2665 (soumitrak)