How to initialize state tranformation updateStateByKey?
I have word counts from previous spark-submit run, and want to load that in next spark-submit job to start counting over that.
One proposal is to add following argument to updateStateByKey methods.
initial : Option [RDD [(K, S)]] = None
This will maintain the backward compatibility as well.
I have a working code as well.
This thread started on spark-user list at: